Defined as extreme events with high impact, Black Swans are infrequent occurrences that pack a punch (i.e. in financial markets the 2008 crisis, or 2010 flash crash). However a new study shows as the combination of machine trading and speed intertwine, these extreme events are occurring more often than previously imagined.  As markets continue to connect and participants become linked, each extreme bounce and/or collision may slowly break the system.

Nassim Nicholas Taleb is the person most responsible for burning the concept of low probability, high impact events into the minds of global business executives.  Coining the term “Black Swans†as the name for extreme outliers with devastating consequences, Taleb has put executives on notice that they need more built-in redundancy and should incorporate slack in business processes to cushion against failure.

However as technology proliferates and advances thus speeding processes, it appears humans are increasingly removed from decision making.  Thus ensuring a little slack in the system may not be enough to protect from system meltdown.

Take for example a complex “system†such as global financial markets.  In an effort to gain competitive advantage, computer scientists, quants, and software programmers are building machines that scan data streams, analyze, and decide trading strategies in micro-seconds.  These individuals (sometimes hedge fund managers) or corporations (such as larger investment banks) are shrinking the window for decision making down to levels where humans cannot react fast enough—microseconds today and nanoseconds in the future.

Trading equities is now a technological “arms raceâ€, where companies compete buying and selling at near light speed. And while the concept of using speed for competitive advantage doesn’t sound like such a bad idea, there are also ramifications for a race to zero.

The first issue with this trading arms race is exclusion of participants who cannot afford the requisite technology.  Just as it takes nearly a billion dollars to win a US election thus ensuring few can join the fray, it takes multi-millions to build and co-locate ultra-fast computerized trading platforms. A second issue is that as trading nears the speed of light, there is ultimately less and less slack in the system to correct trading errors. And since financial markets are tightly coupled, this means that one single error in a fragile system can cascade with cataclysmic results.

Trading at near light speed – in an already fragile and tightly coupled system—is driving more extreme events, which appear to be fracturing global markets. And contrary to common knowledge, these events aren’t just happening once every two to three years.

A team of physicists, system engineers, and software programmers recently published a paper suggesting that abrupt “events†are occurring in the financial markets much more than previously thought. In fact, over the years 2006-11, the authors report a total of 18,520 spikes in stock movements—or extreme events (I’ll call them baby black swans) that arguably should have low probability of occurring according to normal distribution statistical models.

The aforementioned study notes; “There is far greater tendency for these financial fractures to occur, within a given duration time window, as we move to smaller timescales.†Meaning that in financial markets, as faster computers slice decision making windows down to nanoseconds, we should expect more volatility.  Moreover, if a given system is not designed to handle extreme volatility, there is a high probability of fissures and potential for total system breakdown.

In 2010’s Flash Crash, the US stock market plunged 1000 points in nine minutes and then regained those losses just as fast.  Never before had market participants seen thousand point swings within a ten minute timeframe. If the authors in the study cited in this article are correct, this kind of extreme volatility is only the beginning.

Questions:

Billy Beane’s “Moneyball†approach to developing and staffing a professional baseball team has come under intense scrutiny as long time major league scouts and analysts take delight in sub-par performances of the Oakland Athletics.  Beane however seems undaunted in using statistical analysis to undercover market place inefficiencies.  Indeed, it takes courage to solve problems in a whole new—perhaps heretical—manner. And while plenty of folks take pleasure in Beane’s recent comeuppance, there’s a good chance he’s already had the last laugh.

By now, most business professionals have either read Michael Lewis’ “Moneyball†or at the very least seen Brad Pitt’s rendition of Billy Beane in Hollywood’s version.  Beane’s claim to fame is that he had the courage to defy decades of “common knowledge†in assembling a winning baseball team.  For many years, winning teams were accustomed to paying top dollar for the best hitters and pitchers evidenced by metrics such as batting average and earned run average respectively.

However, in the small-town Oakland market, Beane didn’t have the luxury of affording a high payroll. And if he wanted to compete, he needed to find a different (and perhaps better) method of keeping up with the likes of the Boston Red Sox and New York Yankees.

Beane knew there were “inefficiencies†in baseball markets that could be exploited; he just had to discover those undervalued metrics.

Enlisting the help of his Harvard friend Paul Depodesta, Beane used statistical analysis to discover metrics that—in his opinion were more indicative of baseball success—such as on base percentage and slugging percentage for batters, or the ability to get “outs†for pitchers.  Then Beane went about acquiring a motley bunch of players that most teams discarded as near worthless, and positioned each player for success based on his analytical models.

And for a while it worked. With one of the league’s lowest payrolls the Oakland A’s competed with teams spending 2-3x in overall salaries. In fact, a Sports Illustrated article mentions; “From 2000 to 2006, (Oakland A’s) averaged 95 wins, captured four AL West titles and made five playoff appearances.â€

Alas, all competitive advantage is fleeting. As Paul Depodesta left for greener pastures, and a few of Beane’s disciples took other league jobs, major league baseball teams figured out what made Moneyball work and effectively neutralized the Oakland A’s advantage.  With other teams adapting to Moneyball tactics, from 2007 and beyond, the Oakland A’s were a .500 team at best, and sometimes much worse.

And in the process of adopting Moneyball tactics, Billy Beane made plenty of enemies. The “old guard†definitely had a preferred way of defining baseball success and would often scoff at the “fat bodies†Beane employed, and laugh at the un-orthodox pitchers he would trot out in the ninth inning.  And in regard to the lackadaisical record of the Oakland A’s in recent years, one scout sneered; “So much for the genius … He doesn’t look so smart anymore, does he?â€

Competing on analytics takes courage. In response to analyst and pundit criticism leveled at Beane for his Moneyball approach, author Michael Lewis says; “Beane had the nerve to seize upon ideas rejected, or at least not taken too seriously, by his fellow Club members, and put them into practice.â€Â  For a while, Beane had the advantage. It might not have been pretty, but the Oakland A’s were effective at winning as many games as other prestigious teams, for half the cost.

And Beane ultimately may have the last laugh. Today, there are teams like the Boston Red Sox that take the best of Moneyball tactics and overlay them with powerful financial resources to take baseball competitiveness to a whole new level. Baseball has been changed forever from a “gut feel†business to one that’s analytically driven.

Fortunately for today’s analytical professionals, “competing on analytics†is taking hold as more companies understand they’re sitting on literal goldmines of structured and multi-structured data just begging for analysis. But analytics users be forewarned, there’s still plenty of “old guard†that will put up roadblocks, enlist subterfuge and even openly mock your new data driven approaches.

Michael Lewis notes that Beane didn’t invent sophisticated analytical analysis—he just had the courage to use it to create competitive advantage and shake up a traditionally stodgy industry.  There are plenty of industries ripe and ready for the business value that analytics will eventually unlock. Are you ready for the challenge?

In the United States health information privacy is protected by the Health Information Portability and Accountability (HIPAA) act.  However, new gene sequencing technologies are now available making it feasible to read an individual’s DNA for as little as $1,000 USD.  If there is predictive value in reading a person’s gene sequence, what are implications of this advancement? And will healthcare data privacy laws be enough to protect employees from discrimination?

The Financial Times reports a breakthrough in technology for gene sequencing, where a person’s chemical building blocks can be catalogued—according to one website—for scientific purposes such as exploration of human biology and other complex phenomena. And whereas DNA sequencing was formerly a costly endeavor, the price has dropped from $100 million to just under $1,000 per genome.

These advances are built on the back of Moore’s Law where computation power doubles every 12-18 months paired with plummeting data storage costs and very sophisticated software for data analysis.  And from a predictive analytics perspective, there is quite a bit of power in discovering which medications might work best for a certain patient’s condition based on their genetic profile.

However, as Stan Lee’s Spiderman reminds us, with great power comes great responsibility.

The Financial Times article mentions; “Some fear scientific enthusiasm for mass coding of personal genomes could lead to an ethical minefield, raising problems such as access to DNA data by insurers.â€Â Â After all, if indeed there is predictive value via analyzing a patient’s genome, it might be possible to either offer or deny that patient health insurance—or employment—based  on potential risks of developing a debilitating disease.

In fact, it may become possible in the near future to assign a certain patient or group of patients something akin to a credit score based on their propensity to develop a particular disease.

And something like a predictive “score†for diseases isn’t too outlandish a thought, especially when futurists such as Aaron Saenz forecast; “One day soon we should have an understanding of our genomes such that getting everyone sequenced will make medical sense.â€

Perhaps in the near future, getting everyone sequenced may make medical sense (for both patient and societal benefit) but there will likely need to be newer and more stringent laws—and associated penalties for misuse) to ensure such information is protected and not used for unethical purposes.

Question:

For web-based businesses, and of course, those with a web presence (which is just about everyone) there’s a goldmine of behavioral data accessible with the right tools. The trick is getting past static web analytic reporting (bounce rates, page views, session times etc.) and going further into unlocking the rich treasure trove of machine data, text and weblogs that create “big data†insight.

In your business, gigabytes if not terabytes of multi-structured data are likely just waiting to be coupled with your imaginative thinking and analysis to create new data products that ultimately help drive customer interactions and revenues

For example, take a look at what LinkedIn is doing in creating new “products†with data they collect and analyze with a MapReduce approach and other techniques.

According to a recent whitepaper “Building Data Science Teamsâ€, LinkedIn’s former Chief Data Scientist shows how smart thinking can be paired with compute power and huge quantities of multi-structured data to create innovative new products such as:

And of course, many of the above “products†are more than simply focused on nebulous metrics such “customer engagementâ€â€”they can directly tie to revenue improvements.

There are even opportunities to drive news cycles with unlocked insights. Companies such as LinkedIn can use information gleaned from their web server farms to build press releases such as:  “Top Ten Phrases Recruiters Want to See†or “Top Ten Job Growth Areas in the United Statesâ€.  These kinds of press releases are interesting to local newspapers, bloggers and media outlets, especially if there’s a unique angle relevant to readers/viewers.

There are plenty of companies turning to outside firms, crowds, and even their own customers for innovation. And there’s certainly nothing wrong with any of these approaches. However, those approaches may be trying too hard – especially when there’s a goldmine waiting to be unleashed just a few web servers away.