Search This Blog

Sunday, 5 August 2012

Why it’s better having to deal with human mistakes after all: example at Knight Capital shows how a computer can crash your company… in minutes

Two of the most expensive losses in the history of the banking industry were definitely caused by humans:

I. Nick Leeson, a British, Singapore-based trader, personally defaulted Barings bank with a combination of:
·         speculative trading with large sums of money on the presumed direction of the Nikkei index;
·         hiding the mounting losses on a secret account out of sight from his peers and superiors;
·         trying to make up for earlier losses, by executing the one gamble that could wipe out all previous losses, which of course… it didn’t: the Las Vegas-syndrome;

Starting in 1992 with his speculative actions, Nick Leeson caused Barings a mindboggling damage of $1.4 bln, when he eventually fled the bank at broad daylight in 1995. At the time, Barings was finished as a private bank and was taken over by the Dutch banking conglomerate ING Groep NV ($ING)

II. Jérôme Kerviel, an unsuccessful French trader that was dissatisfied with the size of his bonuses, tried to pull the same stunt at the French bank Société Générale ($SCGLY) as Leeson did at Barings:
·         Kerviel speculated with large sums of bank money (north of €50 bln) on the presumed direction of the DAXX, Eurostoxx and FTSE indexes;
·         He created a fake company that would operate as a ‘straw man’ for his orders;
·         His main purpose was not to peculate his profits from the bank (according to Rather to the contrary, he wanted to use those profits for improving his trading results, in order to become a more successful trader and receive a higher bonus in the process;

Although Kerviel’s fraud only lasted for a few months (from the fall of 2007 until the spring of 2008), the results of it were beyond imagination: a direct loss of €1.5 bln and subsequent losses of €3.4 in the process of unwinding the open contracts that Kerviel’s actions left behind. This was officially the biggest fraud case in the history of banking. Only direct intervention of the French government could supposedly prevent SocGen from defaulting in 2008.

These were kinds of fraud previously unheard of in their sheer consequences and the financial havoc they caused in the banking industry. As stated before, Barings ceased to exist as an independent business bank and SocGen suffered also gravely from the consequences.

Still, these cases of fraud paled in comparison to what happened last week: an error in a computerized High Frequency Trading system of Knight Capital Group (KCG ) caused this bank to lose $440 mln in … just 45 minutes!

The absolute amount of the damage caused by the events at Knight Capital is not extremely large (“only” $440 mln), when compared to the previously discussed cases of fraud. What is flabbergasting, however, is the speed in which the events took place and the displayed impotence of the human controllers against errors in the HFT computer systems: they stood by and watched the havoc unfold!

Here are the pertinent snips of an FT article describing the events that took place at Knight:

It took just 45 minutes for a trading glitch to create a $440m loss for Knight Capital this week.

That loss is almost double the revenue generated by the company in its entire second quarter and was enough to wipe three-quarters off the value of the market-maker within 24 hours. For Wall Street, it’s the equivalent of watching a state-of-the-art aeroplane come crashing to earth after a momentary software problem.

Last year, Knight dominated trading in NYSE and Nasdaq-listed stocks and was routinely showered with awards for its advanced information technology and high-speed trading systems.

Knight is now said to be looking for a partner to help shore up its capital and possibly even rescue the company. It’s a stunning defeat for a market-maker that has come to embody the silicone-coated transformation of modern trading.

The question now is whether this series of technical troubles will force financial companies to rethink their use of technology, even after trade “automation” has become the buzzword of Wall Street in the face of new profit-crimping regulation.

CreditSuisse analystsrecentlyestimated that pure “high-frequency trading”, which includes algorithmic strategies, now accounts for almost half of US stock trading. “Real money” stock trading is now at decade lows, they say.

The kind of algorithmic trading employed by Knight Capital thrives on automation; hire a skilled maths expert to create a complex formula that arbitrages minute discrepancies in stock prices, ensure that it’s executed at speeds measured in milliseconds, set it loose in markets and sit back and reap the profits.

For Knight, that automation went spectacularly wrong on Wednesday. The company said it installed new market-making software that ended up “sending numerous erroneous orders in NYSE-listed securities into the market”.

To market-watchers, it looks as though the software was spitting out orders for stocks and then buying them up almost instantaneously, a pattern which, as market data firm Nanex points out, would mean Knight instantly losing the difference in price.

Years into what’s been described as a“high-frequency armsrace” amongfinancials,these companiesare reaching the limits of available technology and pushing what they have to greater extremes.

This article in the Financial Times brings me to the explanation of the title of my article. The combined losses of the damage caused by Leeson and Kerviel are more than ten times as high as the damage of the events at Knight Capital.

The difference is, however, that normal banking controls and regulations, a little bit less trust and more ‘spider sense’ among the managers and peers of these two gentlemen would almost certainly have prevented these events from happening. These guys were humans that acted very human and almost got away with it, because their direct colleagues and management acted very human too. In optimistic times nobody sees a raincloud until it starts to rain.

The scary part at Knight Capital was that no human could have prevented these events from happening, without abolishing the HFT-systems as a whole. Understanding what happens in the HFT code during critical situations is much to difficult for normal humans, especially in the very detailed way that is required to thoroughly test this software. People trust the mathematical magicians, simply because they don’t have the power to prove them wrong.

The Financial Times does a nice job with this article and the question it asks (first red paragraph) is a good one. However, there are two much more fundamental questions that ought to be answered first.

·         Should we implement technology that we fundamentally don’t understand and that we can’t possibly check on being flawed anymore ?

·         Should we implement technology that we can’t control anymore with our human senses and that can create havoc within milliseconds?

Everybody that saw the true purpose of ‘Skynet’ unfold in the surprisingly good part III of The Terminator franchise, knows that these are the most fundamental questions concerning computer technology. Especially in a time where our lives are increasingly interconnected with computer systems and we increasingly lose influence on the quality and maintainability of the computer code in these systems.

Hence, when only 500 of the ca. 6,000,000 lines of computer code in a Boeing 777 contain (fundamental) flaws, then Boeing’s software is 99,99167% error free. On the other hand, would you be the one to find out what the errors in these 500 lines are all about while being in mid-air at 30,000 feet of altitude? The hell you wouldn’t!

And would you like to see your life-savings vaporize at a bank or an insurance company, due to an error that some nerd programmed in a mathematical formula that nobody else in the company fully understood?!

I am a vastly experienced software-tester with more than 14 years of hand-on experience and –  I may say – a darn good one. Today, I had another successful implementation of new software at the global system bank where I work.

However, if someone asked me to test and take the responsibility for the HFT-software that Knight Capital implemented, I would reply that I’m not smart and intelligent enough to do this job and take this responsibility. Seemingly, I was not the only one that should have said this.

The failure at Knight Capital cannot be blamed on the nerd that programmed the mathematical algorithm, although this will definitely happen in the near future. The true failure lies by the people that don’t understand the ultimate consequences of the two very fundamentals questions that I asked. Both questions must be answered with a clear “no”!