Your system is only as good as its data feed

Graduate · enrolled learners

This lesson opens with The Facebook IPO on Nasdaq, 18 May 2012.

What happened

Nasdaq's system for opening the Facebook IPO, the most anticipated listing in years, entered a loop in its order-matching software under a flood of orders and cancellations, and for more than two hours it could not tell firms whether their orders had executed. Trading firms were left blind to their own positions: they did not know whether they owned Facebook shares or not, so they could not manage the risk they might or might not be holding, and the resulting confusion cost member firms an estimated 500 million dollars, with Nasdaq later fined by the SEC and paying tens of millions to firms. Nothing was hacked and no key was stolen; the confirmation feed, the plumbing that tells a system the state of its own orders, broke under load, and a trading system that does not know the state of its own orders cannot act with any confidence. The firms had the capital and the strategies; what they lost was the data telling them what they actually held.

The decision point

A trading system runs on data it does not itself produce: the feeds that report market state and, just as important, the confirmations that report the status of the system's own orders, and when those feeds fail the system does not merely slow down, it goes blind, because it no longer knows what it holds or what it has done. The Facebook IPO is the case: Nasdaq's confirmation feed looped and stalled for hours, so firms could not tell whether their orders had filled, and without that data they could not manage a position they could not even measure, at a cost of around 500 million dollars. This is the first lesson of trading-systems engineering: the data feeds are infrastructure, load-bearing and fallible, not a reliable given, so a system must be built to know when its data is stale, missing, or wrong, and to refuse to act confidently on data it cannot trust. An algorithm reasoning on a feed it assumes is always fresh and correct is reasoning on an assumption, and when the feed fails, as feeds do under exactly the load a strategy most wants to trade, the algorithm keeps computing on stale or absent data as if nothing were wrong. So the discipline is to treat every feed the system depends on, market data and order confirmations alike, as a component that can fail, to detect staleness and gaps, and to make the system's behavior safe when the data cannot be trusted, because the Facebook IPO showed that the most expensive thing a trading system can lose is not its money or its strategy but its knowledge of its own state.

Recorded loss

$500,000,000

Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.

It is free. We do not sell the list and there is nothing to buy at the end of it.