The backtest that hid the tail

Graduate · enrolled learners

This lesson opens with Victor Niederhoffer's fund, 1997.

What happened

Victor Niederhoffer was a celebrated trader, a pioneer of using computers to predict short-term moves in stocks and commodities, whose yearly returns had reached 30 percent and whose strategy included selling out-of-the-money options, which collects a steady premium in almost every period. Selling those options is a bet that a large move will not happen, so the strategy earns a little again and again while quietly carrying the risk of a rare, enormous loss. In 1997, with his funds already down almost 50 percent after a failed attempt that August to pick a bottom in Thai equities, he was caught short S&P 500 puts in the US market fall two months later, when the Dow fell 7.2 percent on 27 October; his broker called for margin he could not produce and the fund was wiped out, and a decade later, in 2007, his Matador fund closed down more than 75 percent after another sell-off. The record had looked outstanding right up to the moment it did not, because the history it was measured over simply did not contain the kind of event that the strategy was exposed to. Nothing about the track record was fabricated; it was an honest picture of the calm periods, and the calm periods were exactly the ones in which a short-the-tail strategy looks best.

The decision point

A backtest measures a strategy over the history in the data, and if that history does not contain the rare event a strategy is exposed to, the backtest will look excellent precisely because it is blind to the loss that matters, so a smooth record can hide a strategy that is short the tail: one that earns a little in almost every period and loses catastrophically in the rare one. Niederhoffer is the case: an option-selling strategy showed years of attractive returns, then a single sharp drop the calm years had not prepared it for wiped out a fund already weakened by a losing Thai bet, because selling out-of-the-money options makes money in the calm and loses everything in the crash, and the calm is what the record was made of. This is the first and deepest lesson of validation: the numbers a backtest reports are conditional on the events the data happened to include, so a strategy whose risk lives in the tail will show its best face in any sample that omits the tail, which most samples do. The error is to read a smooth equity curve as evidence of safety, when for a short-the-tail strategy the smoothness is the symptom, not the reassurance. So the discipline is to judge a strategy by what it does in the tail it is exposed to, not by the smoothness of its average: ask what event would ruin it, check whether the backtest's history ever contained such an event, and if it did not, treat the record as untested against the very thing that can kill it, because Niederhoffer's numbers were real and honest and still told him nothing about the day that ended the fund.

Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.

It is free. We do not sell the list and there is nothing to buy at the end of it.