The backtest that hid the tail

Graduate · enrolled learners

This lesson opens with Victor Niederhoffer's fund, 1997.

What happened

Victor Niederhoffer was a celebrated systematic trader whose strategy included selling out-of-the-money options, which collects a steady premium in almost every period and had produced years of excellent, smooth returns in the record. Selling those options is a bet that a large move will not happen, so the strategy earns a little again and again while quietly carrying the risk of a rare, enormous loss. In October 1997 a sharp market drop, a move his approach treated as improbable, forced losses that wiped out his fund, and he blew up again a decade later on another tail event. The record had looked outstanding right up to the moment it did not, because the history it was measured over simply did not contain the kind of event that the strategy was exposed to. Nothing about the track record was fabricated; it was an honest picture of the calm periods, and the calm periods were exactly the ones in which a short-the-tail strategy looks best.

The decision point

A backtest measures a strategy over the history in the data, and if that history does not contain the rare event a strategy is exposed to, the backtest will look excellent precisely because it is blind to the loss that matters, so a smooth record can hide a strategy that is short the tail: one that earns a little in almost every period and loses catastrophically in the rare one. Niederhoffer is the case: an option-selling strategy showed years of steady, attractive returns, then a single sharp drop his history had never contained wiped out the fund, because selling out-of-the-money options makes money in the calm and loses everything in the crash, and the calm is what the record was made of. This is the first and deepest lesson of validation: the numbers a backtest reports are conditional on the events the data happened to include, so a strategy whose risk lives in the tail will show its best face in any sample that omits the tail, which most samples do. The error is to read a smooth equity curve as evidence of safety, when for a short-the-tail strategy the smoothness is the symptom, not the reassurance. So the discipline is to judge a strategy by what it does in the tail it is exposed to, not by the smoothness of its average: ask what event would ruin it, check whether the backtest's history ever contained such an event, and if it did not, treat the record as untested against the very thing that can kill it, because Niederhoffer's numbers were real and honest and still told him nothing about the day that ended the fund.

Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.

It is free. We do not sell the list and there is nothing to buy at the end of it.