Overfitting: a pattern that was only noise

Graduate · enrolled learners

This lesson opens with The Foolish Four, late 1990s.

What happened

The Foolish Four was a mechanical stock-picking formula popularized by a well-known investing publication, a variant of the Dogs of the Dow that selected a handful of Dow stocks each year by a rigid rule tuned to the numbers, such as dividend yield and price. Run over past data, the formula showed spectacular historical returns, far above the market, and it was promoted on the strength of that record. But the rule had been shaped to fit the very history it was tested on, so when it was scrutinized and applied to data it had not been fitted to, the extraordinary returns fell apart and it did no better, and often worse, than simply buying the market, and the publication eventually retired it. Nothing was fabricated; the backtest was real, but it was a real measurement of a rule that had been curved to match past noise, so its impressive numbers described the past it was fitted to and carried no predictive power forward. The pattern it captured was not a signal about how stocks behave; it was an accident of the particular data it was built on.

The decision point

A rule that is tuned to fit a fixed set of past data will match that data well by construction, but much of what it is matching is noise, the accidental particulars of that sample, so its impressive backtest describes the past it was fitted to and does not carry forward, and on new data the fitted pattern, being noise, disappears. The Foolish Four is the case: a formula shaped to the numbers of past Dow performance showed spectacular historical returns and then, tested on data it had not been fitted to, did no better than the market, because the pattern it had captured was an artifact of its sample rather than a real signal. This is overfitting, and it is the central trap of backtesting: the more a rule is optimized against one dataset, the more of that dataset's noise it absorbs, so a strategy that has been tweaked and tuned until its backtest looks excellent is often excellent only at describing the data it was tuned on. The tell is that the fit is too good and was achieved by adjustment, and the cure is out-of-sample testing: a rule earns belief only by performing on data it was never allowed to see while it was being built, because performance on the fitting data is guaranteed and therefore meaningless. So the discipline is to separate the data used to build a rule from the data used to judge it, to be suspicious of any strategy whose backtest was improved by optimization, and to treat a spectacular in-sample record as a warning to check for overfitting rather than as evidence of a discovery, because the Foolish Four's real, spectacular backtest was a picture of noise the rule had been curved to fit.

Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.

It is free. We do not sell the list and there is nothing to buy at the end of it.