Curriculum·R405 Strategy Development and Validation·about 33 min
The seven sins of backtesting
By the end of this lesson you can
- →Name the seven sins and the specific artifact each one produces in a result
- →Compute the probability of a spurious winner as a function of variants tried
- →Derive the corrected threshold a result must clear given the number of trials
- →State the one number that makes any backtest interpretable, and report it
Senior · enrolled learners
This lesson opens with Backtest overfitting, documented 2014.
- What happened
- In 2014 David Bailey, Jonathan Borwein, Marcos Lopez de Prado and Qiji Jim Zhu published Pseudo-Mathematics and Financial Charlatanism in the Notices of the American Mathematical Society. They demonstrated that high simulated performance is easily achievable after testing a relatively small number of alternative strategy configurations on the same data, a practice they named backtest overfitting, and that the probability a backtest is overfit rises with the number of configurations tried. Their central observation about practice was that most analysts and academics rarely report how many configurations were tried, which means a reader cannot assess the degree of overfitting in any given proposal. Strategies presented as mathematically sound and empirically supported by an outstanding backtest can therefore be artifacts of the search rather than findings about markets.
- The decision point
- Nothing in this requires anybody to be dishonest. A researcher who tries a hundred variants and reports the best one has done what feels like diligence, and the result is a number that describes the search rather than the market. The disclosure that makes a backtest interpretable is the count of configurations tried, and it is the one number almost nobody reports, including to themselves.
What you will be able to answer
- →What are the seven sins?
- →What is the spurious winner probability?
- →What threshold corrects for it?
- →What one number makes a backtest interpretable?
Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.
It is free. We do not sell the list and there is nothing to buy at the end of it.
Sources and review
- https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2308659
- https://www.sciencedaily.com/releases/2014/04/140410103005.htm
- https://scholarworks.wmich.edu/math_pubs/40/
- https://global-q.org/uploads/1/2/2/6/122679606/houxuezhang2020rfs.pdf
Confidence high·Volatility low·Reviewed 2026-08-07·Owner unassigned
Contested
The multiple-testing arithmetic in part one assumes independent trials at a 5 percent significance level, which is a simplification because strategy variants on the same data are correlated. Correlation reduces the effective number of independent tests, so the figures here overstate the probability for a given raw count and understate it for a given amount of genuinely independent searching. The direction of the argument is unaffected.
The seven-sin list is a teaching taxonomy rather than a canonical one, and other authors group these failures differently. What matters is that each named item produces a specific, identifiable artifact in a result, which is the property used throughout the lesson.
R411-02 covers backtesting automated systems, where look-ahead and capacity failures enter through the data pipeline rather than through judgment. Keep the split and route implementation questions there.