Reading a bot's marketing

Sophomore · enrolled learners

This lesson opens with Backtest overfitting, formally.

What happened

Bailey, Borwein, Lopez de Prado and Zhu published in the Notices of the American Mathematical Society in 2014 and proved that high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, that the probability a backtest is overfit rises with the number of configurations tried, and that because analysts and academics rarely report how many were tried, an investor cannot evaluate the degree of overfitting in most investment proposals put in front of them. A bot's parameters, being a range, an interval, a lookback and a threshold, constitute exactly such a configuration space, and a published performance figure is the output of some search across it.

The decision point

The result is not that backtests are dishonest. It is that a performance figure without a configuration count is uninterpretable in principle, not merely unverified, because the same figure is produced by a good strategy and by a search over a bad one. Every bot marketing page displays the figure and none displays the count.

Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.

It is free. We do not sell the list and there is nothing to buy at the end of it.