Curriculum·R411 Building, Validating, and Operating Your Own Automation·about 32 min
Backtesting automation honestly
By the end of this lesson you can
- →Name the four places look-ahead enters through a data pipeline rather than through logic
- →Run a walk-forward validation and state what each fold can conclude
- →Compute the trial count an automated search actually performs
- →Measure the size of your own look-ahead by shifting the data by one bar
Senior · enrolled learners
This lesson opens with The Ariane 5 conversion, read as a validation boundary.
- What happened
- The inertial reference system that destroyed Ariane 5 Flight 501 on 4 June 1996 was, per the inquiry board, practically the same as the one flying on Ariane 4, particularly as regards the software, and Ariane 4 flights using it had never failed this way. It converted a 64-bit floating point horizontal bias value into a 16-bit signed integer, which can represent values only up to 32,767. Ariane 4's flight profile never produced a horizontal velocity large enough to exceed that. Ariane 5 accelerated faster on a different trajectory, the value overflowed within 37 seconds of ignition, and the resulting exception was not handled. Both redundant inertial units ran the same software and failed for the same reason, the backup one 72-millisecond data cycle before the active one, so the redundancy provided no protection at all. The inquiry board found that the specification and equipment tests did not include Ariane 5 trajectory data, so the function was never tested under simulated Ariane 5 flight conditions.
- The decision point
- The code had a validation record of Ariane 4 flights without this failure, and every one of those flights was inside an envelope nobody had written down. A backtest is the same document: a record of correct behavior across a range of inputs, which says nothing about behavior outside that range and does not state where the range ends. Per part two the honest version of a backtest reports the envelope rather than the return.
- Recorded loss
- $370,000,000
What you will be able to answer
- →Where does look-ahead enter?
- →What does a walk-forward fold conclude?
- →What is an automated search's trial count?
- →How do you measure your own look-ahead?
Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.
It is free. We do not sell the list and there is nothing to buy at the end of it.
Sources and review
- https://www.esa.int/Newsroom/Press_Releases/Ariane_501_-_Presentation_of_Inquiry_Board_report
- https://web.archive.org/web/20260726010734/http://sunnyday.mit.edu/nasa-class/Ariane5-report.html
- https://www-users.cse.umn.edu/~arnold/disasters/ariane5rep.html
- https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2308659
- https://global-q.org/uploads/1/2/2/6/122679606/houxuezhang2020rfs.pdf
Confidence high·Volatility low·Reviewed 2026-08-07·Owner unassigned
Contested
The course autopsy and this lesson use the same incident for two arguments: at course level, that software correct in one envelope is not validated for another and identical redundancy is not redundancy; here, that a validation record describes an envelope nobody wrote down. The split is declared. The $370M figure for the launcher and payload is widely cited and varies by source.
R405-03 owns the seven sins of backtesting in general, including the multiple testing arithmetic reproduced in part four. This lesson owns the ways automation specifically introduces those sins through a data pipeline rather than through judgment. Keep the split and do not re-derive the sins here.
Walk-forward validation is one scheme among several, including combinatorial purged cross-validation, and it has known weaknesses where the folds are not independent. It is presented here because it is implementable by hand, which per R411-L is what the lab requires.