Curriculum·R411 Building, Validating, and Operating Your Own Automation·about 32 min
Paper and staged deployment
By the end of this lesson you can
- →Name the four deployment stages and what each one is allowed to conclude
- →Compute what a staged size does to the cost of a deployment error
- →Specify the promotion criteria before the first stage rather than during it
- →Explain why a deployment is a change to a running system rather than a launch
Senior · enrolled learners
This lesson opens with Knight Capital's deployment, 27 July to 1 August 2012.
- What happened
- During a deployment beginning 27 July 2012, a Knight Capital technician did not copy new routing code to one of the eight servers running the firm's order routing system. That eighth server retained dormant code from 2003 whose safety checks had been disabled by a change in 2005. The new deployment repurposed a flag that the old code interpreted differently. When the market opened on 1 August the untouched server began sending orders continuously, executing roughly 4 million trades in 154 stocks totaling more than 397 million shares at a rate near $10M a minute. An internal system had sent 97 automated emails referencing the router before the open, identifying an error, and nobody acted on them. About $440M was lost in 45 minutes and the firm lost its independence.
- The decision point
- The deployment was not a launch, it was a change to a system that was already running with money in it, and the failure was in the difference between the seven servers that received the change and the one that did not. Nothing about the new code was defective. The lesson for an individual is the same at a smaller scale: every deployment is a change to a live configuration, the old version does not disappear when you write a new one, and the only reliable way to find out what changed is to run it at a size where finding out is affordable.
- Recorded loss
- $440,000,000
What you will be able to answer
- →What are the four stages?
- →What does staging cost an error?
- →When are promotion criteria written?
- →Why is a deployment not a launch?
Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.
It is free. We do not sell the list and there is nothing to buy at the end of it.
Sources and review
- https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf
- https://www.sec.gov/newsroom/press-releases/2013-222
- https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/
- https://www.henricodolfing.ch/en/case-study-4-the-440-million-software-error-at-knight-capital/
Confidence high·Volatility low·Reviewed 2026-08-07·Owner unassigned
Contested
R403-06 uses Knight Capital for the absence of a circuit breaker, being a threshold with an action that fires without a human. This lesson uses the same incident for the deployment process that created the condition, being the eighth server, the dormant code and the repurposed flag. Two different arguments from one incident and the split is declared. Keep it.
Accounts of exactly how the flag was repurposed and how the dormant code behaved differ in detail between the SEC order and third-party reconstructions. The load-bearing facts here are that one of eight servers did not receive the change, that the code it retained had been dormant since 2003 with its safety checks disabled in 2005, and that the loss was about $440M in 45 minutes.
R405-01 owns the six pipeline stages including paper and staged live. This lesson owns the deployment mechanics specifically. Keep the split.