Curriculum·R411 Building, Validating, and Operating Your Own Automation·about 32 min
Operating the system
By the end of this lesson you can
- →Specify what the system must do when the venue is unavailable
- →Explain why an order must be idempotent and what happens when it is not
- →Compute the retry volume a naive loop generates during a real outage
- →Write the daily operating checks and the state reconciliation that closes them
Senior · enrolled learners
This lesson opens with The New York Stock Exchange halt, 8 July 2015.
- What happened
- On 8 July 2015 the New York Stock Exchange suspended trading in all its listed securities from 11:32 in the morning until just after 3pm New York time, a halt of about three and a half hours. The exchange attributed it to a configuration issue concerning how its systems interact with one another following a software rollout, and stated explicitly that it was not a cyber breach. Trading in the affected securities continued on other venues including Nasdaq and Bats, which absorbed the flow, so the securities themselves remained tradable elsewhere throughout. In 2017 it was reported that the exchange had been told the outage may have breached securities rules.
- The decision point
- For three and a half hours a venue that had never done this before did it, following a routine change, with no attack involved. Every automated system connected to it faced the same three questions in that window: what to do about orders already resting, what to do about signals firing with nowhere to send them, and what to believe about its own positions. A system that has not been told the answers will improvise, and per part two an improvising system reconnecting to a venue that has come back is how one order becomes several.
What you will be able to answer
- →What are the three outage questions?
- →What is idempotency?
- →What does a naive retry loop generate?
- →What closes the daily checks?
Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.
It is free. We do not sell the list and there is nothing to buy at the end of it.
Sources and review
- https://www.cnbc.com/2015/07/08/trading-halted-on-nyse-floor.html
- https://www.washingtonpost.com/business/economy/nyse-trading-has-been-halted/2015/07/08/46b51974-2588-11e5-b72c-2b7d516e1e0e_story.html
- https://www.bloomberg.com/news/articles/2015-07-08/new-york-stock-exchange-suspends-trading-in-all-securities
- https://www.cnbc.com/2021/06/30/robinhood-to-pay-70-million-for-misleading-customers-and-outages-the-largest-finra-penalty-ever.html
Confidence medium·Volatility low·Reviewed 2026-08-07·Owner unassigned
Contested
The exchange's public explanation attributed the halt to a configuration issue relating to how its systems interact, following a software rollout. More detailed accounts of the internal sequence were not published, and this lesson does not assert one. The load-bearing facts are the duration, the cause category and that other venues continued trading the same securities.
The retry arithmetic in part three assumes a naive fixed-interval loop, which is the failure mode rather than good practice. Exponential backoff with jitter and a maximum attempt count is the standard remedy and is described in part four. The figure is there to show the scale of what an unbounded loop produces.
R404-03 owns the availability of exits from the trader's side and R403-06 owns circuit breakers. This lesson owns the operating behavior of an automated system under those conditions. Keep the splits.