One customer setting takes down much of the web through Fastly
Large-scale supply chain: failure, no attacker8 June 2021
A valid configuration change by one customer triggered a dormant bug in a major CDN, and 85 percent of its network returned errors. Large parts of the web went dark for most of an hour.
- Position
- A content delivery network serving many of the largest sites
- Reach
- 85 percent of the network returning errors at peak; 95 percent restored within 49 minutes
What happened
On 12 May 2021 Fastly deployed software with a latent bug that a particular customer configuration could trigger. On 8 June a customer pushed a valid configuration change that did so. Disruption began at 09:47 UTC, and at peak 85 percent of Fastly's network returned errors, taking down large parts of the web that depended on it. Within 49 minutes 95 percent of the network was operating normally. Fastly said the customer had done nothing wrong and deployed a permanent fix.
The ATG nexus
No attacker was involved. The outage belongs here as a measurement: it showed in one hour how many sites, organizations and readers sit behind a single delivery provider. The research calls such a provider a Digital Supernode, and makes the point that what an outage can stop, an operator holding the same position could alter.
Limits of the comparison
An outage is loud and ends quickly. A manipulation through the same position would be built to do neither.
On this site
- Introduction: Digital Supernodes
- Whitepaper 7.4: CDN as Risk Multiplier
- Concept: The Digital Supernode
- Concept: The Endemic Flaw