One customer setting takes down much of the web through Fastly

Large-scale supply chain: failure, no attacker8 June 2021

A valid configuration change by one customer triggered a dormant bug in a major CDN, and 85 percent of its network returned errors. Large parts of the web went dark for most of an hour.

Position
A content delivery network serving many of the largest sites
Reach
85 percent of the network returning errors at peak; 95 percent restored within 49 minutes

What happened

On 12 May 2021 Fastly deployed software with a latent bug that a particular customer configuration could trigger. On 8 June a customer pushed a valid configuration change that did so. Disruption began at 09:47 UTC, and at peak 85 percent of Fastly's network returned errors, taking down large parts of the web that depended on it. Within 49 minutes 95 percent of the network was operating normally. Fastly said the customer had done nothing wrong and deployed a permanent fix.

The ATG nexus

No attacker was involved. The outage belongs here as a measurement: it showed in one hour how many sites, organizations and readers sit behind a single delivery provider. The research calls such a provider a Digital Supernode, and makes the point that what an outage can stop, an operator holding the same position could alter.

Limits of the comparison

An outage is loud and ends quickly. A manipulation through the same position would be built to do neither.

On this site

Sources

  1. Summary of June 8 outageFastly (Nick Rockwell), 8 June 2021Primary source
  2. Fastly explains web blackoutThe Register, 9 June 2021
  3. Incident review: Fastly outage impacts major websites worldwideCatchpoint, June 2021

Back to this event on the timeline