How One Fastly Customer Broke The Internet

In the wee hours of Tuesday morning, a tech vendor named Fastly experienced a major outage that inadvertently broke some of the biggest sites on the web. And now, we know why. In a blog post published late yesterday, Fastly engineering VP Nick Rockwell explained the company “experienced a global outage due to an undiscovered software bug,” that surfaced when a single customer just… reconfigured his internet connection.

Per Rockwell’s blog, Fastly issued a software update in mid-May that accidentally came pre-packaged with the bug that could be triggered under specific circumstances. In the early morning hours on June 8th, one of Fastly’s customers unwittingly put these circumstances into motion while rejiggering their internet connection via a “valid configuration change.” Before they knew it, 85% of Fastly’s network started returning errors, sites stopped loading, and global pandemonium ensued.

Rockwell explained that Fastly noticed the global outage “within one minute,” and then made quick work of patching up the issue. “Within 49 minutes, 95% of our network was operating as normal,” he wrote. “This outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them.”

It turns out that’s a lot of people. Along with companies like Cloudflare and Akami, Fastly is one of the so-called “content delivery network” (CDN) giants that hold up vast swaths of the internet by letting sites store their data within their proprietary clouds. Storing data like this doesn’t only mean that content gets to your browser faster, but it also means that more people can access that content at once.

At least, that’s what’s supposed to happen. But tech hiccups crop up all the time, and CDN’s aren’t immune. Back in 2019, Cloudflare went through a similar outage to the one Fastly went through this week, inadvertently taking platforms like Dropbox, Medium, and Soundcloud along with it. In 2017, a four-hour AWS shortage knocked out sites like Netflix and Spotify. When the handful of tech companies that underpin the internet are just as fallible as, well, literally every other tech company, you have to wonder if that power consolidation is a good thing.