Application Security , Incident & Breach Response , Managed Detection & Response (MDR)
Widespread Internet Outages: Fastly Reports Problem FixedAccess to Amazon, Google, Twitter and Others Disrupted by Content Delivery Network
Amazon, Google, Spotify and Twitter were among the sites that were unreachable Tuesday morning due to a configuration error at widely used content delivery network Fastly.
See Also: LIVE Webinar | Stop, Drop (a Table) & Roll: An SQL Highlight Discussion
The website Down Detector, which tracks when internet services and apps are unreachable, reported widespread outages on Tuesday. Other unreachable sites included Amazon Web Services, CNN, Discord, HBO Max (part of AT&T's WarnerMedia), The New York Times, Reddit and the U.K. government's public-facing website, among others.
The outage began Tuesday morning and was quickly traced to Fastly, based in San Francisco. After about an hour, services began to get restored.
Fastly says the disruption involved a configuration error in its points of presence. Such CDNs - aka "edge clouds" - are meant to provide more rapid access to widely used sites by replicating the sites' content across points of presence around the world, which are ideally located as physically close to website users as possible.
But if a CDN itself goes down, it can leave numerous sites unreachable.
"We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration," a Fastly spokeswoman tells Information Security Media Group. "Our global network is coming back online."
The company's website lists the current status of its network.
Fastly says the outage began about 6 a.m. U.S. East Coast time. At 6:36 a.m., it reported: "Fastly has observed recovery of all services and has resolved this incident."
Two hours later, the company said that all services had been restored, and the incident had been resolved, although "customers could continue to experience a period of increased origin load and lower cache hit ratio." Origin load refers to the load placed on customers' servers, while cache ratio measures how effective a cache is at meeting requests for content.
We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration. Our global network is coming back online. Continued status is available at https://t.co/RIQWX0LWwl— Fastly (@fastly) June 8, 2021
The disruptions due to the Fastly configuration error did not disrupt access to affected sites from all geographies. For example, some users in Berlin reported no outages, while outages were reported in such locations as London, Texas and New Zealand, The Guardian reports.
This isn't the first outage involving a CDN that knocked downstream customers offline.
On Aug. 30, 2020, for example, CenturyLink was offline for most of that Sunday morning, which in turn led to sites such as Cloudflare, Discord, Feedly, Hulu, PlayStation Network, Xbox Live and many others being unreachable.
The outage for Cloudflare, which is a CDN, in turn led to outages for dozens of its customers as well.
That followed a July 17, 2020, outage at Cloudflare that lasted a half hour, which the company blamed on a configuration error.
"The outage occurred because, while working on an unrelated issue with a segment of the backbone from Newark to Chicago, our network engineering team updated the configuration on a router in Atlanta to alleviate congestion," Cloudflare said at the time in a blog post. "This configuration contained an error that caused all traffic across our backbone to be sent to Atlanta. This quickly overwhelmed the Atlanta router and caused Cloudflare network locations connected to the backbone to fail."