Who Hijacked Google's Web Traffic?Data Routes Through Russia, Nigeria and China, Raising Security Concerns
Google is investigating an unorthodox routing of internet traffic that on Monday sent traffic bound for its cloud services instead to internet service providers in Nigeria, Russia and China.
See Also: Automating Security Operations
The routing problems persisted for about two hours before they were fixed, says Alex Henthorn-Iwane, vice president of product marketing for the security company ThousandEyes.
The fact that it affected such a large swath of Google's networks makes it unlikely the routing was simply an error, especially because it involved network providers within Russia and China, Henthorn-Iwane says.
"It's not a mistake," he says. "There's nothing about this that suggests that this was a mistake."
Traffic bound for Google services went through Nigeria's MainOne, China Telecom and Russia's TransTelekom. In an update on its Cloud Platform service site, Google says its services operated as expected, although that contradicted what networking monitoring companies observed. The cause of the issue appears to be external, Google says.
"We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence," the company says.
BGP is Outdated
ISPs and telecommunications companies use an internet standard called Border Gateway Protocol, or BGP, to communicate changes in routing. Simply put, BGP distributes routing information, enabling routers to connect users with specific IP address prefixes.
BGP is an open standard used by all ISPs. But routing traffic can be a delicate process, and mistakes can end up in outages. Also, there are concerns that nation-states have been tampering with BGP in order to watch internet traffic for surveillance purposes (see: Who's Hijacking Internet Routes?).
Criminals have also taken an interest in BGP. Earlier this year, an attacker successfully created a BGP leak, seized control of IP addresses and intercepted cryptocurrency wallet data (see: Cryptocurrency Heist: BGP Leak Masks Ether Theft).
Some Traffic Ended in China
The Monday traffic routing problem for Google meant that some U.S. users attempting to reach Google may have been bounced through other countries before reaching Google's servers. In some cases, it appears that traffic that went through China Telecom didn't actually end up connecting with Google at all.
China blocks many Google services, so it is possible that the country's so-called Great Firewall, which blocks access to many services as part of the country's censorship regime, just dropped the traffic. It's also possible that too much traffic may have hit China Telecom at once, causing a jam.
ISPs "announce" new internet traffic routes between networks, known as autonomous systems - ASes - using BGP. Those announcements are then often re-announced or propagated by other ISPs, which is intended to ensure that worldwide routing stays efficient while networks come online or others go down.
But computer security experts have long warned that BGP is particularly vulnerable to malicious manipulation because it's an open system. It is possible for ISPs to reject BGP announcements from other ISPs, and many now more closely watch announcements that may be errant, for security reasons.
"It's the irony that the very openness, agility and resilience that the internet displays also creates a vulnerability," Henthorn-Iwane says.
Ongoing Security Risk
Only Google should be announcing its traffic routes. But what happened on Monday is that someone or some entity pretended it had authority to announce new routes to Google services.
BGP is largely an honor system, Henthorn-Iwane says. Network operators hope that other network operators won't make unauthorized BGP announcements on their behalf. But there are no real mechanisms in place to stop that from happening, leaving operators dependent on detecting mistakes on their own.
Alan Woodward, a professor of computer science at the University of Surrey, describes fundamental internet protocols such as BGP and DNS as being "the soft underbelly of the web," in desperate need of a security overhaul. To date, however, the political and business will to fix these legacy infrastructure problems hasn't been found.
Woodward tells Information Security Media Group that the BGP problems stem from the internet being a network of networks, and no one having designed any of its core protocols with security in mind. "Security was never thought to be an issue as no one envisaged it being used as it is now," he says.
"BGP is totally based upon trust at present and if that is broken - by mistake or deliberately - then routing can be subverted. There are initiatives to try to secure BGP, such as Secure Inter-Domain Routing, but they will take a long time to be universal."
As specified by the Internet Engineering Task Force, Secure Inter-Domain Routing is an initiative - not a standard - that would enable the creation of infrastructure that would allow an entity "to verifiably assert that it is the legitimate holder of a set of IP addresses or a set of autonomous system numbers," according to the IETF's overview.
Pending such fixes, BGP security remains an ad hoc affair. "Many countries have the luxury of having someone watch for unusual routing announcements and to correct as needed," Woodward says. "Smaller countries - some may have a handful of ISPs at best - do not."
Rob Joyce, the National Security Agency's senior adviser for cybersecurity strategy to the director, says via Twitter: "I hope this latest fiasco of traffic rerouting through China is the wake-up call for all of us to get serious about addressing the massive and unacceptable vulnerability inherent in today's BGP routing architecture."
Henthorn-Iwane says that the errant BGP announcement affected 150 or so "prefixes," which are essentially blocks of IP addresses. Each Google prefix contains as many as 8,000 individual IP addresses, he says.
It's not clear where the rogue announcement originated. But Henthorn-Iwane later wrote on Twitter that it could have occurred at an internet exchange in Lagos, where China Telecom and MainOne peer. ISPs often connect together at centralized points, called internet exchanges, to save on costs. Such arrangements are known as peering.
Once the routing announcement was made, it appears that China Telecom then broadcast it to the Russian ISP TransTelekom, according to BGPmon, which is part of Cisco's OpenDNS.
This is what we know: Starting at 2018-11-12 21:12 UTC Nigerian ISP AS37282 'MainOne Cable Company' leaked 212 @google prefixes to China telecom. Causing traffic to be redirected and dropped.— BGPmon.net (@bgpmon) November 12, 2018
Leaked BGP Paths via Tier1 ISP NTT disappeared at 22:32 UTC.
The tainted route was then picked up by other ISPs, including NTT and Cogent, before it was eventually fixed, Henthorn-Iwane says.
Traffic hijacking poses security risks, but there is a known defense against anyone being able to study intercepted data: encryption.
Connections between a user and a Google service are encrypted using Transport Layer Security, or TLS, which is often referred to as SSL, short for Secure Sockets Layer. An ISP that's helping to transit traffic would see the originating IP address and know the destination, but the content itself would be encrypted.
But intelligence services around the world actively work to break encryption, and there are certain kinds of attacks designed to break SSL. Decrypting traffic, however, is thought to be impossible for anyone who doesn't possess the necessary TLS keys, which, in theory, should be well protected.
Today's computers aren't powerful enough to rapidly calculate decryption keys, but quantum computing is expected to change that. Experts suspect that well-resourced intelligence agencies may be recording traffic now in the hope that once quantum computing become more prevalent and cost-effective, this historical traffic can be decrypted and analyzed.
MainOne Takes the Blame
Meanwhile, later on Tuesday, Nigerian ISP MainOne took the blame for the BGP routing problem, and said it had been fixed less than 90 minutes after it happened.
We have investigated the advertisement of @Google prefixes through one of our upstream partners. This was an error during a planned network upgrade due to a misconfiguration on our BGP filters. The error was corrected within 74mins & processes put in place to avoid reoccurrence— MainOne (@Mainoneservice) November 13, 2018
"This was an error during a planned network upgrade due to a misconfiguration on our BGP filters," the ISP said.
Despite the mea culpa, many continue to urge caution. "I'll take the Nigerians at their word that they didn't intentionally hijack the BGP routes," says Jake Williams, founder of Rendition Infosec, a security consultancy based in Augusta, Georgia. He's also an instructor at the SANS Institute and a former operator with the NSA's Tailored Access Operations unit.
"But if I were China and wanted plausible deniability, I'd use an ISP in a country known for corruption," he says. "TL;DR don't take everything at face value."
Executive Editor Mathew Schwartz contributed to this story.