SolarWinds: The Hunt to Figure Out Who Was BreachedSome Organizations May Never Know If Data Was Stolen
The Russian hacking group suspected of leveraging a tainted SolarWinds software update to infiltrate as many as 18,000 organizations is presenting a forensics challenge unlike any other.
To ensnare such a large group of private companies, government agencies and organizations is the equivalent of factory ocean trawler scraping the seabed. The question now for organizations is whether they were selected by the hackers for further probing, says Joe Slowik, senior security researcher at DomainTools (see: Target Selection: SolarWinds' Orion 'Big Fish' Most at Risk).
"There's a lot of work to do," Slowik says. "I expect a lot of holidays, unfortunately, to be ruined by this activity as many organizations try to understand what their exposure was and whether or not they were impacted by this."
The attackers infiltrated SolarWinds' infrastructure and seeded a backdoor in a software update for the company's Orion network management software.
That malicious update - essentially a foothold into an organization - could be leveraged for access and used to place other malicious software onto company networks. The activity, which may have started in late February and continued until recently, means the hackers could have been digging around networks for 10 months.
The espionage campaign may have been conducted by a Russian group known as APT29 or Cozy Bear, which is linked to Russia's SVR intelligence agency, according to news reports. But that has not been confirmed, and the investigation is likely to continue for months.
Victims include the U.S. Treasury, Commerce, State and Homeland Security departments, the National Institutes of Health and FireEye. One think tank is also a victim, according to computer security firm Volexity.
Although investigators have identified some of the attackers' infrastructure and the malicious SolarWinds update, there could be more that hasn't been uncovered, including other malicious SolarWinds binaries, Slowik says. "That's where this becomes a really hard security problem," he says.
Sense of Panic?
With 18,000 organizations that installed the malicious update, the task now is to meticulously comb systems to see if the hackers roamed around in them, stealing information (see: SolarWinds Incident Response: 4 Essential Security Alerts).
There's a tangible sense of panic among enterprises that use SolarWinds, says Alex Holden, CISO at the consultancy Hold Security. One of his clients - a large U.S. healthcare provider - plans to change all configurations, passwords, network tokens and more.
U.S. government agencies should be highly concerned, but others caught shouldn't necessarily panic, Holden says. Relatively few organizations likely were targeted, and the solution isn't to "start killing the networks and monitoring systems we've built. Whoever is the adversary would not be a kleptomaniac."
The attackers filtered their victims, says Slowik, who has published an in-depth blog post on the attackers' use of domain name infrastructure.
After the malicious SolarWinds update was installed, the malware - depending on if the victim was on an IP range of interest to the attackers - may have made a DNS request to reach out to subdomains related to a command-and-control (C2) domain, avsvmcloud[dot]com.
The answer appears as a CNAME DNS response, and it's a strong sign that attackers were interested in taking their intrusion further, says Steven Adair, founder of Volexity. The malware then reaches out to that new C2 domain, which is used to siphon data.
"You can pretty much definitively tell if the attackers ever took interest in you," Adair says.
Volexity investigated three intrusions at a U.S.-based think tank starting last year through this year. It has concluded that the last intrusion, which occurred around June or July, is connected to the SolarWinds incident. The think tank used SolarWinds' Orion software.
Adair says Volexity suspected SolarWinds software had some connection to the third intrusion, but at the time was unable to prove it definitively. Volexity has described its findings in a blog post. It does not identify the think tank.
The DNS activity means organizations may be able to tell if they were targeted more deeply by looking at their historical DNS requests, Adair says. But not all organizations necessarily keep that kind of data, he says.
The avsvmcloud[dot]com domain has now been sinkholed, which means it is controlled by investigators, according to a source close to the investigation. Investigators will see pings to that IP address, and combined with DNS requests and responses within organizations, they could help isolate victims.
But monitoring that domain also may be of limited use since the attackers' cover has been blown, Slowik says. The victims of most interest are ones that would have been communicating with C2 domains early on, when the campaign started.
Strong Interest in Email Systems
For organizations that downloaded the malicious update but weren't targeted further, "it really comes down to trying to prove a negative," Slowik says. That will involve extensive forensic analysis and incident response work, he says.
The length of time for potential exposure may hamper investigations, Adair says. Machines have been rebooted and, while organizations store logs for systems, that logging doesn't necessarily stretch back 10 months, he says.
"There are a lot of people that are not going to be sure or not they were breached and probably won't have a way to thoroughly investigate it."
—Steven Adair, Volexity
Otherwise, organizations are left with trying to find clues of something awry, such as odd commands that have been run on endpoints, Office 365 and Exchange telemetry and security information and event management SIEM data, among other sources.
"That might be the best way for someone to take a shot in the dark" to figure out an intrusion, Adair says.
Because the attackers showed strong interest in accessing the email systems of the think tank, Volexity is advising organizations to start their analysis in those systems, Adair says. Some organizations may discover an intrusion earlier in the year was actually tied to the SolarWinds hack, he says.
"There are a lot of people that are not going to be sure or not they were breached and probably won't have a way to thoroughly investigate it." Adair adds.