Supercomputer Intrusions Trace to Cryptocurrency MinersLikely Connected: Attacks Against Systems in US, UK, China, Germany and Beyond
Cryptocurrency-mining hackers appear to be behind a recent spate of supercomputer and high-performance computing system intrusions. But it's unclear if the attackers might also have had data-stealing or espionage intentions.
Attacks against high-performance computing labs, which first came to light last week, appear to have targeted victims in Canada, China, the United States and parts of Europe, including the U.K., Germany and Spain. Security experts say all of the incidents appear to have involved attackers using SSH credentials stolen from legitimate users, which can include researchers at universities and consortiums. SSH - aka Secure Shell or Secure Socket Shell - is a cryptographic network protocol for providing secure remote login, even over unsecured networks.
Supercomputer and high-performance clusters offer massive computational power to researchers. But any attackers able to sneak cryptomining malware into such environments could use them to mine for cryptocurrency, referring to solving computationally intensive equations in exchange for potentially getting digital currency as a reward.
On May 11, the bwHPC consortium, comprising researchers and 10 universities in the German state of Baden-Württemberg, reported that it had suffered a "security incident" and as a result took offline multiple supercomputers and clusters.
The same day, the University of Edinburgh took offline the U.K.'s national high-performance computing system, called ARCHER, due to a "security incident," and noted that similar incidents had also affected research institutions in other parts of Europe (see: 'Security Incident' Knocks UK's ARCHER Supercomputer Offline).
ARCHER was one of a number of high-performance computing environments apparently hit by attackers sneaking two pieces of malware onto Linux systems: a cryptomining-malware loader called "fonts" and a log-cleaning file called "low," both of which were placed in the "/etc/fonts" Linux directory.
In the following days, many more such security incidents came to light in Germany, as well as Spain and Switzerland, as ZDNet has reported. Some of the high-performance computing environments appear to have been hacked as early as January, Bleeping Computer reports.
Two Attack Campaigns, Possibly Connected
On Friday, the security team at European Grid Infrastructure, which coordinates supercomputer research across Europe, said that its members had been seeing attacks that traced to two campaigns, which may be linked. Both occurred in a similar timeframe and involved connections to the same monero mining server, as well as attackers connecting to targeted systems at times via the same compromised server in Poland, although other servers were also used.
Only the second series of incidents, however, appeared to involve the use of the Linux malware.
The first series of attacks reported by EGI members traces to a malicious group that has been hitting HPC labs in Canada and the United States, as well as China and Europe, to install CPU mining software. "The attacker connects to these hosts via SSH, often from Tor," EGI says in its security alert, noting that "the attacker is hopping from one victim to another using compromised SSH credentials."
In at least one attack, "the malicious XMR [monero] activity is configured to operate only during night times to avoid detection," it said.
As staging grounds for the attacks, hackers used compromised systems at the State University of New York at Stony Brook, UCLA, the University of Toronto, Germany's University of Freiburg, the China Science and Technology Network - CSTNET - and also the Poland's University of Krakow, EGI said.
In the second series of attacks, "a malicious group is currently targeting academic data centers for unknown purpose," EGI said. "The attacker is hopping from one victim to another using compromised SSH credentials."
Attackers were logging into targeted systems, EGI said, via compromised systems at University of Krakow, China's Shanghai Jiaotong University and CSTNET.
Attackers used two pieces of malware, placed in the Linux fonts directory: .fonts, which in reality is a SUID file - for setting user access rights - designed to give allow attackers to execute their files with root privileges, and .low, which is a log-cleaning file designed to remove the attackers' traces.
"I work in HPC in the U.K.," one Slashdot user posted on Wednesday. "Yesterday I had to revoke all the SSH certificates on our system because unfortunately some ... idiot users have been using private keys without passcodes. These have been used [by attackers] to hop from system to system as many HPC users have accounts on different systems. They are managing [to achieve] local privilege escalation on some systems and then looking for more unsecured SSH keys to jump to other systems. They maybe using one or more methods for the privilege escalation, possibly CVE-2019-15666."
CVE-2019-15666 is an out-of-bounds array access flaw in versions of the Linux kernel before 5.0.19; the current version is 5.0.21.
"Right now, the U.K. national facility ARCHER is offline as they have suffered a root exploit," the Slashdot poster added. "The actors are coming from the following IP addresses: 220.127.116.11 and 18.104.22.168."
Separately, EGI on Friday tied those IP addresses to exploited user accounts at Shanghai Jiaotong University and CSTN.
Security researcher Tillmann Werner (@nunohaien) has released rules for the Yara malware research and detection tool that can be used to detect signs of this attack.
Thread: High performance computing labs are currently reporting breaches. Germany seems to be impacted the most with several victims.— Tillmann Werner (@nunohaien) May 15, 2020
Robert Helling (@atdotde), a physicist who works at the Leibniz Supercomputing Center in Germany, used the National Security Agency's free reverse-engineering tool Ghidra to decompile the loader and cleaner and found a list of the specific types of files targeted.
Samples of the loader and cleaner files have been uploaded to anti-virus scanning service VirusTotal by users in Germany, Spain, Switzerland and the United Kingdom, meaning that there may be victim organizations in all of those countries, among others, says Chris Doman, co-founder of London-based cloud computer incident response tool vendor Cado, in a blog post.
I took a look at the recent attacks against Supercomputers and found some more details on attacks against Supercomputers in the UK, US, Germany and elsewhere -> https://t.co/EXdxB2jPI1 pic.twitter.com/3gOaLKCfyQ— chris doman (@chrisdoman) May 16, 2020
"The cleaner is quite subtle, and removes traces of the attacker from a number of log files," he says.
Apparent Motivation: Cryptomining
The FBI last week warned that Chinese hackers have been targeting academic and pharmaceutical industry research into COVID-19 therapies (see: US Says China-Linked Hackers Targeting COVID-19 Researchers).
But the discovery of cryptocurrency mining malware on the systems compromised as part of the two security incidents detailed by EGI means that "it is more likely these are criminal, financially motivated attacks," rather than espionage efforts, Doman says.
In the meantime, the affected high-performance computing groups are still trying to get their systems cleaned and restored.
As of Monday, for example, the U.K.'s ARCHER remained offline, as administrators worked to re-issue new passwords and SSH keys to all users.
"When ARCHER returns to service all users will be required to use two credentials to access the service: an SSH key with a passphrase and their ARCHER password," the ARCHER administrators say in a Friday service update. "It is imperative that you do not reuse a previously used password or SSH key with a passphrase."
Potential Culprit: Watchbog Group
Costin Raiu, director of Kaspersky's global research and analysis team, says the attackers may be connected with the Watchbog group, which has been previously tied to attacks that utilized Linux malware to power its cryptomining botnet.
"The Watchbog botnet mines monero cryptocurrency for its owners," security researchers Luke DuCharme and Paul Lee at Cisco Talos said in a September 2019 blog post that described one attack by the crime group against an organization, which was notable in part for attackers' heavy use of SSH.
Cisco Talos found that the attackers first had exploited a known vulnerability, CVE-2018-1000861, in the open source automation server Jenkins to gain remote access to the organization and install their Watchbog malware.
Once they'd gained a foothold, the attackers used a bash script to move laterally across the compromised network. Their script "retrieves the contents of the known_hosts file on the infected system" - referring to the a system's list of other known-good systems - "and then attempts to SSH into those systems." In addition, their script "also checks for the existence of SSH keys and leverages them to authenticate to the systems in the known_hosts file." For this particular attack, if the efforts were successful, the script retrieved the contents from a Pastebin URL to continue the infection process.
Watching for unexpected levels of SSH traffic can help reveal these types of attacks, Cisco Talos says.
"Establish a baseline for internal network traffic and if any significant deviations occur, identify and investigate them - even if there is an existing theory for the activity," it recommends. "In this case, Watchbog generated a noticeable spike in the organization's SSH traffic."