Lessons to Learn From CircleCI's Breach InvestigationBeware: Malware Bypassed Antivirus; Attackers Reused Stolen Single Sign-On Tokens
CircleCI's misfortune to be on the receiving end of an attack that infected an employee's laptop with malware and bypassed two-factor authentication defenses are potential industrywide lessons learned - at least for any company wise enough to learn from another's experience.
See Also: Threat Horizons Report
CircleCI is a juicy target. Its continuous integration and continuous delivery platform is used by over 1 million developers, including those at such organizations as Airbnb, Google, Meta, Okta and Salesforce.
The incident is notable as yet another case in which attackers have bypassed two-factor authentication defenses via technical trickery or social engineering (see: Cisco Hacked: Firm Traces Intrusion to Initial Access Broker).
Details of the attack consequently are worth studying for what they reveal about attackers' tools and tactics and as a goad to ensure that your own house is in order.
Malware used to infect the CircleCI engineer's laptop wasn't detected by the company's endpoint security tools and was then used "to steal a valid, 2FA-backed" single sign-on session cookie, CTO Rob Zuber writes in the latest breach update, issued Friday. This token enabled the attacker "to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems."
While stolen customer data was being stored in encrypted format, the attacker "extracted encryption keys from a running process, enabling them to potentially access the encrypted data," Zuber says.
As a result of the breach, "we recommended that all customers rotate their secrets, including OAuth tokens, Project API Tokens, SSH keys and more," he says. He also recommends that customers look for any suspicious activity on their systems from Dec. 16, 2022, until Jan. 4. "If you stored secrets on our platform during this time period, assume they have been accessed and take the recommended mitigation steps," he adds.
CircleCI says it began rotating customers' project API and personal API tokens on Jan. 4. The same day, it issued requests for GitHub OAuth, AWS and Atlassian's Bitbucket Git-based source code repository hosting service to rotate their tokens.
The company doesn't know how many customers may have been affected. "Because the targeted employee had privileges to generate production access tokens as part of the employee's regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens and keys," Zuber tells customers. "There is no way for us to know if your secrets were used for unauthorized access to those third-party systems."
As of last week, however, he says no more than four customers have reported "unauthorized access to third-party systems as a result of this incident."
Timeline of 19-Day Breach
Here's the timeline of the incident as reconstructed so far by CircleCI, showing how quickly attackers moved from gaining access to exfiltrating data:
- Dec. 16, 2022: Attacker infects employee laptop with malware and steals and uses session cookie.
- Dec. 19: Attacker conducts reconnaissance.
- Dec. 22: Attacker exfiltrates encrypted data as well as encryption keys needed to decrypt data.
- Dec. 29: CircleCI launches security investigation after customer alerts it to suspicious behavior.
- Dec. 30: CircleCI learns "that this customer's GitHub OAuth token had been compromised by an unauthorized third party."
- Dec. 31: CircleCI "proactively" initiates "the process of rotating all GitHub OAuth tokens on behalf of our customers."
- Jan. 4, 2023: CircleCI's probe identifies "the scope of the intrusion by the unauthorized third party and the entry path of the attack," locks down systems, blocks all access for compromised employee's accounts, confirms attacker access eliminated, begins alerting customers and publishes public breach notification.
- Jan. 13: Company publishes updated results from investigation, plus details of tactics, techniques and procedures - including IP addresses, data centers and VPN providers, and malicious files - used by attacker.
Several notable details stand out about the particulars of this attack:
- Timing: Attackers struck in the run-up to the busy holiday period.
- Reconnaissance: The company missed an opportunity to detect the attack when a system was infected by malware or during the three days of reconnaissance.
- Notification: CircleCI learned about the breach from an external source, which while not ideal, isn't uncommon.
- Encryption: Attackers may have bypassed the use of encryption to protect data at rest, which is a serious problem.
- Tokens: Insufficient defenses were in place to spot or stop an attacker who reused a stolen, valid SSO token.
What has CircleCI learned from the attack? The company says it has made or will be implementing a number of discrete security improvements, including refining:
- Endpoint security: Detection is now in place for the malware - as well as behavior exhibited by this type of malware - in its antivirus and also mobile development management tools.
- Access: The company has "restricted access to production environments to a very limited number of employees" while it gets better defenses in place.
- Step-up controls: For anyone who needs production environment access, "additional step-up authentication steps and controls" should block attackers' ability to use stolen session tokens.
- Monitoring: New monitoring and alert triggers are in place "for the specific behavior patterns we identified in this scenario," including behavior tied to "a variety of third-party vendors."
- Reviews: The company is more regularly reviewing its security controls - and while it doesn't go into greater detail, hopefully this entails better penetration testing and red-team exercises.
CircleCI has apologized to customers for the breach and the disruption and added work this is causing them, and it has thanked them for their patience while it conducted its investigation.
Ideally, no company would ever fall victim to a breach, and especially one that imperils customer or consumer data. But in the real world, criminals are working overtime to secure the opposite result. Kudos, then, to CircleCI for sharing information not just about how it got hacked and what's at risk but for also sharing actionable details to help others better protect themselves.