Malware Overreaction Proves CostlyIG: Agency Spent Millions for Unnecessary Fix
A unit of the U.S. Commerce Department overreacted to a perceived malware infection and unnecessarily spent more than half of its IT budget to mitigate the situation, according to an inspector general audit.
In December 2011, the Department of Homeland Security alerted the Commerce Department that it detected a potential malware infection within Commerce's IT systems. The Commerce Department determined the infection resided within systems operating at its headquarters and informed its Economic Development Administration and another agency, the National Oceanic and Atmospheric Administration, of a potential infection in their IT systems.
According to the inspector general audit:
- EDA, believing it had a widespread malware infection, requested on Jan. 24, 2012, that the Commerce Department isolate its IT systems from the headquarters' network. That terminated EDA's operational capacity for enterprise e-mail and website access as well as regional office access to database applications and information residing on servers connected to the headquarters' network.
- The Commerce Department and EDA augmented the department's computer incident response team because of their perception that the department had limited incident response abilities and that the malware infection was extensive. DHS, the Energy Department, the National Institute of Standards and Technology and the National Security Agency, along with a cybersecurity contractor, provided additional incident response support.
- The Census Bureau, another Commerce Department unit, provided EDA with interim e-mail capability and Internet access in early February 2012.
Matt Erskine, the department's deputy assistant secretary for economic development, says EDA's main focus from the beginning of the incident was to fully recover its IT functionality in the most secure, efficient and cost-effective manner possible. "Through the incident," Erskine says in a written response to the audit, "EDA acted out of an abundance of caution in an effort to protect the IT security and privacy of our staff, the Department of Commerce, grantees, other federal partners and clients with whom we interacted electronically."
Persistent Mistaken Belief
EDA spent more than $2.7 million - more than half of its fiscal year 2012 IT budget - in pursuit of these recovery activities. "EDA's persistent mistaken beliefs resulted in an excessive response and ultimately unnecessary expenditure of valuable resources," Allen Crawley, assistant inspector general for systems acquisition and IT security, writes in the audit.
Crawley says EDA based its critical cyber-incident response decisions on the inaccurate information that the incident resulted in a widespread malware infection that could spread to other bureaus if agency computers remained connected to the department's IT network.
EDA destroyed $170,000 worth of IT components - desktops, printers, TVs, cameras, computer mice and keyboards - to ensure that a potential infection could not persist. But the IG found no evidence of a widespread malware infections or of the need to isolate EDA's system from others. "The destruction of IT components was clearly unnecessary because only common malware was present on EDA's IT systems," Crawley says.
EDA had planned to destroy another $3 million worth of IT wares, but didn't because it ran out of money by Aug. 1, 2012, the audit says.
"I cannot recall hearing about any such reaction since the early 1990s, and even then it would have been extreme," Purdue University Computer Professor Eugene Spafford says. "The alert about the problem should have included information about best practices for resolution. All agencies should have access to some best practices and shared resources. Any plan that involves downtown of more than a set amount (perhaps 48 hours) and cost impact above a certain level (perhaps $25,000) should be reviewed by security experts for reasonableness."
The National Oceanic and Atmospheric Administration, the other agency alerted to the malware, took a much more direct approach than the one EDA followed. It analyzed the information regarding the malware infection, remediated it and placed the fixed component back into operation by Jan. 12, 2012.
EDA Didn't ID Inaccurate Information for Months
Deficiencies in the Commerce Department's incident response program hindered EDA's incident response, which contributed significantly to EDA's inaccurate belief that it experienced a widespread malware infection, Crawley says. Commerce's computer incident response team and EDA propagated inaccurate information that went unidentified for months. The IG concludes that incident handlers failed to follow the department's incident response procedures. Handlers for EDA's incident did not have the requisite expertise, and the Commerce Department's incident response team failed to coordinate adequately incident response activities, the audit reveals.
Misdirected efforts also hindered EDA's IT system recovery. The IG report says EDA focused its recovery efforts on replacing its entire IT infrastructure and redesigning its business applications because of its incorrect interpretation of recovery recommendations. Instead, EDA should have concentrated its resources on quickly and fully recovering its IT systems, such as critical business applications.
EDA's system returned to its former capabilities by employing existing shared IT services about six weeks after the incident began.
The IG recommended that Commerce's chief information office should ensure the department's computer incident response team can respond effectively to future cyber-incidents. The IG also recommended that the department ensure incident response procedures clearly define the incident response team as the coordinator for incident responses for Commerce bureaus. And it recommended that incident response team managers maintain proper oversight and involvement in cyber incidents to ensure that required incident response activities take place.
Commerce's Erskine, in a written response to the audit, says the department doesn't dispute the IG's findings and will diligently carry out its recommendations.