Incident Response: The Gaps

Tips for Improving How Organizations React to Breaches

Organizations often do not detect intrusions until after they've been compromised, says Rob Lee of SANS Institute. So how should they improve their incident-response posture? Lee offers three tips.

As the curriculum lead and author for digital forensics and incident response training at the SANS Institute, Lee says not enough organizations are prepared to respond to today's security incidents.

"Most organizations have a really good mindset in terms of information security prevention, but there are very capabilities installed trying to actually detect the intruders," Lee says in an interview with Information Security Media Group's Tom Field [transcript below].

Lee prescribes three recommendations for organizations looking to improve their incident response programs:

  1. Have a dedicated team: "Most teams in organizations are virtual," Lee says, meaning that people are pulled from their normal jobs to do incident response. "But the challenge occurring now is that incident response is never going away at this point." Incident response becomes a full-time job, and thus organizations should have a team prepared for the constant threat cycle.
  2. Prepare for scalability: Organizations need an incident responder to be able to react to not just one machine, but potentially thousands of machines simultaneously, Lee says, referring to what he calls the "scalability equation." This gap ends up being huge for organizations because they typically think of "one responder to one machine."
  3. Bolster additional defense mechanisms: Organizations neglect to implement mechanisms to do a better job detecting intruders. "For most organizations, it's not do you have the right policy or do you have the right people in place, it's are you effectively comfortable knowing that you can detect an intrusion when it occurs," he says.

In an interview about incident response, Lee discusses:

  • Why many organizations aren't even aware of security incidents;
  • Incident response essentials that many organizations lack;
  • New training and certifications available from SANS Institute.

Lee is an entrepreneur and consultant in the Washington, D.C. area, specializing in information security, incident response, and digital forensics. He is currently the curriculum lead and author for digital forensic and incident response training at the SANS Institute in addition to owning his own firm. Lee has more than 15 years of experience in computer forensics, vulnerability and exploit discovery, intrusion detection/prevention, and incident response.

TOM FIELD: Just to set the topic for us here, give us a little bit of your background in incident response, please.

ROB LEE: I've been working on digital forensics and incident response pretty much the entire 15 years of my career so far. Real quick background - I was former Air Force, worked in the AFOSI [Air Force Office of Special Investigations] and also the 609th Information Warfare Squadron. In both of those instances, they were both investigating computer crime intrusions that occurred and also did a lot of incident response. After I was with the Air Force, I went to the intelligence community and worked for an information security group supporting multiple different agencies in the D.C. area. Until most recently, I was at a company called Mandiant and we did a lot of incident response across the Fortune 500 commercial groups that are out there ... looking for advanced adversaries such as advanced persistent threats.

Incident Response Trends

FIELD: Over the past year there have been so many incidents. Which of the security breaches comes to mind when you start thinking about incident response and some of the trends that we've seen?

LEE: I thought this would be one of the questions and it's hard to nail down to just one, and here's why. The major issue that's currently going on in the industry right now is that most organizations, most corporations, are finding themselves in some sort of data breach situation. The estimated percentage of corporations of the Fortune 500 that are currently compromised are up around 40 percent. I'm not hearing a lot of these in the public eye because a lot of it's not being discussed because either payment card information or PII-type data, private identifiable information, is not being stolen. It's more economic and intellectual property theft by advanced adversaries.

The real situation comes down to that there are so many out there that it's almost humorous to potentially quantify. I don't want to focus on one. You could say RSA, you could use the Sony breach, you could use any of these data really popped over the last year that caused a lot of discussion, but the real issue is that almost every organization out there is having to deal with this up-front and primarily why is because they're simply not detecting intrusions. They're completely missing the element that they're currently compromised and they were unaware of it and in some cases the compromise was ongoing for months, if not years, before they're finally informed about it, usually by a third-party entity.

Now the data I have to back that up is not just my own personal experience but there are three reports out there that all echo the same percentages. The Mandiant report, M-Trends from 2012, has around 90 percent of all intrusions of the Fortune 500 that they have investigated were notified by third parties. You have the Trustwave report also; all of the exact same percentages of organizations that have not been able to detect the intrusions were notified by a third party. And you also have, finally, the Verizon Data Breach Report around the same percentage, also showing that most organizations simply cannot detect the fact that they've been compromised. So we're talking about a macroscopic problem, not a very specific one at this point.

Common Gaps

FIELD: When we look beyond the issue of intrusion and we get into responding to the incidents, what are some of the common gaps that you see revealed in organizations' incident response procedures?

LEE: This is a really great question. The biggest gap that I see is typically that for most organizations, incident response policy is not set in reality. If you just go look at the paperwork, the reality basically is that it's set up more for insider threat-type worries. You have HR; you have legal; you have too many entities that are involved in trying to drive the process and you really don't have a nice agile team.

The first thing I usually recommend to organizations in incident response-type situations is that you need to make sure that you have a dedicated team, and this is another part of the problem. Most teams in organizations are virtual, meaning that you pull people off their normal jobs to do incident response but the challenge occurring now is that incident response is never going away at this point. You're constantly in some cycle from here forward dealing with these advanced intruders that your core team it basically becomes their full-time job. Most organizations have not been adequately prepared to do that. That's the first step, that you have the policy and you have to have your team organization at the start, not really prepared for massive on-slot.

The second issue is that the tools that most teams need is what I call the "scalability equation." Now you need an incident responder to be able to respond to not just one machine but potentially 500-1000 machines simultaneously and that could even go to higher numbers if they become decent at what they're doing in trying to respond to that many machines at one time or looking for indicators of an adversary inside their environment. The scalability side ends up being a huge gap that most organizations just aren't able to grasp [themselves] around because they typically have thought one responder to one machine, but now how do you respond to an enterprise with 10,000 machines where you don't know where the intruders are?

FIELD: You mentioned some key stakeholders there: legal, HR. When you think of incident response, what are the key bases organizations really have got to have covered before they proceed?

LEE: It's a very interesting question. I just taught a class on incident response management and there were a lot of managers in there with very similar questions. I said part of the problem is that most organizations simply have a really good mindset in terms of information security prevention that they're trying to make it very difficult for someone to potentially gain a foothold. But there are very few capabilities installed trying to actually detect the intruders, which again we mention the statistics I mentioned earlier. Without decent detection, there's really nothing to respond to unless you receive a phone call, so your entire incident response process - if you focus on your efforts there - ends up being a lot of work without potentially much pay back.

I usually tell organizations that while they're worrying about the information security prevention, they need to start bolstering additional mechanisms to start doing better detection, and once things are detected then you can respond to it. But without detection, you're "fat, dumb and happy," [as] the adage goes. A lot of organizations will count statistics and say, "Hey, we've never had an incident," and that's where you start scratching your head because I'm not sure that I would feel comfortable with those statistics based on what we're seeing today. It probably means that you're saying you can't detect the incidents that are going out, so the real basis coming into the question is that for most organizations, it's not do you have the right policy or do you have the right people in place, it's are you effectively comfortable knowing that you can detect an intrusion when it occurs?

Training and Certifications

FIELD: Let's talk about SANS Institute now. What are the types of training and certifications that SANS offers that are relevant to incident response?

LEE: Good question. This has been an ongoing transition for even SANS over the past five or six years that incident response has become something that used to be virtualized; during a situation you would do one-to-one. Now we have to train individuals to be able to respond to nation-state actors or advanced adversaries who are good, decent and persistent in their activities, and they're probably not going to just go away because you can't arrest them. There are enemies out there that are basically outside of law enforcement. No matter how much we potentially try to go after them, it's not going to stop them. Organizations need to start focusing on, "How do we train individuals to not only detect and respond to them but what I call the transition from the incident responder into a hunter, someone who's able to actively not only respond to the incident, but able to go seek out to the crime or an intrusion that's going on inside the organization?"

As a result of that, we are about to release a brand new version of our forensics and incident response training in May, which basically focuses on a real-world and real live enterprise-level incident that we had set up with the same techniques that the advanced adversaries are using to mimic what an incident responder will see so they can actually get their hands on the data and really get a feel for what it's like to respond, how they're able to penetrate an organization and how you're able to detect where they went. The real point of the course is not only how to effectively respond to detection, but how do you effectively get what you need out of it to respond to the larger organization to make sure that you've definitely scoped out exactly the locations where these responders are. That new course is actually coming out in May and the exercise that has built up has almost taken a year to create and I'm really excited about it, because it's probably the most realistic scenario that an attendee probably get their hands on without actually being in the incident at this point.

FIELD: There are so many people in an organization that have got a piece of incident response. Who among them is best suited for this training you just described?

LEE: Well, for the course on incident response, the type of individual that's best suited for that is someone who's definitely more on the admin, more on the technical side. It's definitely more leaned toward an advanced course because we do cover memory forensics. We do cover advanced time-line analysis. The best type of person that I usually find is someone who has really good problem-solving skills and they're not afraid to be in a situation where they say they're staring at something that they don't know what it is, but they're willing to just nod until they figure it out. We basically, intentionally place some of these items in front of the students in order for them to basically say in a real world, you're going to be facing very similar circumstances, so how do you potentially respond to it?

There's no black and white, "how do you do this," but there's a good process that we established with these individuals. So typically that's from the technical side, but I've actually had individuals who are very motivated who don't have as much technical experience as others, but end up being some of the best responders because they have that key problem-solving skill, the tenacity that says, "You know, I don't know what I'm looking at, but I'm not going to quit until I figure it out." That's the type of individual that usually succeeds well in both incident response in organizations but also in the training environment like we set up.

Preparing for the Next Incident

FIELD: Final question for you. As you say, it's just a matter of time before an organization sees its next incident. What are the key areas that organizations must address before this next incident occurs?

LEE: It goes back to the initial stats I said. Right now we're not going to be in a situation where we could claim that we're making good headway against the adversaries that we're currently up against until most organizations are detecting them, meaning that those statistics I mentioned earlier would fall below 50 percent of companies being notified by third parties that they have an intrusion going on inside their walls. Incident response is a huge component of that because when you start getting into more scalable enterprise incident response using threat detection and threat intelligence - techniques that are now out there - it really helps move an organization to detect these incidents, know where to look for them, start to hunt in their organization more so than they have ever done before, not just placing all your hope in your imaginable line from the defensive.

They will still need to be there, but the organization needs to transition over into where we're going to hunt down these intrusions, assuming that they're probably going to be there and we need to set up some tools and techniques and dedicated individuals that are going to be hunting for them. And I really think those numbers will drop once these organizations start to do that transition. We've seen it by the way in organizations that have been routinely compromised by these advance adversaries. They learn fast. In over a year, it's almost night-and-day between an organization that was pre-knowledge of intrusion and a year later where they're actively detecting, responding and eliminating beachheads, data exfiltration points and malware continuously across their enterprise and organization.

Around the Network

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing, you agree to our use of cookies.