Inside a Forensics Investigation How to Track the Source and Scale of a Breach

Forensics expert Rob Lee says its not new types of attacks that concern him. It's the old ones that continue to impact organizations. How can organizations learn from past incidents and respond in 2013?

The bulk of the cases he investigates are external breaches, not insider cases, says Lee, a seasoned forensics professional and curriculum lead and author for digital forensic and incident response training at the SANS Institute. When analyzing the incidents and reporting back to technical teams or executives, he's often faced with the question, "How do we stop this?"

"Even though we're learning more about what the capabilities are of the hackers and adversaries, we have not done a decent job of being able to truly implement solutions that will slow them down ... and even stop the initial infiltration," he says in an interview with Information Security Media Group [transcript below].

Moving forward, organizations need to address the breach at the point of data exfiltration.

"That ends up being a much louder and significant event on a host and a network and much easier to detect as a result," Lee explains.

The main trend heading into 2013 will be for enterprises to formulate effective breach responses to tactics that continue to overwhelm them. And to get to that point, organizations need to embrace the power of big data, which has been difficult for some entities because of the sheer amount of information gathered within an enterprise.

"But as we're moving forward, we're starting to see some solutions creep forward that will give us that visualization and give us the capabilities to identify these anomalies as they're ongoing," Lee says.

In an interview about the process, skills and tools needed in a forensics investigation, Lee discusses:

  • Typical investigations he conducts;
  • Attack trends and what we can learn from them;
  • Most important skills for forensics pros to master.

Lee is an entrepreneur and consultant in the Washington, D.C., area, specializing in information security, incident response and digital forensics. He also is the curriculum lead and author for digital forensic and incident response training at the SANS Institute, a computer security training, certification and research organization. He has more than 15 years of experience in computer forensics, vulnerability and exploit discovery, intrusion detection/prevention and incident response.

Typical Forensics Cases

TOM FIELD: We've talked about forensics a number of times in the past. What do you find to be the types of forensics cases that you're typically seeing these days?

ROB LEE: I'm typically involved in a lot of information security data breach-style cases where there's some sort of intellectual property that's stolen. There are a lot of different forensics cases that are out there, but that's the kind I'm personally seeing, where some people might be doing intellectual property theft, insider threat-type things, but there seems to be a huge need right now, especially on the data breach investigation side.

In-Demand Skills

FIELD: Let's talk about the skills. What do you find to be the forensic skills that most commonly get called upon to use in these cases that you're involved in?

LEE: One of the key skills that I have found that's particularly useful for investigators is not only to have firm knowledge in digital forensics and as it applies to the operating system, but have a firm knowledge of what adversaries are potentially capable of accomplishing within your enterprise environments. How do they breach a system? How do they gain domain admin? Where do they go? How do they move from system to system? Finally, how do they collect the information from the organization in order to exfiltrate it from your network? Knowing key hacking skills tells you pretty much what to look for on the network.

In the same vein though, having a really firm understanding of how domain environments work, where enterprise environments are laid out in an organization, also ends up being very helpful. I have spent many hours over the past year really trying to educate myself on typical things that probably a domain system administrator finds easy work, but for me it's all brand new.

Inside an Investigation

FIELD: I'm hoping we can walk through what would be a typical case here. You talk about getting typically data breach investigations. At what point does this breach case end up in your lap and what's the typical situation?

LEE: A typical situation that I might find myself in is clients that I'm currently working with have been informed by a third party, usually law enforcement or the FBI, that they have had a data breach, and they're looking at one of the systems that was told to them by law enforcement potentially has been breached. They're trying to figure out if it has been breached. Is there malware on the system? How can we use that information in order to be able to identify additional systems that are breached within our environment?

Basically, it comes down to: I have a system that possibly has evil on it. How do I potentially find that evil malware in order to be able to uncover the next part of the case, which is: Was this the first system involved in the breach, or is this system laterally moved to from another box within your environment? Or was this system the last stop, meaning that this is where they exfiltrated all their data from your network?


FIELD: Let's talk about your evaluation. What are you looking for?

LEE: Typically, I'm looking for signs of the intruder on the system. It starts in a process that I call "malware funneling," stepping through the idea that malware has to follow a core rule set, which is malware can hide, but it must run. It goes against the grain that most people think it's very easy to hide on a system, but if malware is actually active it's usually fairly easy to identify pretty quickly. The hard cases though are the ones where there's actually no active malware sitting on the system. In those situations, either A) that data was collected from the system or B) maybe it was infiltrated by an adversary at one point but they have uninstalled their malware. It actually becomes a little bit more difficult if it's actually not an active malware situation.

The process I use ends up of taking a system, trying to lift everything that's known and trying to really key in on things that end up being fairly particular to an adversary footprint, whether it's on the active side for active malware or going down into the weeds looking for configuration files, bad scripts and other log files that would indicate that an adversary's present had existed on the machine at an earlier point in time.

Common Tools Used

FIELD: What would be the common tools that you use in this process?

LEE: The two things that I end up using more often than not are memory analysis toolkits such as the Volatility toolkit or Redline from Mandiant. Both of those tools are fairly good. Even for a novice investigator to be able to wrap their heads around and be able to step through what's going on in the active system, this will help you identify if there's active malware that's currently running. On the systems level, I'm using a capability called timeline analysis, which is more of an overarching capability in the forensics community [where] it holds all of your artifacts together in a single temporal output and you're able to go around the time that you think is the time of the breach, for example, or time of specific activity, and see if you could identify the adversary's footprint through the timeline.

Between those two activities, it actually ends up capturing almost 80 percent of the type of analysis that I will need to accomplish.

Common Case Findings

FIELD: You've talked about common cases and about common tools. What are the typical findings that you're ending up with in one of these investigations?

LEE: The typical findings are usually never good. They usually think they will be a little bit brighter, but usually what it finds is that the system is usually the tip of the iceberg. It's almost a harbinger of the doom that's already impended within their network. The findings that we're currently uncovering is that the adversaries have been in most of the networks for months, in some cases years, prior to this level of investigation being accomplished. With unfettered access for that amount of time, there are many instances where it's undetermined amounts of data that they've been able to exfiltrate from your organization. That's probably my most typical finding. Even though it feels like this is brand new, it's not brand new to the bad guys who are currently doing this against your network.

On the opposite side of the fence, we're also finding that the adversaries are really almost brazen in the types of things that they're doing within your network. They're using common malware, such as Poison Ivy, moving it system to system and even using remote desktop protocol to be able to examine the system like you would in order to find the data that they're searching for. As a result of that, with all the information security capabilities within our organizations - FISMA compliance, PCI and more - at this point it has not even caused a slight dent in the adversaries' activities in order to thwart them or even slow them down a little bit from being able to achieve their goals.

Presenting the Findings

FIELD: Typically, you've got to present these findings to organizations and individuals who might not be as technologically savvy as yourself. How do you present the findings?

LEE: It really depends on who the audience is and who you're working with. In many cases, I'm actually working with the technical incident response team of an organization. I'm brought in as an individual trying to augment or have additional expertise on the ground, so I might be communicating to them and they in turn communicate to their executives. In the instances where I end up having to present back to the executives, typically the way I'm trying to present my findings is what was the root cause, what could have been prevented, what's the damage assessment, how do we stop it, and try to get to the weeds. Are there any significant impacts through the intrusion? Try to answer those core questions. Being comfortable with saying, "I don't know," ends up being key to doing incident response today. You're under a lot of pressure if you're being asked those key questions from the executives, and if you can't answer one of them, it is best not to guess ever. Always say, "It's still being investigated at this point. This is all we know." Really stick to the facts. A lot of those that are new to this might jump the gun and make a bad assessment and that could end up causing problems down the line.

Attack Trends

FIELD: Based on the cases that you have come across, what would you say are the attack trends that most concern you today?

LEE: It's not the attack trends as we move forward. It's the stuff that still works that concerns me. If you take a look at an adversary's campaign from an initial spear-phish, to gaining domain admins within your enterprise network, to moving inside the network and finally collecting data to be exfiltrated, every singe one of those steps, even though we're learning more about what the capabilities are of the hackers and adversaries, we have not done a decent job of being able to truly implement solutions that will slow them down, retard their access and even stop the initial infiltration.

As a result, one of the questions that I'm usually asked is, "How do we stop this? What can we do to potentially prevent this in the future?" My answer to that question is instead of trying to stop an intrusion at the onset of it, at the beach head, where the spear-phish attack came in, try to move later down in the adversary's campaign, which would be where they're exfiltrating the data. That ends up being a much louder and significant event on a host and a network and much easier to detect as a result.

First, stop the exfiltration of data, and then stop the lateral movement of the adversaries. Then, stop the data collection and try to limit the capabilities for them to be able to fool your domain admin. Start at the finish and move back to the beginning because the hardest one in the chain is actually preventing spear-phishing e-mails as they come in.

The trends that I'm seeing right now are trying to formulate an effective response to these tactics that are basically overwhelming us at this point, without any significant response from the information security community.

Lessons for Organizations

FIELD: What would you say are some of the lessons that organizations can learn from the attacks that they have suffered and that you have investigated?

LEE: We need to shift the mindset. That is, if you build a big enough wall, no one can get in. The perimeter is more porous than most people realize and we need to shift our perception from perimeter-based defense into detection as a whole, and that wraps your mind around a new concept that you've seen all the time in the news as a buzzword, called "big data security." We should be able to have a core horizon view within our enterprise networks as to anything that's going on at any certain time. The problem with that right now is there's almost too much data that we don't have anyone manning the scopes to be able to help us out, identify and detect these intrusions.

But as we're moving forward, we're starting to see some solutions creep forward that will give us that visualization and give us the capabilities to identify these anomalies as they're ongoing. The focus in my mind right now and the lesson that's going to be learned is to keep perimeter defense up. That's great. Have that in play. But you now need to focus internally on the detection element of these intrusions in the first place so it's not law enforcement that's calling you. It's that you're able to self-detect them as they're ongoing.

Key Skills for Forensics Investigators

FIELD: How about for forensics investigators? What would you say are the key skills that they need to master today?

LEE: In my opinion, especially in an enterprise intrusion, it's the ability to analyze tens of thousands of systems, instead of analyzing a single system. Forensics has typically been to pull a hard drive and analyze a single hard drive. That's not scalable. You need capabilities that are scalable in your environment. It's okay to analyze a single system like I was doing to identify a single piece of malware, but once you identify the single piece of malware, you need to use that malware to develop a new technology called security intelligence, which basically says if the malware exists on any system in your network, what would it look like. There would be some key things that you would identity system to system, and then you forensicate and scan the rest of those systems on your network looking for those footprints.

But that has to be a scalable architecture to allow you to do so, so you have one forensic analyst able to scan and forensicate against tens of thousands of hosts across a week. That's going to be extremely valuable as organizations are looking for solutions for both the personnel and the enterprise. Apply the solution and that mindset that you have to have both of those capabilities in place, someone that's understanding of the fact that you have to be able to scale your forensics investigations and have capabilities already built into your organization's enterprise to allow that individual the access they need in order to get the job done.

Around the Network