It's true. Most IT leaders and organizations today are trying to hack themselves secure by relying on penetration testing as the sole method to determine whether their applications are secure enough to go into production. But penetration tests can't be a primary source of assurance because they simply aren't designed to verify all of the controls in an application. (See 5 Application Security Tips)
Penetration tests are simulations of real attacks performed by expert humans armed with hacking tools. The idea is that if an application is strong enough to withstand a simulated attack by highly skilled testers, then it is strong enough for the real world. However, penetration testing is not a positive verification of all of the required security controls, which means that many important security controls will go completely untested. For example, some organizations only test external-facing applications, since they believe those are the most likely to be attacked. Others only test applications with the most critical information, that process credit cards or healthcare data.
Until you've put some serious effort into securing your applications, pen testing will offer no value.
In addition, gone are the days when web applications used simple HTTP messages that were easy to intercept and modify with tools like OWASP's Zap. Today's applications use Ajax to send asynchronous messages at a rate that is too fast for humans to tamper with. Serialized data structures using JSON and XML are difficult to reverse engineer and require special tools to modify on the fly. The custom authentication and protocols used in both mobile and HTML5 WebSocket applications create all kinds of difficulties for pen testers.
These technologies are already in use in most organizations. Individually, each of them make manual penetration testing very difficult - but combined, they make it virtually impossible. We are many years away from having security tools that can even start to allow penetration testing to work on the technologies that we use right now. This puts severe resource constraints on an industry that already has too few skilled practitioners.
Are You Trapped?
Try this simple experiment on your own organization: Take some pen-test reports and calculate the average number of vulnerabilities you find per month (vIn). Next, search your risk register or bugtracking system and calculate the average number of vulnerabilities that are closed each month (vOut). Now, you can calculate your organization's vulnerability flow (vFlow). If the vFlow is positive, you are getting farther and farther behind every month.
vFlow = vIn - vOut
Pen-testing approaches to application security create a very expensive way to gain assurance across your portfolio. On average, each vulnerability costs roughly $1,000 to discover using penetration testing and $4,000 to code, test and redeploy a fix. That means that a typical application with 20 vulnerabilities represents about $100,000 of accrued security debt. And that's *before* you consider the costs of any breaches. If your portfolio has 1,000 applications, the reactive approach is simply infeasible.
It's easy for leaders to become addicted to the metrics they get from their pen-test program, which can grow to dominate an organization's entire application security program.
Escaping the Pen-Test Trap
I've helped a number of companies break out of this reactive pen test trap and establish highly favorable vFlow rates. Rather than attempting to pen test every application, we identify the most important vulnerabilities and put specific programs in place to stamp them out across the entire application portfolio.
Here are a few key steps for IT security leaders to undertake:
- Focus: Focus on eliminating the most important risks to the business. Targeted surveys and analysis can identify the prevalence and severity of many common flaws, as well as the root causes of these problems. Focusing on these problems is the most effective way to reduce application security risk.
- Standardize: Set standards for how the organization will deal with these risks. Create architectural patterns, programming guidelines and other strategic controls. Support developers by giving them simple and strong security controls to meet these standards. Providing them these controls saves significant time and money and reduces risk.
- Simplify: Making security as simple as possible for developers is an effective way to reduce mistakes. Training developers on exactly how to build secure code and providing tool support is more effective than courses teaching hacking techniques. Development teams trained in this manner produce 70 percent less vulnerabilities than untrained teams.
- Verify: Create automated means to verify that the standards are being followed. Verifying that an application has followed a positive practice is far easier than attempting to prove that there are no mistakes. Simple tools can verify these designs and coding patterns.
Once you've done everything you can to secure an application, there is clearly value in having an independent pen testing, which can become your early-warning system for new classes of vulnerability to eliminate from your portfolio. However, until you've put some serious effort into securing your applications, pen testing will offer no value.
Jeff Williams is the co-founder of both OWASP and Aspect Security, a consulting company focused on providing application security.