Euro Security Watch with Mathew J. Schwartz

Governance & Risk Management , Risk Assessments , Security Operations

Is CREST Penetration-Testing Certification Being Gamed?

Leaked Materials Prompt Questions About Test Integrity and Access to Exam Questions
Is CREST Penetration-Testing Certification Being Gamed?
Training documents posted to GitHub, which have been removed

This blog post has been updated with comments from CREST (Aug. 15).

See Also: The External Attack Surface Is Growing and Represents a Consistent Vulnerability

Who watches the penetration-testing testers?

Pen testing refers to simulating real-world attacks against systems, an approach many organizations use to identify vulnerabilities and weaknesses in need of remediation.

But questions are circulating over how some organizations train their employees for the pen-testing exam required to achieve certification from CREST - the Council for Registered Ethical Security Testers - after some leaked, internal training documents appeared to contain legitimate material from past tests.

CREST is a U.K.-based nonprofit organization, operating internationally, that offers accreditations for organizations as well as professional-level certifications for individuals on a number of fronts, including penetration testing, cyber incident response, threat intelligence and security operations center services. The organization's pen-testing qualification is the CRT, which stands for "CREST Registered Tester."

"CREST provides the confidence that penetration testing, threat intelligence and cyber incident response services will be carried out by qualified individuals with up-to-date knowledge, skills and competence, supported by a professional services company with appropriate data handling processes, quality assurance policies and technical methodologies," CREST's website states. "It also provides an independent complaints process, tied to the company and individual codes of conduct."

On Tuesday, however, The Register reported that a suspicious-looking leak of documents that showed up on GitHub appears to have originated with NCC Group, an information assurance firm in Manchester, U.K. The company, which has more than 15,000 clients, trades on the London Stock Exchange.

"The documents, posted to [GitHub] by an account set up last month, were held in a folder marked 'cheatsheets.' They appeared to be a collection of exceptionally frank and well-informed training materials," The Register reported.

Sample of materials uploaded to GitHub, some of which may have come from internal NCC Group training documents (Source: The Register)

"Some content was posted by an unidentified individual on GitHub that appears to be training material from NCC Group for CREST examinations," a CREST spokeswoman confirms to me, adding that "NCC are cooperating fully with CREST" to investigate exactly which types of material have been posted, as well as bigger-picture questions.

The materials posted to GitHub were contained in a top-level folder called "CREST-Exam-Prep" created by the "jaffarahmed" account and labeled as being "cheatsheets and write ups for the CREST CRT and CCT Exams."

Their posting - apparently on Sunday - drew a response from NCC Group CISO Dominic Beecher, who on Monday requested via a GitHub message that whoever had uploaded the documents contact him immediately "to avoid further formal action."

Post to GitHub on Aug. 10, 2020, by NCC Group CISO Dominic Beecher

The uploaded materials have been removed from GitHub, although apparently not before being "forked" - meaning copies were made.

An NCC spokeswoman tells me: "NCC Group is aware that an individual has posted material relating to CREST examinations on Github. The content appears to be a combination of old NCC Group internal training materials and content that has either been incorrectly attributed to NCC Group or which is unconnected to NCC Group. We take our membership of CREST, the integrity of the CREST Code of Conduct and our related obligations very seriously and comply with our obligations as a CREST member. We are currently reviewing the materials that have been posted and are working closely with CREST."

NCC Group is one of the founding members of CREST.

Are Organizations Allowed to Use Test Rigs?

I have asked CREST a number of questions. My inquiries include: Are organizations such as NCC Group allowed to create test rigs that emulate the exam for students? Can they provide invigilators for the CREST exam for students at their company? If so, what guarantees or rules are in place to ensure that doesn't get abused? In addition, did any material contained in the data leak break any contractual obligations NCC Group may have had with CREST?

To the question of whether a company employee can serve as an assessor for an exam being sat by the company's own candidates, CREST's spokeswoman tells me that the organization ensures that never happens: "The schedule is checked when the candidates are booked and the assessors are assigned accordingly."

She has promised to shortly answer all of my other questions in full. But in the meantime, she says that CREST is investigating the leak, and that while its investigation is still proceeding, the organization's panel of assessors has reviewed the leak and found that the information falls into three categories:

  • Internal training material: "Content which is not judged to relate to examination content and appears to be internal training material." CREST says this material will be reviewed in full to ensure it complies with CREST's policies.
  • Current/recent exam content: "Content which relates to current or recent past examination content, whose purpose appears to be to instruct others in passing the examination rather than providing technique instruction in a more general sense." CREST says NCC will be providing it with a written response with further details about these materials.
  • Exam guidance: "Content which makes unsubstantiated suggestions including that further examination content exists, whose purpose appears to be to instruct others in passing the examination." CREST says that NCC will also provide it with a written response with further details about this content, and that "the route that the investigation takes, and the consequential next steps, will depend on the responses that are provided."

"CREST will be appointing an independent panel to investigate the case including the extent to which NCC were aware, or should have been aware, of the content of their training material. For the avoidance of doubt, the CREST GB [Great Britain] chair, Mark Turner (NCC Group) has recused himself from any involvement in this investigation," she tells me.

"CREST takes breaches of its non-disclosure agreements very seriously and expects high standards of ethical behavior from both its member companies and those holding CREST qualifications. CREST will take appropriate action once its investigation has been completed," she says, adding that additional updates will be released as soon as CREST has them.

Further Twist: There's a RAT

In a further twist, The Register reports that at least some of the leaked materials may have been designed to hack whoever attempted to access them.

"At least some of the files in the repo also appeared to be connecting to a domain called canarytokens-dot-net when opened," The Register reports. "VirusTotal entries shown to us suggested that one file was loading something that registered with two detection engines as a generic remote access trojan; however, the canarytoken website appears to be a freely available honeypot-style file tracking token designed to phone home once a file including them was opened."

Bigger-Picture Questions

One of the bigger-picture questions prompted by NCC Group's internal training materials is whether they violate the code of conduct of CREST's certification system. This is one of the questions that CREST has pledged to investigate, once it receives full responses from NCC Group.

Industry watchers say having CREST pen-testing certifications can be mandatory for winning contracts in Britain, Australia and Singapore.

For a business such as NCC Group, ensuring that they have employees with the right skills to pass these exams makes good business sense. And not surprisingly, NCC appears to have devoted substantial resources to helping employees pass.

One employee review of NCC posted to Glassdoor, for example, claims: "Good training program, especially the internal CREST workshops and virtual labs that train you to pass CRT and CCT [Crest Certified Tester] the first time."

But are employees being trained with the skills they require to pass the exams? Or are they being coached to answer the questions correctly, perhaps without understanding the fundamentals?

Where Does Practice Exam Information Come From?

To that end, I posed these questions to NCC:

  • "Do you use test rigs that emulate the exam for students?
  • If so, is that explicitly allowed by CREST? And where does information contained on the practice exams come from?"

While NCC provided me with a statement on the material posted to GitHub material leak, it did not respond directly to these training and certification questions.

In the interest of fairness and baselining industry current practices, I've posed those same training questions to several other large organizations that offer CREST-certified penetration-testing services: Accenture Security's Context, MDSec and Nettitude Group.

I'll update this post with any comments I receive.

In the meantime, CREST's ongoing investigation, once concluded, also promises to shed further light on these questions.


Aug. 15: Blog post updated with comments from CREST.

About the Author

Mathew J. Schwartz

Mathew J. Schwartz

Executive Editor, DataBreachToday & Europe, ISMG

Schwartz is an award-winning journalist with two decades of experience in magazines, newspapers and electronic media. He has covered the information security and privacy sector throughout his career. Before joining Information Security Media Group in 2014, where he now serves as the executive editor, DataBreachToday and for European news coverage, Schwartz was the information security beat reporter for InformationWeek and a frequent contributor to DarkReading, among other publications. He lives in Scotland.

Around the Network

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing, you agree to our use of cookies.