NIST SP 800-115: the four phases, and the one that is not a phase

It is the Technical Guide to Information Security Testing and Assessment, published by the US National Institute of Standards and Technology in September 2008. It is free, it is about eighty pages, and it is the document most credible testing methodologies are recognisably built on — which is why a report that follows its shape is legible to an auditor who was not in the room.

Its scope is broader than penetration testing. It defines three assessment methods and the distinction is worth carrying: testing exercises an object under specified conditions to compare actual against expected behaviour; examination is checking, inspecting, reviewing or analysing to obtain evidence; interviewing is conducting discussions with people. A penetration test is one method among three, and a great many of the questions an assessor needs answered are cheaper to answer by examination or interview than by attacking something.

Where the OWASP guides tell you what to test inside an application, SP 800-115 tells you how an assessment should be run at all. The two are complements rather than alternatives, and a good proposal cites both.

This is the detail that almost every summary of the document gets wrong, and it is the one most worth knowing. NIST states plainly that the reporting phase occurs simultaneously with the other three. The assessment plan — the rules of engagement — is developed during planning. Through discovery and attack, written logs are kept and periodic reports are made to system administrators and management. The document at the end is the last artefact of reporting, not the whole of it.

Read that way, the four-stage figure stops being a waterfall and becomes something more useful: three phases of work, with a thread of documentation running the length of all of them.

It also explains a failure you may have seen without having a name for it. A report whose findings cannot be reproduced, whose evidence is thin, or which cannot say exactly when something was tested, is usually a report that was written at the end from memory. The methodology anticipates that and says: do not do it that way. When you are assessing a test you have already bought, ask when the notes were written. The answer is diagnostic.

SP 800-115 gives the clearest statement of this distinction in the literature, and it is worth quoting the substance: vulnerability scanners check only for the possible existence of a vulnerability, whereas the attack phase of a penetration test exploits it to confirm that it exists.

The mechanism that makes that possible is the loopback. Exploits frequently do not grant maximum access at the first attempt; they tell the tester more about the environment, or change its state, or escalate privileges enough to see resources that were previously invisible. When that happens, the methodology sends the tester back into discovery against the new vantage point — which is why Figure 5-2 is titled for its loop back to the discovery phase, and why the feedback arrow between attack and discovery is the most important line in the whole diagram.

That loop is also the thing a scanner cannot do and an automated report cannot fake. It is why a real test finds the chain — the low-severity disclosure that yields a credential, that grants a foothold, that reaches the database — and a scan finds three unrelated medium findings and misses the path between them. When you read a report, look for evidence that the tester went round the loop at least once. If every finding is free-standing and none builds on another, you are likely holding a scan.

Appendix B of SP 800-115 is a rules of engagement template, and it is the single most immediately useful thing in the document — free, public, and directly usable as a checklist against any proposal you have been sent.

It asks for purpose and scope; assumptions and limitations; the inherent risks of testing and how they will be mitigated. Then logistics: named personnel on both sides with points of contact for the test team, management and the incident response team; the test schedule, including the hours during which testing will run, since testing an operational site in the evening is often the prudent choice; the sites testing is authorised from, physical and equipment access, and the areas the team will not be given access to; and the test equipment, including how your systems will be told apart from the testers’.

Then the two sections organisations most often discover they needed only afterwards. The target system section requires identifying not just what is in scope but what is explicitly not — NIST calls this the exclude list, and naming it in advance is how the production database survives the engagement. And incident handling and response must state the criteria for halting testing, what the team does if a test degrades the network, what happens if a real adversary attacks while testing is underway, your incident response chain of command in quick-reference form, and the process for reinstating the team afterwards.

Every one of those is cheaper to decide before the test than at two in the morning during it. If a proposal does not cover them, the template gives you the language to ask.

Parts of it have aged exactly as you would expect. It has an appendix on Live CD distributions for security testing. It discusses NetBIOS enumeration and Network Information System queries as information-gathering techniques. Nobody should take their tooling advice from it, and the technical specifics are the weakest part of the text.

What has not aged is the structure — the phases, the loop, the rules of engagement, the insistence that reporting is continuous and that authorisation is documented before anything begins. Those are properties of how assessment work is organised rather than of any technology, which is why the document remains Final rather than withdrawn and why later methodologies are recognisably descended from it.

So use it for the shape and not the tools. Pair it with a current technical methodology — the OWASP Web Security Testing Guide for web applications, the API Security Top 10 for APIs, the Mobile Application Security Testing Guide for mobile — and you have both halves: how the engagement is run, and what gets tested inside it.

ISO/IEC 27001 and SOC 2 both expect technical testing on a defined cycle, and both assess it as evidence rather than reading it for the findings. Evidence has properties an assessor checks: it identifies its scope, its method, its dates, who performed it and what they found — and it shows those findings being closed.

This is the practical reason the shape matters more than the severity counts. A report structured the way SP 800-115 describes answers those questions on its own, without anyone having to reconstruct the engagement from an inbox. An email saying the test went fine answers none of them, however true it happens to be.

And the piece most often missing is the last one. The retest letter — the document confirming that findings were fixed and verified — is what closes the loop for an auditor. Buying a test without a retest is buying the half of the evidence that shows you had problems, and not the half that shows you dealt with them.