Penetration Testing: 5 Critical Questions Before You Buy

Penetration testing is the security purchase most often made for the wrong reason. It gets ordered because a client questionnaire asked for one, or because a board paper needed a line item, and what arrives is frequently an automated scan with a covering letter. This is what the product actually is, the five questions worth asking before you sign, and the case for spending the money somewhere else.

Every definition and figure below comes from a primary document read while writing this: the NCSC’s own guidance, NIST SP 800-115, the Cyber Essentials Plus test specification, two live cyber insurance application forms, and the UK government’s 2025/2026 breaches survey. Where a number could not be traced to a source, this post says so rather than borrowing one.

What penetration testing actually is

The NCSC’s guidance gives a definition worth quoting exactly, because almost every vendor page paraphrases it into something softer: “A method for gaining assurance in the security of an IT system by attempting to breach some or all of that system’s security, using the same tools and techniques as an adversary might.”

The next sentence is the one that should change how you buy. Penetration testing, the NCSC says, “should be viewed as a method for gaining assurance in your organisation’s vulnerability assessment and management processes, not as a primary method for identifying vulnerabilities”.

Read that twice. The test is not there to find your vulnerabilities. It is there to check whether the process you already run finds them. The guidance makes the analogy explicit: a test is like a financial audit, where your finance team tracks income and expenditure day to day and an external group confirms that the internal process is sound.

Which produces the line that inverts the entire sales pitch: “In an ideal world, you should know what the penetration testers are going to find, before they find it.” A report full of surprises is not a good result. It is evidence that nothing internal was looking.

That guidance was published in August 2017 and last reviewed in January 2022. The tooling has moved since; the commissioning advice has not needed to.

Question 1: is this a scan, a test, or a certificate?

Three different products get sold under overlapping language, at very different prices, answering very different questions. Blurring them is commercially convenient, so it happens constantly.

A vulnerability scan is automated. A tool compares what it can see against a database of known issues and produces a list. It is broad, cheap, repeatable and should be running continuously rather than annually. OWASP’s Web Security Testing Guide is blunt about the ceiling: automated tools are “necessary and valuable, but they are insufficient on their own”, they excel at “known, signature-based vulnerabilities”, and “they do not understand business logic or context”.

Penetration testing is a person. The value is in chaining things a scanner reports separately, or does not report at all — a low-severity information leak plus a weak default plus an over-permissioned service account becoming domain access. NIST SP 800-115 describes it as looking for “combinations of vulnerabilities on one or more systems that can be used to gain more access than could be achieved through a single vulnerability”.

Cyber Essentials Plus is neither. It is an audited verification that five specific controls are configured, and it is the one most often mis-sold as a test. We read the Cyber Essentials Plus test specification, v3.2, April 2025 end to end. It contains five test cases: a remote vulnerability assessment, an authenticated scan to check patching, a malware protection check, a multi-factor authentication check, and an account separation check. The word “penetration” does not appear in it once. Nor does it appear anywhere in the current Cyber Essentials requirements, v3.3, April 2026.

The purpose of its first test case sets the bar precisely: “To test whether an Internet-based opportunist attacker can hack into the Applicant’s system with typical low-skill methods.” That is a deliberately modelled low-skill opportunist. Penetration testing models a motivated one who has time. Both are useful; they are not substitutes, and anyone selling you one as the other is telling you something about themselves.

A version mismatch worth knowing about: the Cyber Essentials requirements are at v3.3 (April 2026), while the newest Plus test specification we could retrieve from the NCSC is v3.2 (April 2025). If certification matters to you, confirm the current test specification with your certification body rather than assuming the version numbers move together.

Question 2: what is in scope, and who signed it off?

Scope is the entire product. A test against three IP addresses and a test against your whole external estate carry the same invoice line and answer completely different questions.

The NCSC is specific about who belongs in the scoping conversation: all relevant risk owners, technical staff who know the target system, and a representative of the test team. Not just whoever owns the budget.

The output of that conversation should be a written plan stating the technical boundaries of the test, the types of test expected, the timeframe and effort in resource days, the testing team’s own requirements, any compliance obligations the plan must meet, and any specific reporting requirements such as CVSS scores or CHECK severity levels. NIST SP 800-115 calls the same artefact the Rules of Engagement and devotes an appendix to a template for it.

Two clauses are worth insisting on. The first is a technical point of contact available at short notice throughout, so the team can raise a critical finding immediately rather than in a report six weeks later. The second is what happens when the testers find something adjacent to but outside the agreed scope — the NCSC describes the two honest options as changing the scope, which changes time and cost, or recording the exclusion as a documented limitation on testing. Either is fine. Silence is not.

There is also a real risk to accept openly. NIST notes that systems “may be damaged or otherwise rendered inoperable during the course of penetration testing”, and that while experienced testers reduce that risk, “it can never be fully eliminated”. That is an argument for out-of-hours windows and named contacts, not for skipping the work.

Question 3: what will this test not tell you?

This is the question vendors are least keen on, and the primary sources answer it without hedging.

A test, the NCSC says, “can only validate that your organisation’s IT systems are not vulnerable to known issues on the day of the test”. It adds that “it’s not uncommon for a year or more to elapse between penetration tests”, so vulnerabilities can exist for long periods without you knowing, if that is your only means of validating security.

NIST puts the structural limit plainly: “Testing does not provide a comprehensive evaluation of the security posture of an organization, and often has a narrow scope because of resource limitations—particularly in the area of time. Malicious attackers, on the other hand, can take whatever time they need to exploit and penetrate a system or network.”

Three practical consequences fall out of that.

A clean report is not assurance. It is a statement that a specific tester, working a specific number of days, against a specific scope, did not achieve a specific objective. Everything outside those four boundaries is untested and the report should say so.

Penetration testing is not a compensating control for patching. The result decays from the day it is signed. If your patch window is measured in months, an annual test tells you about a machine that no longer exists in that state.

It is the wrong tool for confirming that controls work. The NCSC states it directly: “Assessing whether defined security controls are functioning is not a valuable use of penetration testing resources.” Functional testing belongs in your own regime. It also says that for product-specific testing, penetration testing is not an appropriate technique at all.

Question 4: what must the report contain?

Penetration testing is, in the end, the purchase of a document. Specify it in the contract, because the gap between a good report and a tool export with a logo on it is where most of the disappointment lives.

The NCSC lists five things a report should include: the security issues uncovered, an assessment of the level of risk each one exposes you to, a method of resolving each issue, an opinion on the accuracy of your organisation’s own vulnerability assessment, and advice on how to improve your internal vulnerability assessment process.

The last two are the ones routinely missing, and they are the two that follow from the definition. If penetration testing exists to assure your vulnerability management process, a report that never mentions that process has not delivered the product.

On severity, CHECK reports are required to state risk as HIGH, MEDIUM, LOW or INFORMATIONAL, and CVSS may be used in addition to that scale but not in place of it. Any deviation from a vulnerability’s standard rating should be documented and justified by the testers.

Then the part buyers most often skip. The NCSC is explicit that “risk assessment and decisions on the application of fixes are your responsibility” — testers may rate an issue higher or lower than you would, because they lack your business context — and that “vulnerability risk assessment and mitigation is a business process and should not be wholly outsourced to the test team”. Their suggested fix is also not the only fix: uninstalling unused software, or accepting a risk with additional monitoring, may beat the patch they recommended.

Question 5: is this the right thing to buy first?

The unprofitable answer, which is usually the true one for a business buying its first test: no.

If multi-factor authentication is not enforced on every account, if nobody has restored from a backup this year, if you cannot name the end-of-life software on the network, then you do not need penetration testing to learn that you are exposed. You already know. You will pay a skilled person several days’ fees to write it down, and the report will be a list of things a free afternoon would have found.

That order — assessment first, purchase second — is the argument our cybersecurity risk management work is built on, and it is not a preference. It follows from the NCSC’s own model, which assumes you already have an internal vulnerability assessment and management process for the test to audit. Without one, there is nothing to audit.

The sequence we would follow is: close the identity gaps, prove the backups restore, establish a patch window with a number attached to it, then buy a test to check whether the process you have built actually works. Our guide to cyber security for small business walks through nine controls you are probably already paying for, with the admin centre to check each one in. If you cannot yet answer those nine, the test can wait.

Two more cases where penetration testing is the wrong purchase. If you want to know whether staff will click a link, that is a phishing simulation and a training programme, not a technical test. And if nobody in the business has the time or authority to fix what comes back, buying the report is buying a document that will make an incident worse, not better — because you will have written proof you were told.

Who actually requires penetration testing, and who only seems to

The market talks as though everyone demands it. The evidence is thinner than that.

The UK government’s Cyber Security Breaches Survey 2025/2026, covering 2,112 businesses, found 13% had carried out penetration testing in the previous twelve months. A vulnerability audit was more common at 18%, testing staff with mock phishing more common again at 22%, and a risk assessment covering cyber security most common of the four at 30%. Just over half of businesses, 52%, had done any of the listed activities at all.

Cyber insurance is the requirement most often asserted and least often checked. We read both of Beazley’s published cyber applications — the form for applicants under $250M in revenue and the longer one above it — and neither asks a single question about penetration testing. Not one. They ask about MFA on remote access and email, privileged accounts, email filtering, endpoint protection, backup location, restore-test frequency, patching internet-facing systems, end-of-life software, exposed remote desktop, incident response plans and out-of-band callbacks for changes to bank details.

Our walk-through of the controls underwriters check covers those twelve in full, and penetration testing is absent from that list for the same reason it is absent from the forms: underwriters price the controls their claims data implicates, and a point-in-time test is not one of them.

PCI DSS is the genuine mandate people are half-remembering, and it is where we have to be honest about a limit. The current standard sits behind a click-through licence on the PCI Security Standards Council’s site and returns a 403 to a direct fetch from a server, so we could not read the requirement text during this run — and we are not going to quote a requirement number from memory.

The council does openly publish an information supplement on penetration testing guidance, and it is a real primary document: 43 pages on scoping, methodology, qualifications and retesting. But it is dated March 2015 and written against Requirement 11.3 under the older version of the standard, which was renumbered in PCI DSS v4. It is still useful on method and unsafe on obligation.

So if card data is in scope for you, get the current standard from the council directly and read the requirement yourself, rather than trusting anyone’s summary of it — this post included.

The remaining driver is real and rarely named: enterprise client questionnaires. If a customer contract requires an annual test, that is a commercial obligation, and it is a perfectly good reason to buy one. It is just a different reason from believing it will make you secure.

What it costs, and why we are not publishing a number

Search for a price and you will find confident ranges. Follow any of them back and they terminate in another vendor’s blog post. We could not trace a single published figure to measured data, so we are not going to repeat one.

What we can describe is the shape of the estimate, because the NCSC names the unit: scoping produces “the timeframe and the amount of effort necessary to deliver the testing — usually given in terms of resource days”. Penetration testing is priced in skilled days, so the only quote worth comparing is one that states how many days, at what seniority, against what scope, with reporting and a debrief included or excluded explicitly.

Which means two quotes with the same headline price can differ by a factor of three in what they deliver, and the difference is visible only in the scoping document. Compare days and scope, never totals.

On accreditation: CREST accreditation requires companies to submit evidence covering company processes, service methodologies and data security practices, and to undergo periodic reassessment. That is meaningful — it assures how the firm operates and handles your data. It is a statement about the company, not a guarantee about the individual assigned to your job, and the NCSC notes the quality of a test “is closely linked to the abilities of the penetration testers involved”. Ask who is doing the work. In the UK public sector the equivalent signal is the NCSC’s CHECK scheme, which it recommends for government organisations.

Where we would start instead

If you have read this far hoping for permission to skip the test, that depends entirely on what you already have.

Build the process first. A current asset inventory, continuous vulnerability scanning with someone named as its owner, a patch window expressed in days rather than adjectives, and a restore you have actually performed. Our piece on what a restore drill exposes covers the last of those, and it is the one most often assumed rather than evidenced.

Then buy a penetration testing engagement, and treat the report the way the NCSC intends — as an opinion on whether that process works. Anything the testers found that you did not already know is the actual finding, and the question to ask about it is not “how do we fix this” but “why did our own process not see it”.

Penetration testing is a good product bought at the wrong point in the sequence more often than it is a bad product. It is expensive, it is a snapshot, and it audits a process rather than replacing one. Buy it when you have something worth auditing, scope it in writing with the risk owners in the room, and specify the report before you specify the price.