Quick answer: A vulnerability assessment uses largely automated scanning to find and rank known weaknesses; a penetration test has qualified testers exploit and chain weaknesses to prove what an attacker can actually reach. Of the six frameworks covered here, only PCI DSS v4.0.1, NYDFS 23 NYCRR 500.5 and CMMC Level 3 name both as separate obligations, while SOC 2, ISO/IEC 27001:2022 and the HIPAA Security Rule in force today mandate neither method by name. Auditors still expect both artefacts: scan output to evidence vulnerability management, and a penetration test report with a retest to evidence that controls actually work.
The one-line difference: a vulnerability assessment tells you what might be exploitable, a penetration test tells you what is.
That difference decides which one satisfies an auditor, which one belongs in your release pipeline, and which one you should be buying this quarter. For the basics of what a test is and how one runs, start with our guide to penetration testing.
Definitions
Vulnerability scan: The automated tooling step inside a vulnerability assessment. A scan is one input to an assessment. Neither is a penetration test.
Vulnerability assessment (VA): A broad, largely automated review that discovers, catalogues and prioritises known security weaknesses across a defined set of systems, without confirming that any of them can be exploited. Output is a ranked inventory of potential issues.
Penetration test (PT): An authorised, objective-driven simulated attack in which skilled testers actively attempt to exploit weaknesses, chain them together, and demonstrate real business impact. Output is a set of verified findings with proven attack paths.
Vulnerability assessment vs penetration testing: side-by-side
Dimension | Vulnerability assessment | Penetration test |
|---|---|---|
Core question | What could be wrong here? | What can an attacker actually do here? |
Method | Automated scanning, configuration review, version and patch checks | Manual tradecraft plus tooling: reconnaissance, exploitation, privilege escalation, chaining |
What it finds | Known, catalogued weaknesses with a signature: missing patches, outdated components, misconfigurations, exposed services | The same classes where they are exploitable, plus authorisation, session and business logic flaws that have no signature |
Exploitation | None. Findings are inferred from versions, banners and responses | Yes, inside agreed rules of engagement, to prove impact and chain findings |
False positives | Common, an expected cost of automation; every result needs triage | Rare when every finding needs reproduction evidence: 9 of 1,216 logged findings (0.74%) were declined at review in Stingrai's data |
Cadence and duration | Continuous, monthly or quarterly; minutes per host. PCI DSS sets at least once every three months | Days to weeks per test, annually and after significant change (PCI DSS, NYDFS), or continuously through a programme |
Cost model | Recurring tool subscription, usually per asset or per IP, often bundled into a platform you already own | Priced per engagement by scope, or as fixed packages for defined scope (see the cost section below) |
PCI DSS v4.0.1 | Requirement 11.3: internal scans (11.3.1) and external ASV scans (11.3.2) | Requirement 11.4: methodology (11.4.1), internal (11.4.2), external (11.4.3), correct and retest (11.4.4) |
SOC 2 evidence | CC7.1, whose point of focus is "Conducts Vulnerability Scans" | CC4.1 separate evaluations, whose point of focus names penetration testing |
Penetration testing vs vulnerability scanning
A vulnerability scan is not a penetration test, and the PCI Security Standards Council draws the line in a single table. Its Penetration Testing Guidance information supplement (version 1.1, September 2017) gives the purpose of a scan as to "identify, rank, and report vulnerabilities", and the purpose of a test as to "identify ways to exploit vulnerabilities to circumvent or defeat the security features of system components". It describes a scan as automated tools combined with manual verification, and a penetration test as "a manual process that may include the use of vulnerability scanning or other automated tools". The scan is one input to the test, never a substitute for it.
Three practical differences follow.
Reachability. A scanner reports that a component is affected by a known vulnerability. It does not establish that the vulnerable code path is reachable in your deployment, that exploitation succeeds against your configuration, or that the result is worth anything to an attacker. Authenticated scanning, required for PCI DSS internal scans since 31 March 2025, reads versions from inside the host and cuts noise, but it still does not prove exploitability.
Chaining. A verbose error message is noise. A predictable identifier is noise. An overly permissive role is noise. Combined in the right order they become a full account takeover, and only a tester following the path end to end will report it as one critical finding rather than three low-severity notes nobody prioritises.
Logic. Scanners match against catalogues, so they cannot find flaws that exist only in your own application: broken authorisation between tenants, an insecure direct object reference, a workflow that can be replayed to issue a refund twice. Those bugs have no CVE to match against.
A product that runs automated scans and calls the output a penetration test will not satisfy PCI DSS Requirement 11.4, nor a CMMC Level 3 assessor: the Level 3 requirement itself calls for "automated scanning tools and ad hoc tests using subject matter experts".
What 1,206 verified findings say scanners can and cannot find
Stingrai's State of Penetration Testing 2026 analysed 1,206 verified findings from 55 penetration tests run between October 2024 and August 2026, and sorted every vulnerability class by whether a signature could in principle represent it.

Bucket | Classes | All findings (n = 1,206) | High and Critical (n = 810) |
|---|---|---|---|
Scanner-shaped | Outdated or unpatched software, misconfiguration and hardening, exposed services and default access, denial of service and rate limiting | 449 (37.2%) | 317 (39.1%) |
Context-dependent | Authentication and session, broken access control, IDOR and object-level authorisation, business logic | 425 (35.2%) | 279 (34.4%) |
Neither bucket | Injection, client-side execution, information disclosure | 332 (27.5%) | 214 (26.4%) |
Read it in both directions. Scanner-shaped classes were 39.1% of High and Critical findings, so skipping continuous scanning means discovering two fifths of your severe issues once a year at penetration test prices. Context-dependent classes were 34.4%: roughly one severe finding in three depends on what the product is supposed to permit, which no signature encodes and no vulnerability assessment will report. "Scanner-shaped" describes the class, not the finding: a signature could exist for that category, not that a given tool would have caught that instance.
False positives are the other half. Nine of 1,216 logged findings were declined at review, 0.74%, because a finding only exists once a penetration tester has reproduced it with evidence, and it reaches the client after review by a team lead and then the engagement partner. Scanner output has no such gate, so the triage cost lands on your team.
Which one do you need?
Most organisations eventually need both. The real question is sequence.
Your situation | Start with | Why |
|---|---|---|
No security testing at all today | Vulnerability assessment | Establish asset inventory and clear the known, patchable backlog before paying testers to rediscover it |
Enterprise prospect is blocking a deal on a security review | Penetration test | Buyers ask for a test report and a remediation letter, not scan output |
Preparing a first SOC 2 Type 2 | Both, test scheduled inside the observation period | Scanning evidences your vulnerability management control; the test evidences control effectiveness under CC4.1 |
In scope for PCI DSS | Both, they are separate requirements | 11.3 scanning and 11.4 testing are independent obligations; neither substitutes for the other |
Regulated by NYDFS | Both | 500.5(a)(1) requires an annual penetration test and 500.5(a)(2) requires automated scans at a risk-based frequency |
Shipping application changes weekly | Continuous scanning plus application testing tied to release | An annual point-in-time test cannot cover 50 releases a year on its own |
Custom application handling money or sensitive records | Penetration test | Business logic and authorisation flaws are the dominant risk and scanners do not find them |
Post-incident | Penetration test | You need to know what an attacker can reach now, not what is theoretically unpatched |
The general rule: use vulnerability assessment for coverage and penetration testing for certainty.
What compliance frameworks actually require
Most published comparisons go wrong here, usually by repeating clause numbers retired years ago. This is the current position, quoted from the primary text.
Framework | Penetration test | Vulnerability scanning | What the auditor or assessor looks for |
|---|---|---|---|
PCI DSS v4.0.1 | Required. 11.4.2 internal and 11.4.3 external, at least once every 12 months and after any significant upgrade or change | Required. 11.3.1 internal at least once every three months, authenticated under 11.3.1.2; 11.3.2 external by an ASV at least once every three months | A test run to your documented 11.4.1 methodology, proof that exploitable findings were corrected and retested (11.4.4), and passing quarterly ASV scans |
NYDFS 23 NYCRR 500.5 | Required. At least annually, from inside and outside the system boundaries, by a qualified internal or external party | Required. Automated scans at a frequency set by the risk assessment and promptly after material system changes, in force since 1 May 2025 | The annual test report, scan records, and remediation prioritised by risk |
SOC 2 | Not mandated. Named once, in a CC4.1 point of focus | Not mandated. CC7.1's point of focus is "Conducts Vulnerability Scans" | Whatever your own control description promises, tested against that wording |
ISO/IEC 27001:2022 | Not named in any clause or Annex A control title | Not named. A.8.8 covers management of technical vulnerabilities | Evidence that the controls your risk treatment and Statement of Applicability selected are operating |
HIPAA Security Rule (in force) | Not named | Not named. Risk analysis and periodic evaluation are the obligations | A documented risk analysis and periodic technical evaluation |
CMMC Level 2 | Not named. "Penetration" does not appear in NIST SP 800-171 Rev 2 | Required. 3.11.2: scan periodically and when new vulnerabilities are identified | Scan records and remediation tied to your system security plan |
CMMC Level 3 | Required. CA.L3-3.12.1e, at least annually or when significant security changes are made | Required inside the same requirement, alongside expert-led ad hoc tests | A test that combined automated tooling with subject matter experts |
Status as of 25 October 2026
Framework | Current text | What moved in 2026 |
|---|---|---|
PCI DSS | v4.0.1, published June 2024 with no added or deleted requirements; v4.0 retired 31 December 2024 | PCI SSC ran a request for comments on v4.0.1 from 3 June to 20 July 2026 to inform the next iteration. No new version has been published |
HIPAA Security Rule | The rule at 45 CFR Part 164, Subpart C | The January 2025 proposal is still only a proposal. The Unified Agenda now lists it under Long-Term Actions with final action dated July 2027 |
CMMC | 32 CFR Part 170, with Level 2 identical to NIST SP 800-171 Rev 2 | The Department of War suspended Phase II, due 10 November 2026, on 13 July 2026. Phase I self-assessment requirements remain in place |
NYDFS Part 500 | Second Amendment, effective 1 November 2023 | Nothing new for 500.5; the automated scanning duty in 500.5(a)(2) has applied since 1 May 2025 |
ISO/IEC 27001 | 2022 edition, with Amendment 1:2024 | No new edition published |
SOC 2 | 2017 Trust Services Criteria, with revised points of focus (2022) | No change to the criteria |
PCI DSS v4.0.1: 11.3 is scanning, 11.4 is testing
Under PCI DSS v4.0.1, Requirement 11.3 is vulnerability scanning and Requirement 11.4 is penetration testing. They are separate requirements with separate clocks.
The framework table above gives the clocks. Three details are routinely missed. Internal scans under 11.3.1 need qualified personnel with organisational independence but not an ASV, and since 31 March 2025 they must be authenticated (11.3.1.2). The 11.4.1 methodology has nine required elements, including application-layer testing for the vulnerabilities listed in Requirement 6.2.4 and retention of results for at least 12 months. And segmentation testing applies only where segmentation is used to isolate the cardholder data environment: 11.4.5 sets every 12 months and after changes, and 11.4.6 tightens that to every six months for service providers.
Under PCI DSS v3.x penetration testing sat at 11.3, with segmentation testing at 11.3.4, so any page telling you in 2026 that "Requirement 11.3 requires annual penetration testing" is quoting a retired structure. The confusion is partly forgivable: the PCI SSC's Penetration Testing Guidance supplement is still version 1.1 from September 2017 and still refers to Requirement 11.3. Its methodology advice remains useful; its clause references do not.
The tester must be a qualified internal resource or qualified external third party with organisational independence, and is explicitly not required to be a QSA or ASV. Our PCI DSS Requirement 11.4 guide covers every sub-requirement, the nine methodology elements and the evidence a QSA expects.
SOC 2
SOC 2 does not require a penetration test. In the AICPA's 2017 Trust Services Criteria, penetration testing appears once, in a point of focus under CC4.1 saying management uses "a variety of different types of ongoing and separate evaluations, including penetration testing", independent certification and internal audit. Points of focus are illustrative: the criteria state that their use "does not require an assessment of whether each point of focus is addressed".
Auditors still expect testing evidence. CC4.1 requires separate evaluations of whether controls are present and functioning, and a penetration test is the one most service organisations choose. Your binding obligation is the sentence in your own control description: write "annual penetration test" and the auditor will test you against annual.
One trap: CC7.1 is not the pentest criterion, though it is routinely cited as one. Its point of focus, "Conducts Vulnerability Scans", describes scans "on a periodic basis and after any significant change in the environment". Offer a test report against a CC7.1 request and expect to be asked for scan output. Our SOC 2 penetration testing guide covers timing inside a Type 2 observation period.
ISO/IEC 27001:2022
ISO/IEC 27001:2022, still the current edition with its 2024 amendment, never names penetration testing in any clause or Annex A control title, and sets no testing frequency anywhere. Controls flow from your own risk treatment under clause 6.1.3, and Annex A is a cross-check rather than a mandatory list.
The old A.12.6.1 reference still circulating belongs to the 2013 edition. In the 2022 edition the relevant control is A.8.8 Management of technical vulnerabilities, and its title carries no frequency. A.8.29 covers security testing in development and acceptance, which is event-driven rather than calendar-based. A.5.35, independent review of information security, is a governance activity often wrongly sold as proof that ISO demands a third-party technical test. The "annual ISO pentest" idea comes from the certification body's surveillance audit cycle, which sets when the auditor visits, not how often you test. Our ISO 27001 penetration testing guide covers scope and the evidence auditors ask for.
HIPAA
The HIPAA Security Rule in force today does not use the word "penetration" anywhere. The obligations are a risk analysis under 45 CFR 164.308(a)(1)(ii)(A), "an accurate and thorough assessment of the potential risks and vulnerabilities" to electronic protected health information, and a "periodic technical and nontechnical evaluation" under 164.308(a)(8). Neither names a method and neither sets a frequency.
The proposed update would change that, and it keeps the two activities separate. The proposal published on 6 January 2025 (90 FR 898, RIN 0945-AA22) would add a vulnerability management standard at 164.312(h): automated vulnerability scans at least once every six months, or more often if the risk analysis says so, and penetration testing by a qualified person at least once every 12 months. Comments closed on 7 March 2025; as of 25 October 2026 there is no final rule, and the Unified Agenda lists the rulemaking under Long-Term Actions with final action dated July 2027. Treat annual testing and twice-yearly scanning as strong risk-based practice and a likely future obligation, not as current law.
NYDFS 23 NYCRR 500.5
New York's cybersecurity regulation names both activities outright. Section 500.5, retitled "Vulnerability management" by the Second Amendment, requires "penetration testing of their information systems from both inside and outside the information systems' boundaries by a qualified internal or external party at least annually", plus automated scans, with a manual review of systems the scans miss, at a frequency set by the risk assessment and promptly after any material system change. The scanning duty in 500.5(a)(2) has applied since 1 May 2025, and 500.5 also requires timely remediation prioritised by risk. Our NYDFS penetration testing guide covers scope and evidence.
CMMC
CMMC Levels 1 and 2 impose no penetration testing obligation. The word "penetration" does not appear in NIST SP 800-171 Rev 2 at all; the Level 2 controls mistaken for a mandate (RA.L2-3.11.2 scanning, CA.L2-3.12.1 periodic assessment, CA.L2-3.12.3 monitoring) name no method.
CMMC Level 3 contains CMMC's only explicit penetration testing mandate, CA.L3-3.12.1e: "Conduct penetration testing at least annually or when significant security changes are made to the system, leveraging automated scanning tools and ad hoc tests using subject matter experts." Scanning alone does not meet it. Timing has moved: on 13 July 2026 the Department of War suspended Phase II of CMMC implementation, which had been due on 10 November 2026, while Phase I self-assessment requirements remain in place. See our CMMC penetration testing guide for the current state, and our cross-framework evidence guide for what assessors accept under each framework.
Cost: subscription versus engagement
A vulnerability assessment is normally a recurring tool subscription priced per asset or per IP, often already bundled into a platform you own. A penetration test is priced per engagement by scope: application count, user roles, network size and the depth of business logic testing.
For one web application and associated APIs, Stingrai publishes fixed prices. The Autonomous Pentest (Snipe alone) is US$3,000 one-time or US$650 a month on a 12-month plan and delivers a pentest report. The Hybrid Pentest (Snipe and penetration testers testing together) is US$6,800 one-time or US$1,275 a month on a 12-month plan, and includes the attestation letter that comes only with Hybrid and human-led engagements. Network, cloud and Active Directory scope is human-led and quoted per scope. See pricing and our penetration testing cost guide.
How the two combine in a modern programme
Treating this as an either/or is the actual mistake. A serious programme layers three things.
Layer one: continuous vulnerability scanning. Authenticated scanning across the whole estate, feeding a remediation queue with an agreed service level by severity. It handles the known, patchable population. It is cheap and fast, and it should never be the thing you are proud of.
Layer two: testing in the pipeline. Snipe is Stingrai's autonomous AI penetration testing agent for web applications and associated APIs. Trained on more than 6,000 HackerOne Hacktivity disclosure reports and on our penetration testers' methodology, it hunts complex classes such as IDOR, broken authorisation and business logic, tests black-box and white-box against source, generates AutoFix pull requests, and can gate pull requests so vulnerable code never merges.
Layer three: penetration testing. In a Hybrid engagement, penetration testers and Snipe test the same application at the same time, with the testers directing where Snipe focuses and pursuing the chains that need judgement; network, cloud and Active Directory testing are human-led. The output is the report, evidence and retest an auditor or enterprise buyer will accept, delivered as a one-time annual penetration test or a continuous testing programme.
Scanning keeps the known population under control. Pipeline testing stops new application flaws from shipping. The penetration test proves what an attacker can actually reach.
Frequently Asked Questions
What is the difference between a vulnerability assessment and a penetration test?
A vulnerability assessment identifies and ranks known weaknesses across your systems using largely automated scanning, without confirming that any of them can be exploited. A penetration test uses skilled human testers to actively exploit weaknesses, chain them together and demonstrate real business impact. In one line: a vulnerability assessment tells you what might be exploitable, a penetration test tells you what is. Assessments are broad, fast and repeatable; tests are deep, targeted and produce verified findings with proven attack paths.
Is a vulnerability scan a penetration test?
No. A vulnerability scan is the automated tooling step inside a vulnerability assessment, and no compliance framework treats the two as interchangeable. PCI DSS separates them into Requirement 11.3 for scanning and Requirement 11.4 for penetration testing, and the PCI SSC's Penetration Testing Guidance describes a penetration test as "a manual process that may include the use of vulnerability scanning or other automated tools". The scan is one input to the test, never a substitute.
Do you need both, or can a vulnerability assessment replace a penetration test?
Most organisations need both, and a vulnerability assessment cannot replace a test. Scanners cannot find flaws unique to your own logic, such as broken authorisation between tenants, and cannot chain low-severity findings into a critical attack path; in Stingrai's data from 1,206 verified findings, 35.2% fell into those context-dependent classes. PCI DSS Requirement 11.4, NYDFS 500.5 and CMMC Level 3 also require testing explicitly. With no testing today, start with a vulnerability assessment to clear the known backlog, then run a penetration test.
Which is required for SOC 2?
Neither is mandated by name. Penetration testing appears once in the Trust Services Criteria, inside a non-binding point of focus under CC4.1, and there is no pentest criterion, control number or required frequency. In practice auditors expect testing evidence because CC4.1 requires separate evaluations of whether controls are functioning, and a test is the evaluation most organisations select. Scanning matters separately: the point of focus under CC7.1 is "Conducts Vulnerability Scans", so a test report is the wrong artefact to offer against a CC7.1 request.
Which is required for PCI DSS?
Both, as separate requirements. Requirement 11.3 mandates vulnerability scanning: internal scans at least once every three months under 11.3.1 and external scans by an Approved Scanning Vendor at least once every three months under 11.3.2. Requirement 11.4 mandates penetration testing: internal testing under 11.4.2 and external testing under 11.4.3, both at least once every 12 months and after any significant infrastructure or application upgrade or change, with exploitable findings corrected and retested under 11.4.4. Under PCI DSS v3.x penetration testing sat at 11.3, which is why outdated pages still cite that number.
Do ISO 27001, HIPAA and NYDFS require a penetration test?
ISO/IEC 27001:2022 does not name penetration testing in any clause or Annex A control title; A.8.8 covers technical vulnerability management and your risk treatment decides the method. The HIPAA Security Rule in force today does not name it either, although the January 2025 proposal would require penetration testing at least once every 12 months and scans at least once every six months; no final rule has been issued. NYDFS does require it: 23 NYCRR 500.5 requires an annual penetration test from inside and outside the system boundaries, plus automated scans at a risk-based frequency.
How much does each cost?
They use different pricing models. A vulnerability assessment is normally a recurring tool subscription priced per asset or per IP, and is often already bundled into a cloud platform or vulnerability management suite you own. A penetration test is priced per engagement by scope. For one web application and associated APIs, Stingrai's Autonomous Pentest is US$3,000 one-time or US$650 a month, and the Hybrid Pentest is US$6,800 one-time or US$1,275 a month, both on 12-month plans for the monthly price.
Related reading
The State of Penetration Testing 2026: An Analysis of 1,206 Verified Findings
PCI DSS Penetration Testing: Requirement 11.4 Explained (2026)
SOC 2 Penetration Testing: What Auditors Expect and How to Scope It (2026)
ISO 27001 Penetration Testing: Annex A Mapping, Frequency and Auditor Evidence (2026)
HIPAA Penetration Testing Requirements (2026): Security Rule, Scope, Evidence and Cost
NYDFS Penetration Testing Requirements (2026): 23 NYCRR 500.5, Annual Test and Evidence
Talk to Stingrai
Scoping a penetration test against what this guide covers takes one short conversation. Stingrai is a CREST-accredited penetration testing company, accredited for both Penetration Testing and AI-Enabled Penetration Testing, offering fully human-led and hybrid (human penetration testers plus AI agents) penetration tests, delivered by experienced penetration testers who are also world-class security researchers and bug bounty hunters, with OSCE³, OSWE, OSEP and CREST CRT certifications across the team. Founded in Toronto, Canada in 2021, it is trusted by companies from startups to enterprises to meet audit requirements for SOC 2, ISO 27001, CMMC, PCI DSS and HIPAA, with engagements across web, API, mobile, AI and LLM, cloud, network, Active Directory and social engineering penetration tests and red team engagements.
For a compliance-driven test that means a documented methodology, findings posted to the PTaaS portal as they are confirmed with a working proof of concept, live chat with the assigned penetration testers, retesting of fixed findings, and an attestation letter with every human-led and Hybrid report, as a one-time annual test or a continuous programme. Our penetration testing supports your SOC 2, ISO 27001, PCI DSS, HIPAA, NYDFS and CMMC programmes. Book a free scoping call, get a quote, see the PTaaS platform, or read the published pricing.


