An autonomous AI agent reached number one on HackerOne's US bug-bounty leaderboard in June 2025, submitting roughly 1,060 vulnerability reports across live programs in about 90 days (XBOW, 2025). Six months later, a research agent placed second against ten human professionals on a live 8,000-host network (Stanford, CMU, and Gray Swan, 2025). So here is the direct answer the 2026 benchmarks give: autonomous AI is now genuinely competitive at penetration testing, but it has not replaced senior human testers, because every headline result still depended on human review and the measured accuracy gaps are real. This post is a dated, sourced scoreboard of the benchmarks behind that answer. Every figure carries its scope caveat and a link back to the primary source, so you can read the numbers for what they measured rather than what a launch post implied.
TL;DR: the numbers that matter
Autonomous agent tops a real leaderboard (Jun 2025): XBOW reached number one on HackerOne's US leaderboard with roughly 1,060 reports in about 90 days (XBOW, 2025).
AI places second against pros (Dec 2025): the ARTEMIS agent scored 2nd of eleven, found 9 valid vulnerabilities at an 82 percent valid-submission rate, and beat 9 of 10 human professionals on a live network (arXiv 2512.09882).
Real web-app exploitation stays low: on CVE-Bench, the best agent framework exploited up to 13 percent of critical web CVEs in the zero-day setting and 25 percent in the easier one-day setting (arXiv 2503.17332).
False positives are the tax: the ARTEMIS study reports AI agents carried higher false-positive rates than every human participant and ran at about US$18 per hour against roughly US$60 for professional testers (arXiv 2512.09882).
Adoption is climbing fast: HackerOne recorded a 210 percent year-over-year jump in valid AI reports and a 540 percent rise in valid prompt-injection reports (HackerOne, 2025).
CTF puzzles are largely solved: on Cybench, top agents solved complete tasks that took human teams up to 11 minutes, though the hardest task took humans 24 hours 54 minutes (arXiv 2408.08926).
AI now finds real zero-days: Google's Big Sleep agent found SQLite flaw CVE-2025-6965 and helped cut off an in-the-wild exploit before it landed (Google, 2025).
Humans stay in the loop: only 12 percent of surveyed researchers believe AI will replace them (HackerOne, 2025).
Key takeaways
The replacement question is settled for 2026, and the answer is no. The strongest results all keep a human in the loop: XBOW's findings were automated but reviewed by its security team before submission (XBOW, 2025), and only 12 percent of HackerOne researchers think AI will replace them (HackerOne, 2025).
The competitive result is also real, and that is the bigger story. An AI agent placing second against paid professionals on a live network is not a demo. It means the human edge has narrowed to the hardest work rather than disappeared (arXiv 2512.09882).
A benchmark number means nothing without its setting. CVE-Bench's headline jumps from 13 percent to 25 percent purely by moving from the zero-day setting to the one-day setting, where the agent is handed the vulnerability description (arXiv 2503.17332). Read the setting before the number.
False positives, not misses, are where AI still costs you. Higher false-positive rates than every human (arXiv 2512.09882) turn straight into validation hours, which is exactly the work senior testers should own.
Real-world web-app exploitation is the unsolved frontier. Solving CTF puzzles in minutes and topping a volume-weighted leaderboard are not the same as autonomously exploiting a novel business-logic flaw, and the measured web-CVE numbers make that gap explicit.
Methodology
This scoreboard tracks the AI and autonomous penetration testing benchmarks published between August 2024 and December 2025 that we could verify against a named primary source. Sources used, with publication dates and data windows:
Cybench (Zhang et al., arXiv 2408.08926, published August 2024): 40 professional Capture the Flag tasks drawn from 4 competitions.
CVE-Bench (Zhu, Kang et al., arXiv 2503.17332, published March 2025): 40 critical-severity real-world web-application CVEs dated May to June 2024.
Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing, the ARTEMIS study (Stanford, Carnegie Mellon, and Gray Swan AI, arXiv 2512.09882, published December 2025): a live university network of roughly 8,000 hosts across 12 subnets.
HackerOne 9th Hacker-Powered Security Report (published October 2025): platform data plus a researcher survey.
XBOW HackerOne results (XBOW, June 2025): the vendor's own reporting on its live bug-bounty submissions, corroborated by trade press.
Google Big Sleep (Google, 2025): the vendor's own reporting on real-world vulnerability discovery.
The research cutoff for this pass is July 2026. Where a figure came from a vendor's own program rather than an independent evaluation, it is labeled as vendor-operated or self-reported and caveated in the table. One widely repeated statistic, that 58 percent of researchers say AI misses business logic, could not be matched to the primary HackerOne source on this pass, so it was dropped rather than printed. Every figure below links back to its primary publisher so any claim can be audited inline.
The 2026 AI pentest benchmark scoreboard
This is the tracked table. Each row states what the benchmark actually measured, the headline result, the scope caveat that changes how you should read it, and the primary source.
Benchmark or study | What it measured | Headline result | Scope caveat | Source |
|---|---|---|---|---|
Cybench (Aug 2024) | LLM agents solving 40 professional CTF tasks from 4 competitions | Top agents solved complete tasks that took human teams up to 11 minutes; the hardest task took humans 24h 54m | Isolated CTF puzzles, not live applications; unguided solve rates stayed low on the harder tasks | |
CVE-Bench (Mar 2025) | Autonomous exploitation of 40 critical real-world web-app CVEs | Best agent framework exploited up to 13% in the zero-day setting, 25% in the one-day setting (success@5) | Reproduces known CVEs in a sandbox; the one-day setting hands the agent the vuln description; up to 5 attempts allowed | |
ARTEMIS live-network study (Dec 2025) | 10 human professionals vs 6 AI agents plus ARTEMIS on a live ~8,000-host network | ARTEMIS placed 2nd overall, 9 valid vulns at an 82% valid rate, beat 9 of 10 humans; ~$18/hr vs ~$60/hr | Single network, infrastructure-style pentest; AI carried higher false-positive rates than humans and struggled with GUI tasks | |
HackerOne 9th HPSR (Oct 2025) | Real bug-bounty platform data plus a researcher survey | 210% YoY jump in valid AI reports; 540% rise in valid prompt-injection reports; 560+ valid hackbot reports | A "valid AI report" is AI-assisted with a human in the loop, not AI alone; 12% of researchers think AI will replace them | |
XBOW on HackerOne (Jun 2025) | An autonomous agent submitting to real bug-bounty programs | Reached #1 on the US leaderboard; ~1,060 reports in ~90 days (54 critical, 242 high, 524 medium, 65 low) | Vendor-operated and human-reviewed before submission; the leaderboard rewards volume, and many reports were duplicate or informative | |
Google Big Sleep (2025) | An AI agent hunting real vulnerabilities in production software | Found SQLite flaw CVE-2025-6965 and helped pre-empt an in-the-wild exploit; found 20 flaws in open-source tools | Memory-safety bugs in native code, not web-app business logic; runs inside a supervised research program |
The one real-world dataset worth staring at
XBOW's leaderboard run is the closest thing to a public, at-scale, real-world result. Across about 90 days the agent submitted roughly 1,060 reports: 54 critical, 242 high, 524 medium, and 65 low (XBOW, 2025). That is a real spread of findings on real targets, and it is also volume-weighted and human-gated. The findings were automated but the security team reviewed them before submission to comply with HackerOne's policy on automated tooling, and the public data showed large duplicate and informative buckets alongside the resolved ones. Leaderboard rank measures throughput and reputation, not blind precision.
How to read these numbers
The single most common mistake is treating a controlled benchmark, a real-world platform statistic, and a vendor-operated result as if they are the same kind of evidence. They are not, and the caveats move the numbers more than the models do.
Controlled academic studies (Cybench, CVE-Bench, ARTEMIS) are the most trustworthy because the setup is documented and reproducible. Read the setting first. CVE-Bench's 13 percent becomes 25 percent purely by switching from the zero-day setting to the one-day setting and allowing up to five attempts (arXiv 2503.17332). Cybench's "solved in minutes" applies to tasks humans also solved quickly, not to the 24-hour task (arXiv 2408.08926).
Real-world platform data (HackerOne) is valuable but aggregated and definitional. A 210 percent jump in "valid AI reports" is real growth, but "valid AI report" means AI-assisted with a human still in the loop, not an unsupervised agent finding and confirming a bug alone (HackerOne, 2025).
Vendor-operated results (XBOW, Big Sleep) are the least controlled and should be read with the operator's incentive in mind. They can still be true and important. Big Sleep genuinely found a real SQLite zero-day that was about to be exploited (Google, 2025). Just treat self-reported severity splits and leaderboard ranks as directional, not audited.
Three questions collapse most benchmark hype. What exactly was measured, a CTF puzzle or a live app? What was handed to the agent, nothing or the vulnerability description? And who confirmed the finding, the agent alone or a human reviewer? Run any 2026 AI pentest claim through those three and the picture clears up fast. For a deeper split of the same evidence, see autonomous versus human pentesting: the scope split.
Where AI is competitive, and where humans still win
The honest thesis from all six sources is the same: AI is now competitive enough that the human advantage has narrowed to the hardest, highest-judgment work rather than vanished.
AI is genuinely strong at breadth and speed: systematic enumeration, parallel exploitation, and known vulnerability classes, at roughly a third of the hourly cost of a professional (arXiv 2512.09882). It now finds real zero-days in production code (Google, 2025). Where senior humans still lead is novel business logic, chained multi-step exploits, and the final judgment of whether a finding is real, exploitable, and worth reporting. That last point is why every leaderboard win kept a human reviewer in front of submission, and why the ARTEMIS agents' higher false-positive rate matters more than their ranking.
This is exactly the seam Stingrai's autonomous web-application agent, Snipe, is built for. Generic AI scanners cap out at known-class bugs, but Snipe is purpose-built to hunt the complex, high-impact classes that the benchmarks show machines still fumble: IDOR, broken authorization, and business-logic flaws. It is custom-trained on more than 6,000 HackerOne Hacktivity disclosure reports and on skills distilled from years of Stingrai's human pentesters, it runs both black-box and white-box code review, it can open AutoFix pull requests, and it can gate a pull request before vulnerable code merges. The AI does the breadth; Stingrai's senior human testers own the validation and the novel-logic work the benchmarks say humans still win.
What this means for security leaders
Do not buy on a leaderboard rank or a single percentage. Ask a vendor which setting produced the number and whether a human confirmed the findings. If they cannot answer, the number is marketing.
Budget for validation, not just detection. The measured false-positive gap (arXiv 2512.09882) is real labor. A credible AI pentest bakes senior human review into the price rather than shipping you a raw queue.
Use AI for cadence and coverage, humans for depth. Continuous autonomous testing between engagements plus senior humans on the high-severity findings is the model the 2026 evidence supports. See how the two divide in traditional pentesting versus AI pentesting.
Pilot on your own app before you commit. Benchmarks describe someone else's target. A structured trial on your application, with planted bugs to measure both recall and false positives, tells you what a tool does on your attack surface. The 30-day AI pentest bake-off is a run-it-yourself version.
Match the model to the job. If you are comparing engines, the best AI model for pentesting breakdown pairs benchmark behavior with real testing tasks.
Stingrai runs this hybrid in production through its AI-augmented PTaaS, with Snipe driving breadth and continuous coverage and senior web application penetration testing and red team experts owning validation and novel-logic work. Pentest evidence from those engagements supports your SOC 2, ISO 27001, and PCI DSS compliance programs. Scope and pricing live on the pricing page.
Frequently asked questions
What do the 2026 AI pentesting benchmarks actually show?
They show that autonomous AI has become genuinely competitive at penetration testing but has not replaced human testers. An autonomous agent reached number one on HackerOne's US leaderboard in June 2025 (XBOW, 2025), and a research agent placed second against ten professionals on a live network in December 2025 (arXiv 2512.09882). At the same time, autonomous exploitation of real web CVEs stayed at up to 13 to 25 percent (arXiv 2503.17332), and every headline result kept a human in the loop.
Can AI replace human pentesters in 2026?
No. The evidence points to a hybrid, not a replacement. Only 12 percent of surveyed researchers believe AI will replace them (HackerOne, 2025), and the ARTEMIS agents, despite ranking second, carried higher false-positive rates than every human and struggled with tasks humans handle easily (arXiv 2512.09882). AI adds breadth and speed; senior humans own novel logic and validation.
Which AI pentest benchmark is the most credible?
For controlled, reproducible evidence, the academic studies rank highest: Cybench, CVE-Bench, and the ARTEMIS live-network study all document their setup and can be independently checked (arXiv 2408.08926; arXiv 2503.17332; arXiv 2512.09882). Real-world platform data from HackerOne is useful for trend direction, and vendor-operated results such as XBOW's and Google's should be read as directional rather than audited.
What is the ARTEMIS study and what did it find?
ARTEMIS is a multi-agent framework evaluated in a December 2025 paper that pitted ten human professionals against six existing AI agents plus ARTEMIS on a live university network of roughly 8,000 hosts (arXiv 2512.09882). ARTEMIS placed second overall, found 9 valid vulnerabilities at an 82 percent valid-submission rate, and outperformed 9 of 10 humans, while running at roughly a third of the hourly cost. It also posted higher false-positive rates than the humans, which is the study's central caveat.
How accurate is autonomous AI at exploiting real web vulnerabilities?
Modest, and lower than the leaderboard headlines suggest. On CVE-Bench, which uses 40 critical real-world web CVEs, the best agent framework exploited up to 13 percent in the zero-day setting and 25 percent in the one-day setting, where it is given the vulnerability description and up to five attempts (arXiv 2503.17332). That gap between settings is why the setting matters as much as the model.
Did an AI really reach number one on HackerOne?
Yes. In June 2025, XBOW became the first autonomous system to top HackerOne's US leaderboard, submitting roughly 1,060 reports in about 90 days, including 54 critical and 242 high-severity findings (XBOW, 2025). The findings were automated but reviewed by the security team before submission, and the leaderboard rewards volume, so it is best read as a scale milestone rather than a precision one.
What can AI pentesting still not do reliably?
It still fumbles novel business-logic flaws, long chained multi-step exploits, and the final judgment of whether a finding is real. It also generates more false positives than humans (arXiv 2512.09882) and solves only a minority of real web CVEs unaided (arXiv 2503.17332). Purpose-built agents like Stingrai's Snipe are designed to push into IDOR and authorization classes, but senior human validation remains the standard for high-severity work.
Where can I find the latest AI pentest benchmark data?
Track the primary sources directly: arXiv for Cybench, CVE-Bench, and the ARTEMIS study, HackerOne's annual Hacker-Powered Security Report for platform trends, and vendor and lab posts from XBOW and Google's Big Sleep for real-world runs. All of them are linked in the References section below, and the scoreboard above is refreshed as new dated benchmarks publish.
References
Zhang et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. August 2024. https://arxiv.org/abs/2408.08926. Evaluates LLM agents on 40 professional CTF tasks from 4 competitions, benchmarked against human first-solve times.
Zhu, Kang et al. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities. March 2025. https://arxiv.org/abs/2503.17332. Measures autonomous exploitation of 40 critical real-world web CVEs in a sandbox under zero-day and one-day settings.
Stanford, Carnegie Mellon, and Gray Swan AI. Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing (ARTEMIS). December 2025. https://arxiv.org/abs/2512.09882. Head-to-head of 10 human professionals against 6 AI agents plus ARTEMIS on a live 8,000-host network.
HackerOne. 9th Annual Hacker-Powered Security Report. October 2025. https://www.hackerone.com/report/hacker-powered-security. Platform data and researcher survey on AI-assisted vulnerability discovery.
HackerOne. Report Finds 210% Spike in AI Vulnerability Reports Amid Rise of AI Autonomy. October 2025. https://www.hackerone.com/press-release/hackerone-report-finds-210-spike-ai-vulnerability-reports-amid-rise-ai-autonomy. Press release with the 210 percent and 540 percent figures.
HackerOne. The Top Researcher Signals From HackerOne's 2025 HPSR. 2025. https://www.hackerone.com/blog/2025-hpsr-researcher-signals. Source for the 12 percent replacement and 43 percent hackbot figures.
HackerOne. 3 Signals from the 2025 Hacker-Powered Security Report. 2025. https://www.hackerone.com/blog/ai-security-trends-2025. Context on valid AI report growth and human-in-the-loop framing.
XBOW. How XBOW Ranked #1 in Autonomous Penetration Testing. June 2025. https://xbow.com/blog/top-1-how-xbow-did-it. Vendor report on reaching #1 on HackerOne's US leaderboard with severity breakdown.
TechRepublic. AI Bug Hunter Sets Milestone by Claiming Top Spot on HackerOne's Leaderboard. 2025. https://www.techrepublic.com/article/news-ai-xbow-tops-hackerone-us-leaderboad/. Independent trade-press corroboration of the XBOW leaderboard result.
Google. Google's Latest AI Security Announcements (Big Sleep). 2025. https://blog.google/innovation-and-ai/technology/safety-security/cybersecurity-updates-summer-2025/. Vendor report on Big Sleep finding SQLite CVE-2025-6965 and pre-empting an in-the-wild exploit.
