Dor Cohen published an analysis titled *What Can AI Actually Hack? Inside Frontier Labs' Tests*: he divides AI's offensive capabilities into four levels—from finding a flag in a CTF challenge to turning a known bug into a working exploit—and aggregates statistics on how quickly models pass each level. The lower levels are already saturated, and labs are closing the corresponding benchmarks, while the frontier has shifted to exploit synthesis: on ExploitBench, models grew from single-digit percentages to 73% for Claude Opus 5.5 in just a few months. The analysis provides the first quantitative assessment of defenses: with mitigations enabled, successful exploits are nearly three and a half times fewer.

image
image

What Happened

Researcher Dor Cohen published the article *What Can AI Actually Hack? Inside Frontier Labs' Tests* on his blog, which subsequently gathered discussion on Hacker News. In it, he breaks down the offensive capabilities of language models into four levels of complexity and aggregates the dynamics of results in one place. At the lowest level, where the model searches for a flag in a CTF challenge, the InterCode-CTF benchmark shows growth from 35% in March 2024 to 95% in December 2024. The middle levels, where the model turns a known bug into a working exploit, are practically solved: OpenAI closed CVE-Bench after GPT-5.3-Codex solved 90% of tasks in February 2026, and Anthropic closed CyberGym after Claude Mythos Preview reached 83% in April 2026. The current frontier is the fourth level: synthesizing exploits to code execution. On ExploitBench, the share of models achieving code execution grew from approximately 2% in May 2026 to 73% for Claude Opus 5.5 by September 2026. This map also includes the defensive line: in ExploitGym, enabling mitigations reduces the number of successful exploits from 157 to 45.

Context

The map is easier to read if you remember two things. First—the change in measurement format: early cyberbenchmarks were binary (the model either found the flag or it didn't), and such a metric coarsely and late reflects capability growth. ExploitBench from Carnegie Mellon University is structured differently: it is a capability-graded benchmark for V8 exploit synthesis that separately records intermediate steps on the path to code execution, so progress is visible before the final success. Second—the lifecycle of the metrics themselves: according to the author's observation, public offensive capability benchmarks saturate in approximately nine to eleven months, after which labs close them, leaving the market with almost no independent open measurement points for the previous generation. ExploitBench and SEC-bench Pro are the only ones that remain effectively alive, where, for example, GPT-6 Astra shows 85.4% compared to 51.75% for GPT-6 Luna. This has a consequence for competition: the lower levels of attacks are accessible to everyone through standard APIs, and the ability to quickly find bugs ceases to be an advantage, becoming a standard feature.

Why This Matters for the Industry

For the industry, the signal is twofold. On one hand, the window between "the model can" and "the model does it in production" is narrowing: the lower levels are already automated through existing APIs, so agentic copilots for triage and prioritization of vulnerabilities can be built now without waiting for new models, while manual pentest services at these levels will face pressure from automation. On the other hand, defensive products have gained a metric: the delta in successful exploits with mitigations enabled, obtained in ExploitGym, turns the sandbox from a general practice into a measurable feature that can be shown to clients. In parallel, evaluation practice is forced to move to level 4, to graduated ladders with intermediate steps and its own, leak-protected task corpora: if the saturation cycle persists, the top point of ExploitBench for Claude Opus 5.5 is already close to the saturation threshold, and vendors, following the closed benchmarks of the previous wave, will move to private held-out corpora. For startups, the window in defensive products and live capability tracking remains open.

Why This Matters for Users

For the reader, this is a clear map of where the boundary currently lies. Models no longer just find known bugs but take them to the execution of their own code, for example in the V8 engine of the Chrome browser. Progress can be tracked independently via the live benchmarks ExploitBench from Carnegie Mellon and SEC-bench Pro, without waiting for press releases. If you use AI agents with the right to execute code, the practical conclusion is simple: it is worth conducting an inventory of all places where the agent receives such rights and checking isolation—sandbox, access rights, outgoing traffic control. The same applies to your services where the model generates or runs code: defensive measures in this map are backed by numbers, so isolating such points is long past being a formality.

What Is Still Unknown / Limitations

The conclusion about saturation in nine to eleven months is the author's generalization based on several points, including InterCode-CTF, CVE-Bench, and CyberGym, and not a meta-analysis; it has no independent verification. Closing a benchmark by a vendor may reflect the maturity of the tool itself, not just task saturation. The main validity risk is contamination: public CVEs and write-ups could have ended up in training data, and neither ExploitBench nor ExploitGym describes held-out or private tasks, so part of the growth from approximately 2% to 73% may be explained by memorization rather than generalization. The composition of mitigations in ExploitGym and their overheads are not disclosed in the source, so the obtained delta cannot yet be transferred to specific production configurations. Finally, this is a map of benchmarks, not a product launch: no new production tool has appeared here.

Sources

Author

Look at AI, editorial team