Google Project Zero conducted a direct quantitative comparison of the classic generative fuzzer Fuzzilli and the agent-based system BigSleep, built around a large language model, on the V8 engine that powers the Chrome browser. From April to September 2026, BigSleep found 45 fixable vulnerabilities compared to 37 for Fuzzilli, while the overlap of findings in production was only about a quarter. The researchers' conclusion: the two approaches search different areas of the error space and complement each other, so it makes sense for vendors to run both fuzzing and agent-based analysis in parallel.

image
image

What happened

The comparison was published in the Project Zero blog in October 2026. From April to September 2026, both systems operated on the V8 engine: BigSleep found 45 fixable vulnerabilities during this period, Fuzzilli found 37, and the overlap of bugs found in production was 20 out of 81, or about 25%. In a separate series, the authors tested how the agent handles known errors: out of 112 historical Fuzzilli bugs from 2023-2026, BigSleep on the Gemini 3.6 flash model reproduced about 62% with a fixed prompt and up to 70% with a changed prompt and entry point, with a budget of 25 attempts allocated for each commit. To exclude hints from training data, a contamination check was performed: no significant memorization of bugs found before the model's January 2026 cutoff was detected—the agent searches for errors independently rather than reproducing them from memory.

Context

Fuzzilli represents classic coverage-guided fuzzing: the tool massively generates JavaScript programs and relies on code coverage signals, which is why it has been considered the primary method for catching specific JIT compilation errors in engines for years. BigSleep is structured differently: it is a Project Zero agent based on a language model that independently decides which parts of the code to explore and deliberately constructs tests instead of blindly enumerating inputs. This methodological difference is crucial for interpreting the results: the coverage fuzzer has a dense feedback gradient but lacks an understanding of program logic, while the agent has the opposite— the ability to reason about the intended behavior of the code. The combination of mass generation and targeted exploration determines what classes of errors each tool is capable of finding.

Why this matters for the industry

For the security industry, this is the first large-scale quantitative measurement of how well LLM agents actually compete with mature coverage-guided fuzzing on a production engine. The picture is symmetrical: the agent wins on complex logical bugs—for example, missing TDZ checks in the JIT—where the coverage fuzzer does not receive rewards for the right combination of constructs, but the fuzzer remains stronger on specific JIT bugs. This leads to a direct conclusion for vendors and security teams: methods should be run in parallel, and due to the small overlap of findings, deduplication and unified triage become mandatory parts of such a pipeline. BigSleep itself remains a Project Zero research system, not a product, so the realistic next step is offline pilots of agent-based analysis alongside existing fuzzing, plus transferring the publication methodology to internal evals: replay on historical bugs, contamination checks, and a fixed attempt budget provide a fair template for comparing agents with classic tools.

Why this matters for users

For readers following where large language models are actually useful and where they are not, the Project Zero measurement provides a sober benchmark: agent-based analysis has reached the level of a mature specialized tool in terms of effectiveness, but this is not a story about AI defeating fuzzers—bugs that are "complex" for the fuzzer are often complex for AI as well, and the overlap of findings is higher for easier errors. Chrome users receive a direct, albeit not immediate, benefit: the more independent methods hunt for engine errors, the higher the chance that a vulnerability will be closed before it is exploited by attackers. Those choosing tools for auditing their own code should rely on such measurements rather than marketing promises.

What is still unknown / limitations

Reading "45 vs. 37" as a capability rating is not possible: the measurement was conducted on a single engine over a six-month period, and the sources do not provide variance between runs or confidence intervals, so the difference of 8 bugs over such a horizon is statistically unconvincing. Reproduction assessment is sensitive to configuration: the discrepancy between the fixed prompt and the variation of the prompt and entry point was about 8 percentage points, so agent pipeline results depend on settings. The economics of implementation are also unknown—the sources contain no data on the cost, latency, and API of the agent run—and there are no independent replications on other engines like SpiderMonkey and JavaScriptCore or in other domains yet.

Sources

Author

Look at AI, editorial team