A UC Berkeley team led by Dawn Song released Vero — the first open benchmark that evaluates AI agents at the level of entire multi-module repositories with formal verification in Lean 4. The best tested configurations solved only a portion of the projects, and 10 instances were not solved by any model: according to the authors, repository-level verified synthesis remains an unsolved problem.
What happened
The benchmark consists of 43 multi-module projects in Lean 4, containing a total of 743 public APIs and 2,705 specifications; they were translated from real code in Python, Dafny, Verus, and Coq. In codeproof mode, the agent must write not only implementations but also machine-checkable proofs for each specification. Each run is limited to 90 minutes of wall-clock time. The best result was achieved by GPT-5.5 (xhigh) in Codex: 27 of 43 instances were fully solved, corresponding to 87.3% of all specifications; Claude Opus 4.8 (xhigh) solved 8 of 43. Ten projects were not solved in any tested configuration, and the authors conclude that current agents are not yet capable of repository-level verified synthesis.
Context
A key feature of Vero is its evaluation criterion: the correctness of each specification is confirmed not by another model's assessment, but by machine checking in Lean 4, which eliminates the typical subjectivity of LLM-as-judge in code evaluation. Scale is also important: previous agent evaluations were mostly conducted on individual functions, whereas here the unit of measurement becomes an entire multi-module repository. The benchmark includes an audit mechanism: the agent is formally allowed to prove the incorrectness of the benchmark's own specification and correct it, which protects results from silent errors in the curation pipeline — a standard problem with manual benchmarks of verified code. Vero is not a product but an open evaluation platform: the curation pipeline and evaluation harness, published together with the paper on arXiv (2608.13522).
Why this matters for the industry
For the industry, Vero creates a publicly measurable metric: "agent-created verified software" has become a measurable quantity on a common reproducible standard, allowing vendors of coding agents to compare their models against each other. A leaderboard with progress curves provides a visible indicator of whether the capability gap is closing, and it can be expected that model vendors will start optimizing for Vero, publishing results in model announcements, and building hybrid agent + automated theorem prover pipelines aimed at the remaining unsolved instances. At the same time, the benchmark provides a sober production signal: no model provides a verified repository "out of the box," so repository-level verified generation is not yet a reliably deployable capability.
Why this matters for users
The benchmark is fully open: the code, curation pipeline, and evaluation harness are available in the GitHub repository sunblaze-ucb/vero, the methodology is described in arXiv paper 2608.13522, and a public leaderboard with progress curves allows tracking dynamics over time. Any team can run its own agent on the 43 repositories and track results against public curves. A separate value is the list of 10 repositories that no frontier model has yet solved: from floating-point formalizations to the distributed KV store ironkv. This is a clear map of the current limits of agents in verifiable code.
What is still unknown / limitations
Results were obtained for specific model and scaffolding configurations (GPT-5.5 and Claude Opus 4.8 in xhigh mode) with a 90-minute wall-clock budget; different time or scaffolding could yield a different picture. The conclusion about the inability of current agents is based only on the tested configurations, not on the entire class of possible systems. The benchmark is research infrastructure, not a product for end users, and predictions about vendors optimizing for Vero and about a future class of "verified by default" software remain scenarios, not confirmed facts.
Sources
Author
Look at AI, editorial team