UC Berkeley Releases Vero — the First Benchmark for Formally...

UC Berkeley Releases Vero — the First Benchmark for Formally Verified Code

A UC Berkeley team led by Dawn Song released Vero — the first repository-level benchmark: 43 multi-module Lean 4 projects with 743 APIs and 2,705 specifications, where the best agent (GPT-5.5 in Codex) fully solved 27 projects, while 10 were unsolved by any agent.

Compiled by Sergey KostenchukPublished 2026-08-23Updated 2026-09-18

2026-08-23 Research Anthropic

🧮 UC Berkeley Releases Vero — the First Benchmark for Formally Verified Code

Dawn Song’s team (Zhe Ye, Hantao Lou, Yuechun Sun, and others) assembled 43 multi-module Lean 4 projects — 743 APIs and 2,705 specifications derived from real code in Python, Dafny, Verus, and Coq. In codeproof mode, the agent writes both implementations and machine-checkable proofs.

🌍 Evaluation is shifting from individual functions to entire repositories with machine checking: GPT-5.5 (xhigh) in Codex fully solved 27 of 43 projects (87.3% of specifications), while Claude Opus 4.8 solved 8 of 43. No configuration solved 10 projects — repository-level verified synthesis is still beyond agents’ capabilities.

👤 Open resources: GitHub sunblaze-ucb/vero, arXiv 2608.13522, and a public leaderboard. It shows which 10 repositories (from floating-point to the ironkv KV store) frontier models cannot solve — you can run your own agent.

Source 1: https://vero.verina.io/ Source 2: https://arxiv.org/abs/2608.13522

Sources