An article by Sid Trehan from Keller AI, published on November 10, 2026, examines what formal verification of AI-written code actually proves: it confirms only that the code matches a specific mathematical specification under stated assumptions, rather than proving the code is "bug-free." Against this backdrop, the article reviews two recent examples: approximately 180 million lines of proofs in Rocq generated using Claude by Anthropic and Theorem Labs, and the PDD Kit prototype by GitHub Next and Microsoft Research, which integrates the Verus verifier into the development cycle. The key takeaway is that value shifts from the fact of "proof" to the quality of the specification and the list of assumptions, which now require manual review.
What Happened
The event is the publication on November 10, 2026, of the article "AI proved your code correct. What did it actually prove?" by Sid Trehan from Keller AI. The article's thesis: verifier checks confirm that code matches a specific mathematical specification under stated assumptions, but do not guarantee that the specification aligns with business requirements, nor do they cover unmodelable effects like timing attacks. Two recent works are cited as supporting examples. The first is a result from Anthropic in collaboration with Theorem Labs: using Claude, they generated approximately 180 million lines of proofs in Rocq and proved crash-freedom for 239 entry points in 123 OSS-Fuzz libraries on RISC-V architecture with inputs up to 64 MB; incidentally, hundreds of bugs were found, including a logic bug that had persisted for over 10 years. The second is the PDD Kit prototype, released on November 2, 2026, by GitHub Next in collaboration with Microsoft Research: a human reviews the intent, an agent writes the Rust code, and the Verus verifier checks it.
Context
Formal verification has remained a niche for critical systems for decades: according to the article, seL4 required approximately 20 person-years for verification, and the Rocq/Coq and seL4 tools have existed for a long time. The novelty of the current moment lies not in theory, but in economics: LLMs as proof synthesizers reduce the production of machine-checkable artifacts from person-years to weeks, meaning this is a scaling result in verification cost, not a new verification algorithm. Technically, it is important that proofs in Rocq are checked by the proof kernel, so the verdict is binary—"accepted or not accepted"—and is almost immune to manipulation. This leads to the main procedural shift: review moves from line-by-line code reading to checking the specification and assumptions, and the verifier's "green checkmark" itself begins to require audit, otherwise "vibe verification" with trivial invariants arises.
Why This Matters for the Industry
For the industry, this is a transition of verification from a niche for critical systems to a standard stage of the pipeline: agents write both code and proofs, Verus checks are integrated into CI for Rust components, and formal guarantees cease to be manual expertise for individual projects. For product developers and startups, this is primarily a change in cost structure and the trust model for agent-generated code: value shifts from the fact of having a proof to the quality of the specification and the trusted base of assumptions, making the audit of specs and assumptions mandatory in code review for generative pipelines. Within a six-month horizon, it is logical to expect independent replication of the Anthropic result by third-party teams, the first verifier gates in CI for agent pipelines, and benchmarks for specification quality—these will show whether the trend is sustainable. Within a two-year horizon, proof-driven development may become normalized for critical segments like parsers, serialization, and cryptographic deserialization, and review skills will shift to formulating specs and auditing assumptions.
Why This Matters for Users
For readers who use AI agents for code, it is important to understand what the phrase "the agent proved correctness" means: a specific property, such as crash-freedom, is proven under specific assumptions—single-threading, RISC-V architecture, inputs up to 64 MB—rather than the absence of any bugs. The practical conclusion: instead of trusting the verification status, read the specification and the list of trusted assumptions; PDD Kit provides sign-off cards for each external component, including SQLite, bcrypt, and JWT, for this purpose. The Keller AI article is interactive: you can manually construct a proof and see how induction differs from tests. This approach works outside the prototype as well: reviewing intent, specs, and the list of assumptions before launching an agent provides a more honest guarantee than blind faith in the verifier's green checkmark.
What Is Still Unknown / Limitations
The result from Anthropic and Theorem Labs is currently confirmed by a single report without a public independent replicative run; the question of the openness of the proof sources for independent re-verification remains open. PDD Kit is a prototype, and access conditions are unclear according to sources; the Anthropic report, for its part, is not accompanied by an API, pricing, or latency figures. The signals are very fresh: PDD Kit was released on November 2, 2026, and the article on November 10, 2026, so there are no commercial products with prices and SLAs yet. The domain is currently narrow—mainly Rust with Verus and RISC-V architecture—industrial coverage of all code remains hypothetical. Fundamental limits remain: aligning the specification with business requirements and unmodelable effects like timing attacks remain a human task and are outside the verifier's guarantees.
Sources
- AI proved your code correct. What did it actually prove? — Keller AI (Sid Trehan)
- LLMs are making formal software verification feasible at a much larger scale — Anthropic Research (jointly with Theorem Labs)
- PDD Kit — GitHub Next (jointly with Microsoft Research)
Author
Look at AI, editorial team