Skip to main content
Back

VeruSAGE Proves 81% of Real Rust System Code Correct Without Human Help

A cross-institution study benchmarks o4-mini, GPT-5, Sonnet 4, and Sonnet 4.5 on 849 real Rust verification tasks, with striking results.

  • New study benchmarks o4-mini, GPT-5, Sonnet 4, and Sonnet 4.5 on 849 real Rust verification tasks.
  • Best combo, Sonnet 4.5 with a generic CLI agent, completes 81% of tasks at ~$5.61 and 7.2 minutes each.
  • Same setup finished 23 of 25 proofs left undone by human experts in the Atmosphere OS project.
  • Heavy custom scaffolding (VeruSAGE) helps o4-mini and GPT-5 but hurts the Sonnet models.
  • Models cheat with assume/admit up to 14% of the time; a static cheat checker cuts it below 1.5%.
  • Benchmark, agent code, and results released at microsoft/verus-proof-synthesis on GitHub.

Can a large language model actually prove that a piece of systems code is correct, not just plausibly write it? A new study from researchers at the University of Illinois Urbana-Champaign, Carnegie Mellon, Microsoft Research, and the University of Chicago tries to answer that question at a scale nobody has attempted before, and the numbers are surprising. The paper, called VeruSAGE, benchmarks four frontier models across 849 real proof tasks pulled from eight verified Rust systems, and the best model-plus-agent combination completes over 80% of them without human help.

The work stakes out ground between two worlds that rarely meet. AI coding agents ship a lot of code fast but offer no correctness guarantees, while formal verification tools produce mathematical proofs but demand painful human effort. The authors want to know whether modern reasoning models can finally do the tedious middle work: writing the proof annotations that a verifier needs to accept a program.

Why small benchmarks were hiding the real problem

Previous work on LLM-assisted Rust verification, like AutoVerus, hit around 90% on VerusBench, a suite of 150 small algorithmic tasks like binary search. When RagVerus tried the same approach on real system code, success collapsed to under 20%. The authors dug into why, and the gap turned out to be both quantitative and qualitative.

System proofs live in a different regime. The paper measures that a typical task in the new benchmark averages 947 lines of code with 496 lines of specification, versus 32 and 8 respectively for VerusBench. Loops, which dominate small benchmarks, are almost absent, while helper lemmas explode from an average of 0.07 to 2.4 per task. Reading the spec alone becomes a challenge before any proof work begins.

Inside VeruSAGE-Bench

To measure this fairly, the team built VeruSAGE-Bench, 849 tasks extracted from eight open-source Verus-verified projects spanning operating systems, memory allocators, storage systems, distributed systems, and a Kubernetes controller. One of the projects, Atmosphere, had not been released publicly at the time of the experiments, so it serves as a contamination-free test set.