Vals AI Finds Claude Opus 5.5 Beats GPT-6 Astra on Math Proofs

Vals AI tested GPT-6 Astra, Claude Opus 5, and Claude Opus 5.5 across every reasoning-effort setting each model supports, from low through max, on ProofBench v1.1. The results show that higher effort can reduce total cost by shortening an agent’s trajectory, while pushing beyond a model’s accuracy plateau burns tokens without producing more accepted proofs.

Lean makes every proof accountable

ProofBench contains a public split and a private test split, each with 100 graduate-level mathematics problems. Every task pairs a natural-language question with a Lean 4 statement. The model must produce a proof that passes Lean’s kernel, the small trusted component that checks whether every logical step is valid. Only kernel-accepted proofs receive credit.

The benchmark’s formalizations were written by experienced Lean users, including doctoral researchers and domain experts, then reviewed for errors. The problems come from qualifying exams and graduate textbooks across probability, measure theory, real and functional analysis, commutative algebra, algebraic geometry, number theory, and model theory. The benchmark paper describes the construction and evaluation process.

Each model operates through an agent harness with three tools: lean_loogle searches Mathlib, Lean’s main theorem library; lean_run_code compiles candidate code; and the one-shot submit_proof call sends the final answer. Agents receive up to 40 turns per problem. Every accepted submission is also checked with #print axioms, which disqualifies proofs that depend on sorry, admit, or newly introduced axioms.

Higher effort can lower the bill

GPT-6 Astra’s move from medium to high effort increased reasoning spend per turn by roughly $0.70 but reduced the run by 117 turns. Accuracy rose while total cost fell. On an agentic task, a more expensive turn can save money when it prevents repeated searches, failed compilations, and abandoned proof attempts.

Claude Opus 5 showed the opposite behavior after reaching high effort. Its accuracy plateaued at 98%, while xhigh and max consumed additional tokens without yielding another accepted proof. At the upper end, the run used about 4.4 million tokens and cost roughly $167.

The efficient settings sit below max

Vals used k=3 sampling, giving each task three sampled attempts per configuration to reduce variance. The selected results below reproduce the reported effort-sweep figures; costs are approximate per-run totals.

Opus 5.5 gained its final percentage point by moving from medium to max, but the cost rose by about $56, or 156%. GPT-6 Astra also reached 100%, at roughly $14 more than Opus 5.5’s max run.

Claude Opus 5 failed on the same two remaining problems across its upper effort settings. That repeated pattern indicates a model capability ceiling that additional inference compute did not move. A benchmark run at one effort level would conceal the shape of that plateau.

Perfection crowds the leaderboard

The full ProofBench v1.1 leaderboard records separately submitted runs and lists three systems at 100%: Claude Opus 5.5, Claude Fable 5.1, and Logical Intelligence’s AlephProver. GPT-6 Astra and Claude Opus 5 each appear at 99%. Those entries are separate from the controlled k=3 effort sweep, so Astra’s 100% sweep result and 99% leaderboard entry belong to different evaluation records. The same distinction applies to Opus 5’s 98% sweep plateau and 99% leaderboard result.

Each split contains 100 problems, which means one accepted proof changes accuracy by one percentage point. With leading systems clustered between 99% and 100%, repeated trials, cost, and agent behavior provide more information than the headline score alone.

Inkling submits a final proof on only 28% of tasks and receives no accepted proofs. Its result reflects agent completion behavior alongside proof-generation ability, making raw accuracy an incomplete comparison for systems that frequently exhaust their turn budget without submitting.

AlephProver costs $9.35 per task, the highest average on the leaderboard. Opus 5.5 matches its perfect score at roughly one-tenth of that rate. One Hartshorne algebraic-geometry exercise about intersections of affine varieties with hypersurfaces consumed $113 of AlephProver’s budget, about twelve times its own per-task average.

Tool-call spirals burn budget

Gemini 3.1 Pro Preview (02/26) averages 88 tool calls per task, while GPT-5.6 Luna averages 74; both score well below the leaders. Their failed trajectories often contain repeated Mathlib searches for a theorem or concept that does not exist under the attempted names, followed by minor query variations instead of a new proof strategy.

The strongest Claude configurations average roughly 11 to 14 tool calls per task. Their typical sequence is shorter: search for a lemma, test a candidate proof, repair any compiler errors, and submit. The allocation between search and execution does not predict accuracy by itself. Unproductive repetition drives the larger cost differences.

Tune against accepted proofs

Teams integrating reasoning models into formalization, verification, mathematical assistance, or long-horizon coding should treat effort level as an evaluated deployment parameter. The ProofBench results support four practical steps:

• Benchmark every supported level. Record accepted outcomes, total cost, turns per task, tool calls, and the share of runs that end without a submission.
• Begin production tuning at medium. Opus 5.5 delivered 99% accuracy there for about 39% of its max-setting cost. Every tested model lost measurable accuracy at low effort.
• Stop at a measured plateau. Opus 5 gained no accepted proofs above high effort, despite a substantial increase in tokens and cost.
• Optimize the full trajectory. Per-turn expense cannot predict total cost when stronger reasoning reduces retries, searches, and compiler failures.

Lean exposes these patterns clearly because its kernel supplies an exact correctness signal. Similar plateaus and tool-use failures may occur in general software agents, although ProofBench does not establish that transfer. Teams should reproduce the effort sweep on representative internal tasks before choosing a default setting.