Coding agents struggle after the first working version

Vals AI’s Vibe Code Bench 1-100 evaluates sequential software maintenance. A model inherits a working web application, receives up to ten product requests, and must implement each one without breaking new or existing behavior.

No evaluated model reaches 30% on the benchmark, and leading systems fail the first requested modification on many tasks. Claude Opus 5 ranks first at 28.53%, followed closely by Claude Fable 5.1 and GPT-6 Astra.

Ten changes, one codebase

Each of the 100 application arcs begins with a fully passing web app from Vibe Code Bench v1.1. An arc consists of one base application and a sequence of related requests. The model edits the same repository throughout while retaining its conversation and tool history, so later changes depend on earlier implementation decisions.

Product requests use the plain language common in tools such as Cursor and Claude Code, without prescribing an implementation. A request such as “add a private Saved area for posts, with notes” may require changes across storage, permissions, navigation, and user-interface state while preserving the four features added earlier.

After every edit, a browser agent runs acceptance steps for the new request and selected regression workflows for existing behavior. Every step must succeed for a workflow to pass. Credit on an arc stops at the first incomplete implementation, incorrect result, or regression, and the reported percentage aggregates that consecutive progress across all arcs. Coverage is limited to the specified acceptance paths and selected regression workflows.

Five ways continuity breaks

The benchmark contains 939 requests divided among five maintenance categories:

Reverts produce the highest pass rates across models, while schema migrations produce the lowest. Data-model changes create more opportunities to lose identifiers, relationships, or historical records, especially after several earlier edits have altered the application’s assumptions.

Regressions cause four in ten failures

Across recorded workflow failures, 60.7% affect only the new request. Existing behavior accounts for the remaining 39.2% through isolated regressions or failures that damage both old and new features.

Failures expose familiar engineering hazards

• Privacy scope: A private resume field leaked into a publicly visible cover letter.
• Accounting invariants: A payment ledger retained receipts but dropped the paired rows required for double-entry bookkeeping.
• Identity matching: A historical import created a duplicate user after failing to associate records with an existing account.

Self-checking tracks the leaderboard

Tool-use logs show distinct validation strategies among the leading models. Opus and Fable devote about one-third of their tool calls to browser interactions. Astra uses the browser for roughly one-ninth of its calls and relies more heavily on task tracking, averaging 11 tracker calls per iteration compared with 3.7 for Opus.

Grok spends most of its tool calls in the file editor and rarely opens the browser. Astra compensates for lighter browser use by probing the application with HTTP requests from the shell. Models that perform no browser validation generally appear in the bottom third of the leaderboard, while Astra’s HTTP checks provide another way to inspect running behavior.

The observed association supports testing the deployed application during long edit sequences, although the benchmark does not establish that a specific tool-use pattern causes a higher score.

A six-point gap carries a steep price

Reported prices widen the small score differences. Opus 5, Fable 5.1, and Astra cost 8 to 30 times as much per run as fourth-place Luna, while Luna trails Opus by roughly six percentage points. Fable averages about $150 per evaluated task, putting its reported 50-task run near $7,500.

A risk-based router could send routine edits to Luna and reserve higher-cost models for schema migrations or cross-cutting policy changes. Every route still needs the same acceptance and regression gates. Production costs will also vary with repository size, context length, tool use, retries, and fallback policies.

Two caveats affect the rankings

Astra sometimes requests clarification and then waits indefinitely because the benchmark provides no user response. The reported cases involve ambiguities that the agent is expected to resolve or requirements for which several reasonable implementations would pass. Non-interactive pipelines using similar agents need explicit defaults, timeouts, and escalation paths.

Claude Fable 5.1 refuses a turn on 14 of its 50 evaluated tasks. Claude Opus 5 handles each refused turn, after which Fable resumes control before submission. Fable’s reported score therefore includes Opus-generated work on those tasks, which complicates direct comparisons with models evaluated without that fallback.

Controls for production use

VCB 1-100 measures the maintenance work common in established codebases: extending inherited software, preserving earlier decisions, migrating data, and enforcing policies across multiple screens. Scores below 30% indicate that current agents lose functional continuity quickly under sequential changes.

Teams deploying coding agents can contain that failure mode with a maintenance loop built around concrete safeguards:

1. Run each change in an isolated branch or sandbox with a recoverable snapshot.
2. Convert product requests into executable acceptance checks.
3. Retain regression workflows for privacy, identity, accounting, permissions, and data integrity.
4. Test the running application after every iteration and block merges when any workflow fails.
5. Define automated handling for clarification requests, refusals, timeouts, and model handoffs.
6. Track cost per accepted, regression-free change when comparing models.

The benchmark’s strongest operational signal comes from the early failures: even leading agents need external tests, rollback points, and supervision before their changes reach a shared branch or production environment.