Warp's WarpBench Slashes Coding Agent Costs by 63% Using Your Own Codebase

Public coding benchmarks like SWE-bench and Terminal-Bench have a well-known problem: their tasks have likely leaked into training data for most frontier models, and their leaderboards are saturated enough to be nearly meaningless for choosing which model to run on your actual codebase. Warp is proposing a fix with Factory Benchmarks, a system that builds custom coding-agent benchmarks from your team's own past agent runs, then uses the results to route work to the cheapest model that still hits your quality bar.

Warp used the tool internally to cut its cost-per-PR by more than 60%, and is now opening it up in early access to Warp Factories customers.

Replaying your own agent history

Factory Benchmarks pulls from the runs already sitting in your Warp Factories history and recreates each task in a sandbox. Git rehydrates not just the original code but the factory configuration at that point in time, so agent definitions and skills stay constant across tests. Built-in LLM-as-a-judge scorers then rate each run on cost, quality, verbosity, and any custom dimensions you define.

For their internal benchmark, called WarpBench, Warp's foreman agent searched past agent runs for a representative set of 30 tasks spanning roughly S, M, L, and XL scope levels. The tasks covered their Go/React server codebase and their Rust client, including UI changes, database updates, server business logic, and systems programming.

The 63% cost cut, in numbers

Warp tested a mix of frontier and open-weight models: Opus 5, GPT-5 Sol, Gemini 3.7, Grok 4.6, and GLM 5.3 Flash, all run through the Warp Agent harness. The full benchmark took 3 hours 46 minutes and cost $2,130.57, which is why Warp recommends running these only when new models drop or when you meaningfully change your code, skills, or MCPs.

The first iteration flagged Grok 4.6 (high) as the winner for simple UI and backend tasks at half the cost of their previous default, which had leaned heavily on Opus 5 for complex work. Cost per completed PR dropped from roughly $80 to $30, with merge rate holding flat. Internal scorer agents rating quality and correctness showed task compliance climb from 69% to 87% after the switch.

A second, broader benchmark then surfaced GPT-5.6 Sol as a better cost-quality trade-off. Warp made it the new default, projecting an additional ~25% reduction in inference spend.

From benchmark to model router

Results feed directly back into the platform through custom model routers defined in YAML. Instead of one default model, you can dispatch tasks to different models based on language, complexity, or which part of the codebase they touch. Frontend UI work might go to one model, Go server changes to another.

Warp plans to slice its own task sets by language and complexity to build more granular routing rules. Two capabilities are still on the roadmap:

• Cross-harness benchmarking, so you can compare Warp Agent against Claude Code and Codex on the same replayed tasks.
• A/B testing of model routing rules in production, since current benchmarks are retroactive and assume future tasks resemble past ones.

Strengths, costs, and constraints

The scoring covers correctness, code quality, efficiency, and custom rubrics like Figma mockup alignment or end-to-end test quality. Setup takes under 30 minutes according to Warp, and the output plugs directly into routing infrastructure. Those are real advantages over generic leaderboards.

The constraints are equally concrete. Running a full benchmark costs thousands of dollars, so it only makes sense when a model generation turns over or your codebase changes significantly. The system also requires a meaningful corpus of existing agent runs in Warp Factories, which makes it a feature for established customers rather than a general-purpose tool. And because replay is retroactive, it assumes your future task distribution resembles your past one, which the planned A/B testing is meant to address.

How to get it

Factory Benchmarks is in early access as part of Warp Factories. Qualified teams can request access and get up to $10,000 in free factory usage to seed their first benchmarks. If your team is already routing a nontrivial portion of PRs through coding agents and the inference bill is a concern, the cost math is straightforward. If you are still deciding whether to adopt an agent platform at all, this is a reason to look at Warp rather than something you can bolt onto an existing setup.