Warp Scorers grade coding agents at fleet scale

Warp has added Scorers to its Factories platform, giving teams an automated way to evaluate coding-agent runs. An observer LLM reads each selected transcript, including its tool calls, and grades the work against a user-defined rubric.

The resulting scores feed dashboards, regression tracking, benchmarks, and self-improvement workflows. This gives teams a way to monitor hundreds of agent runs without requiring a human to inspect each one, while keeping proposed code and configuration changes subject to human approval.

Inside a scoring run

Each Scorer combines a rubric with a sampling policy. After a coding-agent run finishes, Warp checks which agent produced it and whether the run falls within the configured sample. A cloud observer agent then receives the full conversation and tool-call history before assigning a grade based on the rubric.

The rubric is a prompt that defines the criteria the observer should apply. Teams can configure several controls:

• Scope: which agents and runs are eligible for scoring
• Sample rate: how many eligible runs the observer evaluates
• Cadence: when scoring jobs start, with Warp defaulting to every couple of hours
• Batch size: how many runs each observer job processes

Sampling and batch size directly affect inference spending because every observer run consumes model capacity in addition to the original coding-agent run.

Four rubrics out of the box

Warp includes the scorers it uses for its own agents. The starter set covers four common evaluation dimensions:

The efficiency rubric illustrates how these evaluations work. It penalizes behavior such as rereading information already available in context or taking extra cycles to reach the correct solution. Warp grades its implementation agent from highly inefficient to highly efficient, using a pass threshold of 1.0. Runs below that threshold can enter the self-improvement workflow.

From failed grade to reviewed change

Warp’s self-improvement workflow groups failed runs, looks for recurring causes, and starts follow-up runs that propose fixes. Those fixes can target application code or the factory definition, including model selection, context, and agent skills.

1. A completed run is selected for scoring.
2. An observer grades it against one or more rubrics.
3. Failed runs are grouped by pattern.
4. A self-improvement agent proposes a change.
5. A human reviews the proposal before adoption.

Proposals commonly arrive as pull requests against factory.yaml or agent skill files, with the failed runs cited as evidence. Warp does not automatically merge them.

Production runs become an evaluation stream

LLM-as-a-judge is an established evaluation pattern in which one model grades another model’s output. Warp packages that pattern as persistent infrastructure attached to agent runs, replacing isolated release-time evaluations with a continuous stream of production measurements.

Aggregate scores can reveal regressions in the same way application monitoring exposes changes in latency or error rates. Warp’s dashboards chart rubric scores over time and support drill-downs into the runs behind a decline, such as a drop in redundant-test quality.

Cost metrics alone can push a factory toward cheaper models or smaller context windows while allowing defect rates to rise. Tracking compliance, efficiency, verbosity, and quality alongside spending makes those trade-offs visible.

Scorers and Benchmarks serve different jobs

Scorers and Benchmarks share the same grading system but operate on different sources of agent work:

Benchmark runs recreate curated tasks inside sandboxes with their original environments. Git restores both the code and the factory configuration from the relevant point in time, keeping agent definitions and skills consistent across experiments. Built-in and custom scorers then compare dimensions such as cost, quality, and verbosity.

Access, compatibility, and cost

Scoring is included with Warp Factories, which remains in early access. Warp says qualified companies can receive up to $10,000 in preview credits during the closed preview.

The platform is model-agnostic and supports Warp Agent, Claude Code, and Codex harnesses. A harness is the software layer that runs the coding model and manages its tools, context, and task execution. Scorers can therefore evaluate runs across those supported environments.

Limits to budget for

Scorers inherit the weaknesses of LLM-based judges. Grades can vary between runs, shift with small changes to rubric wording, and favor the judging model’s preferred style. Teams should calibrate rubrics against a human-reviewed sample before treating score changes as evidence of a regression.

Observer runs also add inference costs to the original agent bill. Sampling reduces that expense, though smaller samples can miss uncommon failures. Self-improvement proposals depend on the patterns an observer can infer from the available failed runs, which makes Warp’s human review step an important safeguard.

Selected transcripts and tool-call records are processed by a cloud observer agent. Teams working with sensitive repositories should review Warp’s data-handling, retention, and security terms before enabling broad production sampling.

At fleet scale, Scorers give engineering teams a consistent signal for whether coding-agent performance is improving or degrading. Their usefulness depends on the quality of the rubrics, the reliability of the judging model, and a sampling policy that balances coverage against cost.