Delta matches native coding harnesses in Zed’s tests

Zed has published benchmark results for Delta, its collaborative coding agent. In Zed’s tests, Delta’s harness matched or exceeded Codex and Claude Code on task pass rate across four frontier models, while its cost per successful task ranged from 0.80 to 1.12 times that of the native harness. The benchmark write-up also examines what pass rates omit: repository artifacts, failed tool calls, retries, and elapsed time.

Delta narrows the vendor advantage

Zed tested OpenAI’s GPT-5.6 Sol and GPT-6 Astra against Codex, then tested Claude Opus 5 and Claude Fable 5.1 against Claude Code. All four comparisons used Terminal-Bench 2.1, a suite that measures whether an agent can complete terminal-based tasks.

Across the four models, Delta’s reported cost per successful task fell between 0.80× and 1.12× the native result. Cost per pass accounts for both inference spending and the number of tasks completed successfully.

For the open-weight Kimi K3 model, Zed used the 30-task suite from FrontierHarness. Delta reportedly exceeded the other included harnesses on pass rate while remaining competitive on cost.

These are publisher-run benchmarks whose scope covers the reported tasks, models, and configurations. Independent reruns and tests across larger repositories will determine how well the findings generalize to production work.

The harness shapes the run

A coding-agent harness is the software between a model and the development environment. It defines available tools, executes shell commands, edits files, manages retries, preserves context, and supplies prompts that guide the model’s behavior.

Model vendors can train and tune their systems alongside their own harnesses while inspecting internal failures. An independent developer-tool company lacks that access, so comparable results suggest that harness design can offset some of the native vendor advantage.

A pass can hide the implementation

Terminal-Bench checks whether the environment reaches an expected end state. That scoring method can award identical passes to agents that leave substantially different code and files behind.

Reviewable implementation files give a team code it can inspect, test, rerun, and modify. A one-off shell command may satisfy a benchmark checker while leaving little reusable work in the repository.

Retries fall outside the score

Zed reports that harnesses can absorb failed tool calls and retry them before producing a final result, causing pass-rate metrics to omit the extra requests and latency. The company characterizes Claude Code’s reported tool-call failure rate as high and measures Delta’s rate at about 1.5% on Anthropic models. Zed also says both harnesses receive model tokens at the same speed, placing more of the user-visible delay in tool execution and retry handling.

DeltaDB records file versions and the history of prior agent actions. When a patch could match two identical lines, that context helps Delta select the intended occurrence. When a teammate changes a line after the model has read it, Delta can flag the stale content and instruct the model to read the file again before editing.

Half the requests on one task

In a Terminal-Bench task that counts tokens in a Hugging Face dataset, GPT-6 Astra completed the task through both Codex and Delta. The traces differed in request count, command handling, and total time.

Zed attributes the difference to a terminal tool that waits for commands to finish and a harness that can run independent operations concurrently. The example used seven fewer model requests and 32 fewer seconds, although Terminal-Bench assigned both runs the same passing result.

A fuller agent scorecard

Selecting a coding agent requires measures that capture the full development loop, including reliability, latency, and the quality of committed work. Useful evaluation criteria include:

• Task success: pass rate across representative repositories and repeated runs.
• Cost efficiency: total spending per successful task, including failed attempts and retries.
• Latency: wall-clock time, model requests, polling, and command wait time.
• Tool reliability: failed edits, rejected calls, stale reads, and recovery behavior.
• Repository quality: reviewable code, tests, reproducible scripts, and minimal cleanup.
• Collaboration: behavior when files change during a run or several contributors edit related code.

Research projects such as HarnessTax and FrontierHarness aim to measure model and harness contributions independently. Their work reflects a growing need for evaluations that expose the surrounding agent system instead of reporting a single aggregate score.

Zed says its next evaluations will move beyond Terminal-Bench and focus on the collaborative editing loop Delta was designed to handle, with results published regularly. Tasks involving concurrent edits, stale file state, and reviewable patches would test those design claims directly. Within the current benchmark scope, the results show that harness choice can change completion rate, cost, latency, and repository output while the underlying model remains the same.