Understanding the “Harness Tax” Behind Coding Agents
21 combos, 60 tasks, 5x cost swing, and a 4-tool harness on the Pareto frontier
- Same model costs up to 5x more across harnesses without a matching success gain, per HarnessTax across 21 model-harness combinations.
- Claude Fable 5 solved 97.8% in Claude Code and 96.7% in Pi for $1.33 against $0.67 per attempt, a 1.1-point gap for roughly double the bill.
- Both averaged 15 turns, but Claude Code carried more than 10 times the initial context from longer instructions and larger tool definitions.
- Pi gives the agent
read,write,editandbashand still holds the cost-success Pareto frontier on both benchmarks. - An alternative harness posted top success in 9 of 12 comparisons, including GPT-5.6 Sol at 83.3% and $0.42 with Pi against 78.9% and $0.76 with Codex CLI on Terminal-Bench 2.0.
Two developers can use the same model on the same coding task and end up paying very different inference costs.
The difference can come from the harness. Whether you use Codex, Claude Code, Pi, or some other harness can make a huge difference.
A new study finds that simple harnesses can be surprisingly competitive, more complex setups can add substantial cost for small gains, and models do not always perform best inside their own vendor's harness.
Today we look at what that means when choosing a coding-agent stack.
How to avoid the harness tax when choosing coding agents
Giving a coding task to the same model but with different harnesses can change its inference cost by as much as 5x without producing a comparable change in task success. That is the key finding of HarnessTax, a new study that tests how much the software surrounding an AI model affects coding-agent performance and cost.
The harness is the layer that determines the model's instructions, tools, memory and context management, and how the agent executes a task. When choosing between Claude Code, Codex CLI, Pi and other coding agents, you're choosing two things at once: a model and the harness.
The study suggests three useful rules of thumb:
- Harness choice can move cost much more than aggregate success
- A minimal harness can compete with feature-rich alternatives
- A model does not necessarily perform best in the harness built by its own provider
For developers, the key lesson is to evaluate the model and harness as one system instead of accepting the default pairing.
What HarnessTax tested
The researchers tested a total of 21 model-harness combinations, composed of seven models and three harnesses: Claude Code, Codex CLI and Pi. They randomly selected 30 tasks from SWE-bench Lite, which tests an agent's ability to fix software issues, and 30 from Terminal-Bench 2.0, which tests command-line tasks.
The researchers measured both task success and token cost based on API pricing. This makes it possible to see whether one harness causes the same model to consume more tokens instead of mixing that effect with differences in product pricing.
Finding 1: The harness can move the bill more than the score
On SWE-bench Lite, Claude Fable 5 solved 97.8% of attempts with Claude Code and 96.7% with Pi. But an average attempt cost $1.33 with Claude Code and $0.67 with Pi. That means you're paying roughly double the cost for an observed success difference of 1.1 percentage points.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves
Comments
