How to choose an agent harness for your workload
Your agent harness is an architectural decision that directly dictates task success and API costs. Here is how to match it to your workload.
- A study of 176 configurations across four models found that the optimal action interface is model-dependent: dedicated file/edit tools raised Nemotron-3 30B's SWE-Bench success from 10.2% to 25.2%, while Nemotron-3 550B performed better with a Bash-only shell that lets it combine operations into fewer calls.
- Explicit persistent plans primarily help weaker models complete multi-step tasks by tracking remaining steps, while for stronger models they mainly reduce verification turns and cost with slight success-rate decreases.
- Context management research shows the primary value is preventing context-limit overflows; removing older tool outputs before summarizing outperforms adding a retrieval tool, as models rarely use the retrieval option when it is available.
- Mechanisms like Action Fusion (combining an edit and follow-up command in one tool request) and deferred large-output delivery (SoL-Pi's ObservationPack) can reduce API costs by roughly 33% with only a small accuracy trade-off on reported benchmarks.
- The article recommends validating harness candidates on actual task types—including failure attempts and interruptions—tracking cost per successful task rather than average benchmark scores, and recording exact configurations for retesting when the model or workload changes.
An agent stuck in a loop trying to execute a bad shell command will burn through your API budget while accomplishing nothing. When you choose an agent harness, you decide how the model turns a request into actions. You choose which tools it can call, which information it retains when the context window becomes very long, and how it responds when you interrupt it. Those choices affect whether it completes the work and what each attempt costs.
The requirements differ by task. An agent that repairs repository issues overnight must edit files, run tests, and retain enough state to finish without help. A research assistant must preserve access to sources and accept corrections while a lookup runs. A benchmark score alone cannot tell you whether a harness meets those requirements.
Our HarnessTax analysis examines how harness choice changes cost and performance. Recent research examines the mechanisms behind those differences. One study of 176 configurations finds that dedicated tools help one model while a shell interface works better for another. Its results, alongside research on context management and execution, give you specific choices to examine before you shortlist a harness.
Choose an Action Interface Your Model Can Use
A file edit illustrates the first decision. With a dedicated edit tool, the model supplies a path and a replacement. With a shell, it must express the operation as a command or script. The shell offers flexibility, while the dedicated interface can package checks and error messages around a narrower operation.
The aforementioned study tests 176 settings across four models on SWE-Bench Verified and Terminal-Bench 2.1. For Nemotron-3 30B, dedicated tools raise SWE-Bench success from 10.2% to 25.2% versus Bash-only. In the shell condition, the model sometimes emits calls to unavailable tools. Nemotron-3 550B instead benefits from Bash-only and combines operations into fewer calls. These comparisons use a 128k window with plans and staged context management; interface checks also differ.
A call to a nonexistent tool fails before any command reaches the environment. Conversely, an interface that forces each operation into a separate request adds calls when the model can already express the sequence in a script.
For repository repair, prefer explicit file and edit tools if your model struggles to translate operations into shell commands. Consider a smaller shell-centered interface when the model already handles your repository tasks reliably. Tool count alone cannot establish either condition.
A research assistant may need tools for source lookup, document retrieval, and citation extraction. An advantage on terminal tasks gives you little reason to remove those interfaces. Choose tools around the actions your task requires, then examine whether the model uses them correctly.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves
Comments
