Where Agent Capability Actually Comes From

You ask a coding agent to fix a failing test. It inspects the error, edits a file, runs the tests, and fails again. After several attempts, the task remains incomplete. Should you switch to a stronger model, give it more attempts, or change how it selects and executes commands?

The failed runs don't tell you which option would help. The model might never generate a useful fix, or it might produce one occasionally but fail to reach it consistently. It could also generate a useful action that the agent fails to select. Each explanation points to a different problem, but the final benchmark score doesn't distinguish between them.

Before investing in additional training or inference compute, you need to understand what the model can produce, how reliably it produces successful trajectories, and which actions the agent actually executes.

Two recent papers examine these problems from different directions.

A 9-billion-parameter AI agent solved 50% of tasks on TerminalBench-Lite. Researchers increased its success rate to 68.03% without retraining the model. Instead, they had it generate eight possible actions at each step and used a stronger model to select which one to execute.

Another study found a different result. On WebShop, the base Gemma-4-31B model solved 87.6% of tasks at least once across 128 attempts. Its post-trained counterpart reached only 56%, even though post-training improved its first-attempt performance.

The two experiments don’t measure the same thing, but they expose a common problem with how we interpret agent benchmarks. A successful run depends on the actions a model generates, how frequently it produces useful alternatives, and which actions the surrounding system executes. A failure doesn’t immediately tell us which part of that process is responsible.

Mid-Harness, the source of the TerminalBench-Lite result, changes how the harness selects actions. Sharpening Tax in Post-Training, the source of the WebShop result, looks at what post-training does to the range of actions a model produces. Alongside the findings, comments from Mid-Harness first author Minki Kang explain why even recognizing a useful action can be harder than his team expected.

What repeated attempts reveal about failure

A failed agent run shows that the system did not complete the task under the tested conditions. But it doesn’t prove that the same model would fail on every attempt.

In “Sharpening Tax in Post-Training,” Changdae Oh and colleagues tested this across 14 pairs of base and post-trained model checkpoints from four families, using three agentic benchmarks.

They measured two aspects of performance:

• Pass@1: How often a model succeeds on a single sampled attempt.
• Pass@K: The probability that at least one of K independently sampled attempts succeeds.

One model can perform well on the first try while another reaches more tasks when given additional attempts.

The WebShop results show the difference. In this benchmark, agents interact with a shopping environment to satisfy specified requests.

With Gemma-4-31B, the post-trained model performed better on individual attempts. But after approximately three attempts, the base model overtook its post-trained counterpart in coverage, the share of tasks solved at least once.

At 128 attempts, the base model reached 87.6% pass@128, compared with 56.0% for the post-trained model.

The researchers found that post-training shifted tasks toward two extremes: those the model could solve consistently and those it failed across all sampled attempts. Tasks that the base model solved occasionally became less likely to yield success under repeated sampling.

They call this loss of coverage under additional sampling the sharpening tax.

Post-training increases the probability of some behaviors while reducing the probability of others. This can improve first-attempt reliability while making less common successful trajectories harder to reach.

The harness also affected what the researchers could observe.

Base models often struggle with the structured tool-call formats expected by agent frameworks. To evaluate them, the researchers used a lightweight system prompt and more permissive tool-call parsing.

Across eight base models, this harness increased average pass@32 on BFCL (a benchmark that evaluates LLMs on function calling and tool usage) from 15.19% to 49.13%. However, it slightly lowered base-model performance on ACEBench (a benchmark for assessing tool-use capabilities across diverse domains) and reduced average performance when applied to post-trained models.

A harness that helps one model interact with its environment can interfere with another model’s established behavior, so model comparisons depend partly on the interface through which each model operates.

The researchers also proposed posterior-tempered group sampling (PTGS), which adjusts sampling temperature during reinforcement learning according to estimated task difficulty. It encourages more exploration on difficult tasks while retaining more conservative sampling on easier ones. In their Sokoban and FrozenLake experiments, PTGS improved both first-attempt accuracy and coverage relative to the evaluated training baselines.