Static probes forecast coding-agent performance before post-training

Choosing a base checkpoint for expensive agentic post-training remains an empirical bet. Confirming the choice can require days of reinforcement learning (RL) or supervised fine-tuning (SFT), followed by a full agent evaluation on SWE-bench. A study from NVIDIA and UC Berkeley researchers proposes three cheaper probes that score base models at the decisive point in successful coding trajectories and produce rankings that closely track downstream agent performance.

The paper, Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models, addresses a weakness in end-to-end pass@K evaluation. Pass@K samples K complete attempts and counts a task as solved when at least one succeeds. For raw base models, the result mixes coding ability with reliable tool syntax, state tracking, and interaction over many turns. Malformed tool calls can end a run before the model’s proposed fix is tested.

One-shot patches hide agent skill

Single-shot patch evaluations remove the tool protocol by presenting a fixed prompt and requesting one edit. That setup measures patch generation under supplied context. Coding agents face a longer task: they must preserve relevant state across file reads, shell commands, test results, and successive edits while the repository changes beneath them. Compressing the interaction removes capabilities that agentic post-training is designed to develop.

Find the action that flips the tests

To isolate a verifier-backed target, the researchers replay successful trajectories from an already post-trained reference agent. After every code-changing action, they apply the cumulative patch and rerun the benchmark verifier. The first action after which the repository changes from failing to passing becomes the