Does Self-Improvement Still Work on an Engineered Agent Harness?
How self-improving agent harnesses can look better in validation than they perform on unseen tasks.
- A GPT-5.6 Sol optimizer rewrote instruction-layer prompts for two SWE-bench agent harnesses by mining worker failures, with all other components (model, tools, runtime, budget) held fixed.
- The sparse harness's validation score doubled (3/10 → 6/10) after one accepted edit, but on 24 sealed unseen tasks it produced zero net gain (15/24 → 15/24), swapping one win for one loss.
- The engineered harness showed only a modest validation improvement (3/14 → 5/14) yet gained two solves on the sealed test (13/24 → 15/24), reversing the intuition that a heavily engineered harness has no room left to improve.
- Patch-level analysis showed the instruction changed the agent's search strategy rather than just search depth — one new solve used fewer API calls than the original's failed attempt — but the same behavioral-contract bias that won three tasks also caused a regression on a fourth.
- The central finding is that the selection gate, not the optimizer, is the critical failure point: a validation signal can look strong (doubled score, confirmed across two rollouts) while predicting nothing about held-out performance, arguing for larger gates, paired regression checks, and cost tracking alongside accuracy before promoting any harness edit.
One of my agent's harness edits doubled its validation score. If I had stopped there, I would have shipped it.
The sparse harness went from 3/10 to 6/10 across its validation runs, after GPT-5.6 Sol rewrote its instructions from the agent's own failures.
Then I opened a sealed set of 24 SWE-bench tasks the optimizer had never seen. The improved harness went from 15/24 to 15/24. It solved one task it had missed before and broke one it had solved before.
The other harness never looked as convincing while it was being tuned. Its one accepted edit moved from 3/14 to 5/14 on its own gate. On the sealed test, that harness went from 13/24 to 15/24.
That was the problem I wanted to understand. Once a human has already engineered an agent's harness, can letting the agent rewrite its own instructions still find anything useful, and how do you tell a real improvement from a lucky one before you ship it?
I kept the worker model, tools, runtime, evaluator, and budgets fixed, so only the instruction layer could change. What followed was a pattern that showed up three times: the numbers kept refusing to behave the way I expected them to.
Why engineered agent harnesses are a harder test of self-improvement
Self-improving harnesses are no longer a novelty. Agents can already optimize their own prompts, memory, skills, tools, and orchestration, and there are working systems that mine their own failures and propose fixes.
Prime Agent's Continual Harness, for instance, exposes a /refine pipeline that reads the agent's own trajectory and applies "the smallest relevant CRUD edit," which its authors describe as improvement that is "evidence-backed rather than arbitrary."
|
Poetiq frames the same idea more aggressively, as self-improvement whose gains compound with every step. Most such demonstrations share a quiet advantage, though: they start from a minimal or deliberately weak scaffold, which hands the optimizer a large amount of obvious headroom. Improving a bad harness is not surprising.
The harder question, and the one worth a practitioner's attention, is what happens after a human has already done the engineering. If your agent already has a carefully written workflow, does letting it rewrite its own instructions find anything real, or does it just recover engineering that weak harnesses were missing?
That is the question I set out to answer. The experiment did not answer it cleanly. Instead it kept redirecting me toward a different and more useful problem, the one those systems quietly depend on. The question is not whether the optimizer can propose a good edit. It is whether you can tell a good edit from a lucky one before you ship it. When Prime Agent calls a refinement "evidence-backed," the evidence is a gate. That gate is what I ended up studying.
More prompt engineering did not produce a better agent harness
Before any self-improvement could happen, I needed starting harnesses to improve. The setup, in one line: a fixed coding model does the work, and a second model rewrites the instructions it runs under. This section is about the harnesses I gave the first model to start from, and the assumption I had about them that turned out to be wrong.
The plan was a three-level harness ladder. Three starting harnesses, same everything else, increasing amounts of human engineering:
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves
Comments
