---
title: "Does Self-Improvement Still Work on an Engineered Agent Harness?"
subtitle: "How self-improving agent harnesses can look better in validation than they perform on unseen tasks."
url: "https://alphasignal.ai/news/does-self-improvement-still-work-on-an-engineered-agent-harness"
author: "Akruti Acharya"
author_url: "https://alphasignal.ai/author/akrutireports"
published: "2026-08-19T15:56:50.266Z"
updated: "2026-08-19T15:56:50.806Z"
type: "Deep Dive"
topics:
  - "Agents"
  - "Development"
subtopics:
  - "Code Agents"
  - "Agent Frameworks"
  - "Guardrails"
  - "Code Agents"
upvotes: 4
comments: 0
truncated: true
---

# Does Self-Improvement Still Work on an Engineered Agent Harness?

> How self-improving agent harnesses can look better in validation than they perform on unseen tasks.

Self-improving agent harnesses let agents mine their own failures and rewrite the instructions or code around themselves. The harder problem is deciding which changes actually deserve to persist. I let GPT-5.6 Sol optimize two SWE-bench agent harnesses to test that.

One of my agent's harness edits doubled its validation score. If I had stopped there, I would have shipped it.

The sparse harness went from 3/10 to 6/10 across its validation runs, after GPT-5.6 Sol rewrote its instructions from the agent's own failures.

Then I opened a sealed set of 24 SWE-bench tasks the optimizer had never seen. The improved harness went from 15/24 to 15/24\. It solved one task it had missed before and broke one it had solved before.

The other harness never looked as convincing while it was being tuned. Its one accepted edit moved from 3/14 to 5/14 on its own gate. On the sealed test, that harness went from 13/24 to 15/24.

That was the problem I wanted to understand. Once a human has already engineered an agent's harness, can letting the agent rewrite its own instructions still find anything useful, and how do you tell a real improvement from a lucky one before you ship it?

I kept the worker model, tools, runtime, evaluator, and budgets fixed, so only the instruction layer could change. What followed was a pattern that showed up three times: **the numbers kept refusing to behave the way I expected them to.**

## Why engineered agent harnesses are a harder test of self-improvement

Self-improving harnesses are no longer a novelty. Agents can already optimize their own prompts, memory, skills, tools, and orchestration, and there are working systems that mine their own failures and propose fixes.

Prime Agent's [Continual Harness](https://www.google.com/url?q=https://arxiv.org/abs/2605.09998&sa=D&source=editors&ust=1787156255775989&usg=AOvVaw0b8wg4M%5F2wa9ScC%5F1diHtw), for instance, exposes a `/refine` pipeline that reads the agent's own trajectory and applies "the smallest relevant CRUD edit," which its authors describe as improvement that is "evidence-backed rather than arbitrary."

| ![](https://imagedelivery.net/kokADdxuw1FCXyR2p8D2CQ/9f71e959-d75b-4bfc-3709-4b26cb492c00/public) |
| ------------------------------------------------------------------------------------------------- |

Poetiq frames the same idea more aggressively, as [self-improvement whose gains compound with every step](https://www.google.com/url?q=https://poetiq.ai/posts/rsi%5Fperspective/&sa=D&source=editors&ust=1787156255777238&usg=AOvVaw3aP%5F8BPpbC%5FIERWtCDWzrK). Most such demonstrations share a quiet advantage, though: they start from a minimal or deliberately weak scaffold, which hands the optimizer a large amount of obvious headroom. Improving a bad harness is not surprising.

The harder question, and the one worth a practitioner's attention, is what happens after a human has already done the engineering. If your agent already has a carefully written workflow, does letting it rewrite its own instructions find anything real, or does it just recover engineering that weak harnesses were missing?

That is the question I set out to answer. The experiment did not answer it cleanly. Instead it kept redirecting me toward a different and more useful problem, the one those systems quietly depend on. The question is not whether the optimizer can propose a good edit. It is whether you can tell a good edit from a lucky one before you ship it. When Prime Agent calls a refinement "evidence-backed," the evidence is a gate. That gate is what I ended up studying.

## More prompt engineering did not produce a better agent harness

Before any self-improvement could happen, I needed starting harnesses to improve. The setup, in one line: a fixed coding model does the work, and a second model rewrites the instructions it runs under. This section is about the harnesses I gave the first model to start from, and the assumption I had about them that turned out to be wrong.

The plan was a three-level harness ladder. Three starting harnesses, same everything else, increasing amounts of human engineering:

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Openai](https://openai.com/index/previewing-gpt-5-6-sol/)
- [Arxiv](https://arxiv.org/abs/2605.09998)
- [Poetiq](https://poetiq.ai/posts/rsi_perspective/)
- [Github](https://github.com/swe-bench/SWE-bench)
- [Github](https://github.com/swe-agent/mini-swe-agent)
- [Weco](https://www.weco.ai/blog/first-evidence-of-recursive-self-improvement)