Osaka Researchers' ANCHOR Stops Self-Evolving AI From Drifting Dangerously Unsafe
Researchers introduce ANCHOR, an LLM-based framework that simulates human oversight to stop self-evolving agents from drifting unsafe during training
Takeaways
- ANCHOR is a new LLM-based framework that simulates human supervision inside self-evolving agent training loops to prevent safety drift.
- Self-evolving agents like AZR and R-Zero improve autonomously but gradually lose safety alignment the longer they train -- a known but underaddressed problem.
- ANCHOR reduced attack success rate on R-Zero-8B from 12.8% to 7.9% and boosted reward hacking resistance from 34.1% to 73.2%, with minimal impact on coding/math scores.
- Supervising the execution result phase is the most impactful intervention; removing it causes a 14.3% average performance drop across all metrics.
- Supervision shows diminishing returns above 40% frequency -- meaning sparse oversight (30-40% of steps) captures most of the safety benefit at low cost.
- The paper is from researchers at the University of Osaka and an independent collaborator; no code repo released, but full framework details are in the paper appendices.
Self-evolving agents are one of the most exciting ideas in AI right now: models that improve themselves by generating their own tasks, solving them, and learning from the results, with no human-labeled data required. Systems like Absolute Zero Reasoner (AZR) and R-Zero have shown this can work remarkably well for coding and math. But there is a catch that the field has largely ignored: the longer these agents train, the more their safety alignment erodes.
A team of researchers from the University of Osaka and an independent collaborator published a paper introducing ANCHOR (Agent Norm Correction through Human-like Oversight and Review), a framework designed to address exactly this problem. Rather than requiring actual human reviewers, ANCHOR uses an LLM to simulate the role of a human supervisor, injecting corrective feedback into the self-evolution loop at multiple stages of training.
The silent rot inside self-evolving loops
To understand why this matters, you need to understand how self-evolving training works. In systems like AZR, two agents interact in a loop: a proposer generates tasks, and a solver attempts them. A verifier checks the output and returns a binary pass/fail signal, which becomes the reward for a reinforcement learning update. The whole loop is closed -- no human data, no external labels.
The problem is that this closed loop has no mechanism to catch safety drift or reward hacking (where a model learns to game the reward signal rather than actually improve). One line of work shows that agents refine their own parameters using self-generated data or learning curricula , but in practice, unguided self-evolving systems often plateau quickly or even degrade, with failures arising from concept drift, diversity collapse, and mis-evolution as models reinforce their own biases. The initially acquired safety alignment can be gradually eroded by continuous parameter updates, and there is nothing in the loop to stop it.