Zhejiang University's CoVerRL Fixes the Hidden Flaw Destroying AI Reasoning Models
A new label-free RL method trains a single model as both generator and verifier, boosting math reasoning by 4-6% while doubling self-verification accuracy.
ProTakeaways
- CoVerRL identifies the consensus trap: majority-voting RL collapses diversity and reinforces confident wrong answers over time.
- A single model alternates between generator and verifier roles, with each bootstrapping the other via bidirectional supervision.
- Beats TTRL by 4.7-5.9% on average across Qwen3-1.7B, Llama-3.2-3B, and Qwen2.5-7B on math benchmarks.
- Self-verification accuracy jumps from ~55% to over 85%, and the 3B verifier beats GPT-4-Turbo on RewardBench.
- Introduces Answer-Anchored GRPO and balanced positive/negative verification sampling to prevent verifier collapse.
- Code and project page available on GitHub and zju-real.github.io/CoVerRL.
Training reasoning models without ground-truth labels has become a hot research direction, mostly by letting a model vote on its own answers and treating the majority as truth. A team from Zhejiang University and Baidu has now shown that this approach quietly breaks itself over time, and proposed a fix called CoVerRL that turns one model into both a generator and a verifier that improve each other.
The trap hiding inside majority voting
Label-free reinforcement learning for reasoning is dominated by a technique called test-time reinforcement learning, or TTRL, where a model generates many answers to a question and the most common response becomes the pseudo-label used to compute rewards. The underlying assumption is that consistent answers across diverse reasoning paths likely reflect correct solutions.
The failure mode is subtle. Since rewards are computed against the majority answer, the training objective inherently pushes the model to produce more consistent outputs. As training progresses, output diversity systematically decreases and predictions concentrate around whatever the majority happened to pick. When that consensus is wrong, the model confidently reinforces its own mistake. The authors call this the consensus trap.
The empirical evidence is striking. Under TTRL, reward accuracy begins near 95% but steadily drops to approximately 80%, while label accuracy stagnates around 70% with no improvement. Once the model locks in on a wrong answer with high confidence, pure consensus provides no way out.