NUS's TTRL-CoCoV Beats Supervised AI Training With Zero Labels
TTRL-CoCoV uses a model's own verification ability to fix label noise and prevent diversity collapse during label-free RL training, beating fully supervised methods on math benchmarks
- TTRL-CoCoV is a new label-free RL training method that beats fully supervised RL on math benchmarks by up to +5% absolute Pass@1.
- It identifies two failure modes in existing test-time RL: noisy pseudo-labels for uncertain samples and diversity collapse for confident ones.
- A three-way confidence split routes each training sample to a different strategy: verifier-filtered labels, exploration rewards, or skip -- depending on model confidence.
- Average gains over baseline TTRL: +9.8% Pass@1 and +18.7% Pass@16 across 6 math reasoning benchmarks.
- The key insight is that verification capability leads generation capability, making the model's own verifier a reliable signal when used conditionally.
- Code is open-source on GitHub; the paper is on arXiv.
Training large language models to reason better has traditionally required one thing: ground-truth labels. Reinforcement Learning with Verifiable Rewards (RLVR) -- the technique behind DeepSeek-R1 and the Qwen reasoning family -- trains models on problems with automatically checkable outcomes, optimizing them to produce correct final answers and refine their chain-of-thought generation beyond supervised fine-tuning. But what happens when you have no labels at all?
Test-Time Reinforcement Learning (TTRL) showed that common practices like majority voting yield surprisingly effective reward signals for driving RL training on unlabeled data. The idea is elegant: generate many candidate answers, treat the most common one as the pseudo-label, and train on that signal. TTRL can boost the Pass@1 performance of Qwen-2.5-Math-7B by approximately 211% on AIME 2024 using only unlabeled test data. But researchers at the National University of Singapore have now identified two critical failure modes in this approach -- and built a fix that outperforms even fully supervised RL.
Two problems hiding in plain sight
The new paper introduces TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), but the real contribution starts with a careful diagnosis of why vanilla TTRL falls short. The team focused not just on Pass@1 (accuracy on a single attempt) but on Pass@k -- the probability that at least one correct answer appears among k independent samples. Pass@k is a measure of how broadly a model can explore the solution space, which matters enormously when you're doing label-free training and need the model to keep discovering new correct paths.
Through empirical analysis, they found two distinct failure modes depending on how confident the model already is on a given problem:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
