Skip to main content
Back

NYU Study Reveals Why Llama Fails Where Qwen Thrives Under Weak AI Training

A new study pinpoints why reinforcement learning with weak reward signals works for some LLMs and fails for others, and offers a concrete recipe to fix it.

  • New study identifies training reward saturation dynamics as the key predictor of RLVR generalization under weak supervision.
  • Reasoning faithfulness, not output diversity, distinguishes models that learn from models that memorize during RL.
  • Qwen models generalize from as few as 8 examples on math and science; Llama saturates in under 100 steps and fails.
  • Continual pre-training plus SFT on explicit reasoning traces recovers Llama-3.2-3B across all weak supervision settings.
  • Proxy rewards like majority voting cause reward hacking and policy collapse in most non-math-specialized models.
  • Code, checkpoints, and project page all open source at GitHub and HuggingFace.

Reinforcement learning with verifiable rewards, or RLVR, has become the dominant recipe for teaching large language models to reason. Feed the model math or science problems, check whether the final answer is right, hand out a binary reward, and watch benchmarks climb. The reports coming out of the last year have been strange, though: one paper shows RLVR working from a single training example, another shows it working with random rewards, and a third shows it collapsing entirely on the wrong model family. A new study from NYU, UCLA, and Google tries to make sense of the mess.

The paper, When Can LLMs Learn to Reason with Weak Supervision?, runs a controlled sweep across Qwen and Llama models on math, science, and graph-reasoning tasks under three flavors of weak supervision: scarce data, noisy rewards, and proxy rewards. The authors find that generalization is governed by saturation dynamics. Models with extended pre-saturation phases generalize from as few as 8 examples, tolerating noisy rewards and even proxy rewards, while rapidly saturating models fail. The root cause, they argue, is whether the model's intermediate reasoning actually justifies its final answer before RL starts, rather than exploration or diversity.

Why weak supervision is suddenly the whole ballgame

RLVR's original selling point was that you only need a checker, not a labeler. As frontier models start solving problems their human supervisors cannot verify, the checker itself becomes the bottleneck. Stronger LLMs make reliable reward signals harder to build, which is why the authors study when RLVR generalizes under three realistic degradations of the ideal setup:

  • Scarce data: only a handful of problems with ground-truth answers, sometimes as few as 8.
  • Noisy rewards: a verifier that mislabels a fraction of examples, simulating imperfect graders.
  • Proxy rewards: no verifier at all, just self-generated signals like majority voting across rollouts or the model's own confidence in its tokens.