Skip to main content
Back

Mila Proves RL Teaches AI New Reasoning Skills Beyond Smarter Sampling

A new Mila study puts the sharpening hypothesis head to head with task rewards in a unified RL framework, and one clearly wins.

  • Mila paper compares distribution sharpening and task-reward RL inside a single KL-regularized training framework.
  • Sharpening favors short outputs because log-likelihood sums are non-positive, driving instability on variable-length generation.
  • Task-reward RL beats beam search and power sampling on AIME 2024/2025 and HMMT 2025 with Qwen3-4B.
  • On AIME 2024, task-reward RL raises Qwen3-4B Pass@1 from 29.5 to 43.4; sharpening RL collapses without early stopping.
  • Lower KL weight, meaning stronger reliance on task reward, improves both final accuracy and training stability.
  • Experiments run on NeMo RL with Llama-3.2-3B, Qwen2.5-3B, and Qwen3-4B on MATH and DeepScaleR.

There's an ongoing debate in the reasoning-model community about what reinforcement learning actually does to a base LLM. One camp says RL teaches genuinely new skills. The distribution sharpening camp argues RL just concentrates probability mass on outputs the base model already found plausible, meaning any gains could be replicated with a smarter sampler. A new paper from Mila, Beyond Distribution Sharpening, tries to settle it by running both paradigms inside the exact same training loop.

Two hypotheses, one training loop

The sharpening view has real evidence behind it. Recent results show that inference-time procedures which concentrate probability mass on high-likelihood trajectories can recover strong performance on reasoning tasks, and separate analyses have found that RL fine-tuning tends to reinforce already-likely responses rather than discover new ones. If that story is right, reward design is a distraction and better decoding is the real lever.

The authors express both paradigms as instances of the standard KL-regularized RL objective, then vary the reward and the KL target to isolate what each contributes. That yields four training regimes:

  • Task-Reward RL: reward is the verifier (did the math answer match?), no KL penalty.
  • Tilted Sampling: task reward plus a KL pull toward a tempered version of the base model.
  • Distribution Sharpening: reward is the base model's own log-likelihood, effectively an RL version of beam search.