Score centering targets policy drift in LLM reinforcement learning

Reinforcement learning for large language models often uses one system to generate rollouts and another to compute gradients. Small numerical differences between those systems can destabilize training, even when both load the same model weights. In a new paper, Together AI researchers Martin Marek and Max Ryabinin trace that instability to a biased policy-gradient term and propose an additive correction called score centering.

The method subtracts the sampler’s expected score at each token prefix. This removes the bias without importance-ratio clipping, adds less than 1% wall-clock overhead in the reported experiments, and can be combined with existing importance-sampling methods.

One policy, two numerical paths

Production RL stacks commonly separate rollout generation from optimization. An inference engine such as vLLM or SGLang samples tokens, while a training framework runs forward and backward passes. The two systems can produce different next-token distributions from the same checkpoint because of:

• Different numerical precision, kernels, or reduction orders
• Autoregressive inference paired with sequence-parallel training
• Quantized weights, activations, or key-value caches
• Stale rollout workers that generate from older checkpoints

Supervised fine-tuning often tolerates much larger differences because its data distribution stays fixed. A model can learn from another model’s outputs through ordinary maximum-likelihood training. Online RL creates a feedback loop: the updated trainer produces the next sampler, whose numerical approximation then influences the following update.

Marek and Ryabinin isolate that feedback in a Qwen3-1.7B Countdown experiment by perturbing the sampler’s weights. Online training with uniformly positive rewards proves least stable, even though positive labels are benign in offline training. The result points to a reward-dependent bias that compounds as the sampler is refreshed.