Fair Cast Stops bf16 Exports From Silently Killing RL Training Gains
A one-line change to how fp32 weights get cast to bf16 recovers RL policy updates that round-to-nearest was silently throwing away.
- Round-to-nearest fp32 to bf16 conversion silently discards most RL updates; only 0.4 to 0.5% of weights change per step.
- The served rollout policy lags the fp32 optimizer by 50 to 150 steps, invisible to existing mismatch fixes.
- Fair Cast swaps round-to-nearest for stochastic rounding or error feedback at the export call.
- Across 25 seed pairs on Qwen3-0.6B to 4B, Fair Cast wins 18, ties 6, loses 1, mean gain 6.1 points.
- Export cost is 22 to 38 ms for 1.7B parameters, with no new hyperparameters introduced.
- Complementary to FP16 approach that eliminates mismatch by changing precision end-to-end.
Reinforcement learning post-training can discard most optimizer updates when an fp32 master policy is exported to bf16 for inference. In “bf16 Rounding Silently Discards Most RL Policy Updates”, Andrew and William Gao measure this loss and propose Fair Cast, an export method that preserves updates in expectation. The Fair Cast paper reports higher benchmark scores, closer agreement between checkpoints and served models, and an export cost of 22 to 38 milliseconds for 1.7 billion parameters.
Where the updates disappear
Many LLM reinforcement learning systems divide one policy across two numerical formats. The optimizer maintains fp32 master weights for accurate updates, while an inference engine receives bf16 snapshots and generates the rollouts used as training data. A typical synchronization cycle includes:
- The optimizer updates the fp32 master weights.
- The training system converts those weights to bf16.
- The inference engine serves the bf16 policy and generates rollouts.
- The trainer computes a policy-gradient loss from those rollouts.
Bf16 stores seven explicit mantissa bits, so adjacent representable values can be farther apart than an AdamW update at learning rates between 1e-6 and 1e-5. Round-to-nearest leaves a served weight unchanged whenever the fp32 update remains below half of the local bf16 spacing. The master weight must accumulate several updates before crossing the next rounding midpoint.
On Qwen3-1.7B, the paper finds that only 0.4% to 0.5% of served weights change after each optimization step. The resulting bf16 policy trails the fp32 master by an estimated 50 to 150 steps, so the model generating rollouts can be substantially older than the optimizer state recorded for that step.
Why existing corrections miss the lag
Round-to-nearest creates correlated error in this regime. Small updates repeatedly map to the same bf16 value, and the discarded portions accumulate in the fp32 master until a rounding boundary is crossed. The served policy therefore changes through delayed jumps rather than tracking each optimizer step.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
