humans& Open-Sources the First Stable 4-Bit RL Training Recipe at 1.73x Speedup
humans& open-sources a stable NVFP4 reinforcement learning recipe that matches BF16 training dynamics while delivering major throughput gains on Blackwell GPUs
- First stable open-source 4-bit RL recipe: humans& released NVFP4 RL training that matches BF16 reward curves, contributed across TransformerEngine, FlashInfer, and SGLang.
- Three-part fix for RL instability: per-token activation scaling (forward), dequantized backward pass, and Four-Over-Six adaptive block scaling -- all three required together.
- Major throughput gains: NVFP4 delivers up to 1.73x speedup over FP8 and up to 9x more ops/sec than BF16 on NVIDIA Rubin GPUs.
- 70% peak memory reduction in the backward pass from avoiding redundant quantized tensor copies, contributed to TransformerEngine.
- Online serving bonus: same quantization path enables direct checkpoint deployment via
--quantization nvfp4_onlinein SGLang, no calibration needed. - Bit-exact contract across training and inference: 4/6 scaling matches >99.97% of the time between TransformerEngine and FlashInfer, preventing trainer-sampler mismatch.
Running reinforcement learning at 4-bit precision sounds like a recipe for disaster. The quantization errors that are merely annoying in pretraining become actively destabilizing in RL, where the policy being sampled and the policy being trained can drift apart fast. humans&, a startup focused on long-horizon multi-agent RL, just open-sourced a complete recipe that makes it work -- stably, and without sacrificing reward curves.
Why RL makes 4-bit quantization so much harder
NVFP4 is a hardware-supported 4-bit floating-point quantization format that accelerates LLM training and inference on NVIDIA Blackwell GPUs. Compared to FP8, NVFP4 offers 2-3x higher arithmetic throughput and approximately 1.8x memory reduction. The hardware case is clear. The RL case is where things get complicated.
In pretraining, gradient signals are dense and averaged across enormous token counts, so quantization noise tends to wash out. RL is different: the gradient is already a noisy estimator because it depends on sampled rollouts, advantage estimates, reward signals, and KL regularization. Quantization noise must be small enough that it does not degrade the per-update true policy gradient signal. Stack on top of that the fact that in asynchronous RL, the model being sampled and the model being trained can diverge -- and you have a compounding instability problem that no existing open-source recipe had solved end-to-end.
humans& co-founder Yuchen He described the company's goal as training models using long-horizon and multi-agent RL. Long-horizon RL trains models to plan, act, revise, and follow through over time. Multi-agent RL trains for environments where multiple AIs and humans are in the loop. For rollouts that span dozens of training steps, the throughput-stability tradeoff is not academic -- it directly determines what you can train.
Three instabilities, three fixes
The team identified three distinct failure modes and built a targeted fix for each. All three are required together -- the experiments show that any two out of three still produces gradient norm spikes.