humans& Open-Sources the First Stable 4-Bit RL Training Recipe at 1.73x Speedup

humans& open-sources a stable NVFP4 reinforcement learning recipe that matches BF16 training dynamics while delivering major throughput gains on Blackwell GPUs

·
·
humans& Open-Sources the First Stable 4-Bit RL Training Recipe at 1.73x Speedup
Authorhumans&
Read2 min
SubtopicRl
  • First stable open-source 4-bit RL recipe: humans& released NVFP4 RL training that matches BF16 reward curves, contributed across TransformerEngine, FlashInfer, and SGLang.
  • Three-part fix for RL instability: per-token activation scaling (forward), dequantized backward pass, and Four-Over-Six adaptive block scaling -- all three required together.
  • Major throughput gains: NVFP4 delivers up to 1.73x speedup over FP8 and up to 9x more ops/sec than BF16 on NVIDIA Rubin GPUs.
  • 70% peak memory reduction in the backward pass from avoiding redundant quantized tensor copies, contributed to TransformerEngine.
  • Online serving bonus: same quantization path enables direct checkpoint deployment via --quantization nvfp4_online in SGLang, no calibration needed.
  • Bit-exact contract across training and inference: 4/6 scaling matches >99.97% of the time between TransformerEngine and FlashInfer, preventing trainer-sampler mismatch.

Running reinforcement learning at 4-bit precision sounds like a recipe for disaster. The quantization errors that are merely annoying in pretraining become actively destabilizing in RL, where the policy being sampled and the policy being trained can drift apart fast. humans&, a startup focused on long-horizon multi-agent RL, just open-sourced a complete recipe that makes it work -- stably, and without sacrificing reward curves.

Why RL makes 4-bit quantization so much harder

NVFP4 is a hardware-supported 4-bit floating-point quantization format that accelerates LLM training and inference on NVIDIA Blackwell GPUs. Compared to FP8, NVFP4 offers 2-3x higher arithmetic throughput and approximately 1.8x memory reduction. The hardware case is clear. The RL case is where things get complicated.

In pretraining, gradient signals are dense and averaged across enormous token counts, so quantization noise tends to wash out. RL is different: the gradient is already a noisy estimator because it depends on sampled rollouts, advantage estimates, reward signals, and KL regularization. Quantization noise must be small enough that it does not degrade the per-update true policy gradient signal. Stack on top of that the fact that in asynchronous RL, the model being sampled and the model being trained can diverge -- and you have a compounding instability problem that no existing open-source recipe had solved end-to-end.

humans& co-founder Yuchen He described the company's goal as training models using long-horizon and multi-agent RL. Long-horizon RL trains models to plan, act, revise, and follow through over time. Multi-agent RL trains for environments where multiple AIs and humans are in the loop. For rollouts that span dozens of training steps, the throughput-stability tradeoff is not academic -- it directly determines what you can train.

Three instabilities, three fixes

The team identified three distinct failure modes and built a targeted fix for each. All three are required together -- the experiments show that any two out of three still produces gradient norm spikes.

Keep reading

Don't miss what's next in AI

Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.

  • Full access to in-depth AI research breakdowns
  • Be the first to know what's trending before it hits mainstream
  • Daily curated papers, repos, and industry moves