humans& Open-Sources the First Stable 4-Bit RL Training Recipe at 1.73x Speedup

Running reinforcement learning at 4-bit precision sounds like a recipe for disaster. The quantization errors that are merely annoying in pretraining become actively destabilizing in RL, where the policy being sampled and the policy being trained can drift apart fast. humans&, a startup focused on long-horizon multi-agent RL, just open-sourced a complete recipe that makes it work -- stably, and without sacrificing reward curves.

Why RL makes 4-bit quantization so much harder

NVFP4 is a hardware-supported 4-bit floating-point quantization format that accelerates LLM training and inference on NVIDIA Blackwell GPUs. Compared to FP8, NVFP4 offers 2-3x higher arithmetic throughput and approximately 1.8x memory reduction. The hardware case is clear. The RL case is where things get complicated.

In pretraining, gradient signals are dense and averaged across enormous token counts, so quantization noise tends to wash out. RL is different: the gradient is already a noisy estimator because it depends on sampled rollouts, advantage estimates, reward signals, and KL regularization. Quantization noise must be small enough that it does not degrade the per-update true policy gradient signal. Stack on top of that the fact that in asynchronous RL, the model being sampled and the model being trained can diverge -- and you have a compounding instability problem that no existing open-source recipe had solved end-to-end.

humans& co-founder Yuchen He described the company's goal as training models using long-horizon and multi-agent RL. Long-horizon RL trains models to plan, act, revise, and follow through over time. Multi-agent RL trains for environments where multiple AIs and humans are in the loop. For rollouts that span dozens of training steps, the throughput-stability tradeoff is not academic -- it directly determines what you can train.

Three instabilities, three fixes

The team identified three distinct failure modes and built a targeted fix for each. All three are required together -- the experiments show that any two out of three still produces gradient norm spikes.