humans& Open-Sources the First Stable 4-Bit RL Training Recipe at 1.73x Speedup

Running reinforcement learning at 4-bit precision sounds like a recipe for disaster. The quantization errors that are merely annoying in pretraining become actively destabilizing in RL, where the policy being sampled and the policy being trained can drift apart fast. humans&, a startup focused on long-horizon multi-agent RL, just open-sourced a complete recipe that makes it work -- stably, and without sacrificing reward curves.

Why RL makes 4-bit quantization so much harder

NVFP4 is a hardware-supported 4-bit floating-point quantization format that accelerates LLM training and inference on NVIDIA Blackwell GPUs. Compared to FP8, NVFP4 offers 2-3x higher arithmetic throughput and approximately 1.8x memory reduction. The hardware case is clear. The RL case is where things get complicated.

In pretraining, gradient signals are dense and averaged across enormous token counts, so quantization noise tends to wash out. RL is different: the gradient is already a noisy estimator because it depends on sampled rollouts, advantage estimates, reward signals, and KL regularization. Quantization noise must be small enough that it does not degrade the per-update true policy gradient signal. Stack on top of that the fact that in asynchronous RL, the model being sampled and the model being trained can diverge -- and you have a compounding instability problem that no existing open-source recipe had solved end-to-end.

humans& co-founder Yuchen He described the company's goal as training models using long-horizon and multi-agent RL. Long-horizon RL trains models to plan, act, revise, and follow through over time. Multi-agent RL trains for environments where multiple AIs and humans are in the loop. For rollouts that span dozens of training steps, the throughput-stability tradeoff is not academic -- it directly determines what you can train.

Three instabilities, three fixes

The team identified three distinct failure modes and built a targeted fix for each. All three are required together -- the experiments show that any two out of three still produces gradient norm spikes.

1. Forward pass: per-token activation scaling

The naive approach to NVFP4 uses a single global scale computed over the entire activation tensor. This creates two problems: the same token gets quantized differently depending on what else is in the batch, and later tokens can leak information backward through the shared scale. To solve this, the team uses per-token activation scaling, where each token computes its own FP32 activation scale across the hidden dimension. This keeps quantization local to each token and avoids a separate calibration step.

2. Backward pass: dequantized gradients

The standard conservative approach runs the forward pass in NVFP4 but keeps the backward pass in BF16. This sounds safe, but it introduces a chain-rule inconsistency: the backward pass is differentiating a different function than the one used in the forward pass. The result is occasional gradient spikes. The fix -- called dequantized backward -- uses the BF16-dequantized value of the exact quantized tensor from the forward pass. The backward pass still uses BF16 operands, but the operands reflect the same NVFP4 quantization decisions that were used in the forward pass. A collaboration with NVIDIA also reduced peak memory consumption during training by 70% by avoiding redundant quantized tensor copies.

3. Weights and activations: Four-Over-Six (4/6) scaling

NVFP4 normally maps the largest representable value to ±6, which means values near 5/6 of the range can have errors up to 1/6 of the full range. The Four-Over-Six technique adaptively maps the maximum to ±4 when doing so reduces quantization error, cutting the worst-case error to 1/8 of the range. The original paper applied this only to activations during pretraining; humans& extended it to weights as well. This matters more for RL because the training starts from already-trained weights, and errors in their quantization have disproportionate impacts.

Implementing 4/6 efficiently required a custom FP16 fast path using hardware-native PTX operations, achieving a ~2.8x speedup over the naive approach while matching scaling factor choices over 99.97% of the time. Critically, the implementation is bit-exact across training (TransformerEngine) and inference (FlashInfer) -- a hard requirement to prevent trainer-sampler mismatch in the RL loop.

Selective precision: spending bits where they matter

Not every layer gets the same treatment. Following NVIDIA's pretraining work, the recipe keeps the final 15% of layers in BF16. For MoE architectures (like the Qwen3-30B-A3B used in experiments), the shared expert -- the one active for every single token -- also stays in higher precision. In DeepSeek-V3-style architectures, MoE experts account for 97% of total parameters, so aggressively quantizing MoE layers captures most of the memory benefits. The selective high-precision layers are a small but impactful exception.

What you actually get

With all three optimizations combined, the gradient norm is stable across five independent runs, and the reward curve tracks BF16 closely. Rollout time is also meaningfully reduced compared to BF16 and competitive with MXFP8. NVFP4 enables high-throughput, 4-bit mixed-precision training on NVIDIA Blackwell and Rubin platforms, achieving up to 1.73x speedup over FP8 baselines with negligible accuracy loss. The blog also notes that on NVIDIA Rubin GPUs, FP4 supports up to 9x more operations per second than 16-bit training.

A bonus side effect: the same quantization path enables online post-training quantization for serving. Instead of running a separate calibration or QAT pipeline after training, you can deploy checkpoints directly with --quantization nvfp4_online in SGLang. In an internal benchmark using GLM 5.1 as a judge on multi-turn tool-use rollouts, the correlation between FP8 deployment and NVFP4 deployment was 0.924 -- close enough for production.

Everything is open-source

The full recipe is contributed across the stack, not just as a blog post. Key PRs include:

• TransformerEngine row-scaled recipe for per-token NVFP4 MoE
• TransformerEngine dequantized backward with 70% peak memory reduction
• TransformerEngine 4/6 recipe for weights and activations
• FlashInfer per-token NVFP4 MoE inference kernel
• SGLang support for per-token NVFP4 MoE
• FlashInfer 4/6 recipe with bit-exact contract across training and inference
• SGLang online NVFP4 serving via --quantization nvfp4_online

What this unlocks

Before this work, the only open-source options for low-precision RL were INT4 QAT recipes that kept activations in 16-bit (like Kimi K2 Thinking), or MXFP4 recipes that used MXFP8 activations (like DeepSeek-V4). This approach uses NVFP4 quantization for both weights and activations -- the first stable, hardware-native 4-bit RL recipe in the open-source community.

The practical implication is significant for anyone running long-horizon RL at scale. More rollouts per GPU-hour means more signal per dollar, which compounds over long training runs. And because the recipe is a drop-in replacement for BF16 -- the team used the same hyperparameters without tuning -- adoption friction is low for teams already using TransformerEngine, FlashInfer, or SGLang.

The broader assumption this challenges: that RL training requires higher precision than inference to stay stable. With careful co-design of the quantization configuration and the training environment, 4-bit is now a viable training precision, not just a serving optimization.