LMSYS Drops Miles, an Open-Source RL Framework Running 4x Faster on Blackwell GPUs

LMSYS and humans&, in collaboration with NVIDIA, have released two Blackwell-native low-precision recipes for reinforcement learning post-training inside Miles, an open-source RL framework built on SGLang and Megatron-LM. The recipes bring end-to-end MXFP8 (8-bit microscaling) and NVFP4 (4-bit) training to the full RL pipeline, open-sourced across every layer of the stack.

Why Blackwell changes the math

Previous low-precision RL work used Hopper-era FP8 recipes, where scaling factors were computed in software around the Tensor Core path. Blackwell hardware changes that equation. Normalizing NVIDIA HGX platform dense Tensor Core specs to per-GPU throughput: a B200 delivers 2.25 PFLOPS at BF16, 4.5 PFLOPS at FP8, and 9 PFLOPS at FP4. Moving to FP4 is a 4x compute multiplier over BF16 on the same chip.

Microscaling (MX) formats combine narrow floating-point data types with per-block scaling factors for fine-grained tensor quantization. The critical difference from older software-scaled FP8 is that Blackwell handles rescaling natively in hardware, which is what makes MXFP8 and NVFP4 worth targeting specifically.

Keeping rollout and training in sync

In low-precision RL, rollout, training, checkpoint conversion, and live weight updates must agree on one precision contract, or the sampler and trainer policies will diverge. If the model generating samples and the model being trained are quantized differently, you are effectively training on data from a different policy, which destabilizes learning.

RL workloads also differ from pretraining in a structural way: rollout generation dominates compute. Modern RL training may spend 70–90% of GPU time generating long sequences across thousands of parallel environments, making rollout the primary bottleneck and the primary target for precision optimization.

Recipe 1: End-to-end MXFP8

MXFP8 is a microscaling format where every 32 consecutive E4M3 values share one local E8M0 scale, and the block is one-dimensional. The recipe covers rollout, forward propagation, weight-gradient GEMMs, and data-gradient GEMMs, while selected tensors remain BF16 through precision-control rules.

One notable design choice: TransformerEngine materializes both row-wise and column-wise quantized copies of activations during quantization. This uses more memory but avoids an extra requantization step and reduces additional quantization error in the backward path. For RL, where mismatch compounds across weight updates, that trade-off is worth it.

Recipe 2: Per-token NVFP4 for MoE experts

NVFP4 is more aggressive, so the team applies it selectively. It quantizes MoE experts because they dominate model size and rollout memory traffic, while the rest of the model stays in BF16 unless explicitly configured otherwise. For a model like DeepSeek-V3, MoE experts account for roughly 97.8% of total parameters, so targeting them captures almost all the memory benefit.

The key innovation is per-token activation scaling. Standard NVFP4 uses a coarse FP32 scale over a larger tensor scope, which creates two problems for RL: per-tensor scaling can make training batch-variant, and inter-token scale sharing can leak future-token information into past-token representations. The fix is one FP32 activation scale per token, computed online and fused directly into FlashInfer's quantization kernel so it adds no extra pass.

The recipe also departs from the standard NVFP4 pretraining recipe in its backward pass. RL has a different failure surface than pretraining:

• Pretraining: Gradient signals are stable, weight updates are large. The goal is preserving the coarse optimization direction.
• RL: Gradients are noisy, rewards are high-variance, and useful updates are small and delicate. Quantization noise must stay below the true update signal; otherwise it can overwrite fragile capabilities and cause performance collapse.

To handle this, the NVFP4 recipe offers two selectable backward modes:

• High-precision backward: Forward and rollout use NVFP4 for MoE experts; backward GEMMs use the original BF16 operands.
• Dequantized backward: Backward GEMMs run in BF16 but consume BF16 dequantizations of the exact low-precision operands from the forward pass, keeping the backward consistent with what was actually computed.

The bit-exact quantizer contract

If the training and rollout sides quantize a tensor differently, the policies used for sampling and learning are no longer the same low-precision model, and the mismatch accumulates across weight updates. The team solved this by aligning the FlashInfer and TransformerEngine quantizers to the same MXFP8 and NVFP4 bit-level contract, verified with unit tests checking exact byte-level agreement across random, boundary, all-zero, and maximum-value tensors.

One practical flag matters here. For serving-only workloads, FlashInfer uses fast math in parts of the FP4 quantization path. For RL, disable it:

FLASHINFER_DISABLE_FP4_QUANT_FAST_MATH=1

Fine-grained precision control

A single global precision switch is insufficient. Some layers must stay in BF16, and that decision has to be enforced consistently across checkpoint conversion, Megatron training, SGLang rollout, and live weight export. Miles implements this through count-based and name-based BF16 exceptions, using flags like --num-layers-at-start-in-bf16 and --num-layers-at-end-in-bf16 at conversion time, and --first-last-layers-bf16 in Megatron training.

Keeping the last 15% of layers in BF16 meaningfully reduces train-inference mismatch. For MLA-based models like DeepSeek-V3, the kv_b_proj projection layers are also kept in BF16, because MXFP8's one-dimensional block structure interacts badly with MLA's absorbed and non-absorbed contraction axes.

Results on Qwen3-30B-A3B

All experiments ran synchronous GRPO-style RL on Qwen3-30B-A3B with 8x B200 GPUs, split 4/4 between rollout and training, using the dapo-math-17k dataset with 8 rollout samples per prompt and a maximum response length of 8,192 tokens. Six configurations were compared:

• BF16 baseline
• End-to-end MXFP8
• MXFP8 with high-precision backward
• MXFP8 with dequantized backward
• NVFP4 with high-precision backward
• NVFP4 with dequantized backward

All five low-precision reward curves closely track the BF16 reward curve, while MXFP8 and NVFP4 reduce rollout time. The training-side gap for NVFP4 reflects a current implementation limitation: TransformerEngine applies per-token FP32 scaling as a separate PyTorch operation rather than a native per-token NVFP4 GEMM path with scaling fused into the kernel epilogue. Fused cuDNN frontend kernels have been implemented and upstreamed; TransformerEngine integration is still pending.

Memory savings from high-precision backward mode are also substantial. For NVFP4 linear layers, switching from default to high-precision backward cuts backward-pass allocated memory by 72.6% and end-to-end allocated memory by 70.4%.

What is still being worked on

• Extra BF16 weight copy: Megatron still saves an additional BF16 weight copy even when rollout and training execute the same low-precision recipe, increasing memory consumption and limiting the practical benefit of the low-precision path.
• Gradient spikes: The high-precision-backward NVFP4 variant can still show occasional gradient spikes; dequantized backward reduces but does not eliminate them.
• Weight-update interface: Low-latency FlashInfer backends require padding, swizzling, and backend-specific weight layouts that complicate live RL weight updates and RDMA.

How to use it

Miles is an open-source RL framework for large-scale post-training of language and multimodal models, building on the SGLang and Slime RL ecosystem and targeting production-grade RL pipelines. The NVFP4 recipe is available now. To reproduce the per-token NVFP4 high-precision-backward setup, set these environment variables before launching:

NVTE_NVFP4_ROW_SCALED_ACTIVATION=1
NVTE_BACKWARD_OVERRIDE=high_precision
NVTE_NVFP4_DISABLE_2D_QUANTIZATION=1
NVTE_NVFP4_DISABLE_RHT=1
NVTE_NVFP4_DISABLE_STOCHASTIC_ROUNDING=1
TRTLLM_DISABLE_FP4_QUANT_FAST_MATH=1
FLASHINFER_DISABLE_FP4_QUANT_FAST_MATH=1
SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1

Switch NVTE_BACKWARD_OVERRIDE=dequantized to select the dequantized-backward variant. The full recipe requires Blackwell hardware (B200 or newer). Contributions are upstreamed across SGLang, TransformerEngine, FlashInfer, Megatron, and cuDNN frontend, so the stack is available through each project's main branch. humans& also uses the same recipe family in production for large-scale, long-context, multi-agent asynchronous RL research.