LMSYS Drops Miles, an Open-Source RL Framework Running 4x Faster on Blackwell GPUs
LMSYS brings end-to-end MXFP8 and NVFP4 RL training to Miles, matching BF16 reward curves while cutting rollout time on Blackwell B200 GPUs.
- LMSYS and humans& released two Blackwell-native RL recipes for Miles: end-to-end MXFP8 and per-token NVFP4 for MoE experts, fully open-sourced.
- On Qwen3-30B-A3B with 8x B200 GPUs, all five low-precision configs match the BF16 reward curve while reducing rollout time.
- A bit-exact quantizer contract between FlashInfer and TransformerEngine eliminates train-rollout policy mismatch caused by differing quantization implementations.
- NVFP4 uses novel per-token activation scaling (not per-tensor) to prevent batch-variant training and future-token information leakage during RL.
- Fine-grained per-layer precision control is enforced consistently across checkpoint conversion, Megatron training, SGLang rollout, and live weight export.
- The Miles framework and all supporting PRs across SGLang, TransformerEngine, FlashInfer, and cuDNN frontend are publicly available now.
LMSYS and humans&, in collaboration with NVIDIA, have released two Blackwell-native low-precision recipes for reinforcement learning post-training inside Miles, an open-source RL framework built on SGLang and Megatron-LM. The recipes bring end-to-end MXFP8 (8-bit microscaling) and NVFP4 (4-bit) training to the full RL pipeline, open-sourced across every layer of the stack.
Why Blackwell changes the math
Previous low-precision RL work used Hopper-era FP8 recipes, where scaling factors were computed in software around the Tensor Core path. Blackwell hardware changes that equation. Normalizing NVIDIA HGX platform dense Tensor Core specs to per-GPU throughput: a B200 delivers 2.25 PFLOPS at BF16, 4.5 PFLOPS at FP8, and 9 PFLOPS at FP4. Moving to FP4 is a 4x compute multiplier over BF16 on the same chip.
Microscaling (MX) formats combine narrow floating-point data types with per-block scaling factors for fine-grained tensor quantization. The critical difference from older software-scaled FP8 is that Blackwell handles rescaling natively in hardware, which is what makes MXFP8 and NVFP4 worth targeting specifically.
Keeping rollout and training in sync
In low-precision RL, rollout, training, checkpoint conversion, and live weight updates must agree on one precision contract, or the sampler and trainer policies will diverge. If the model generating samples and the model being trained are quantized differently, you are effectively training on data from a different policy, which destabilizes learning.
RL workloads also differ from pretraining in a structural way: rollout generation dominates compute. Modern RL training may spend 70–90% of GPU time generating long sequences across thousands of parallel environments, making rollout the primary bottleneck and the primary target for precision optimization.
Recipe 1: End-to-end MXFP8
MXFP8 is a microscaling format where every 32 consecutive E4M3 values share one local E8M0 scale, and the block is one-dimensional. The recipe covers rollout, forward propagation, weight-gradient GEMMs, and data-gradient GEMMs, while selected tensors remain BF16 through precision-control rules.
One notable design choice: TransformerEngine materializes both row-wise and column-wise quantized copies of activations during quantization. This uses more memory but avoids an extra requantization step and reduces additional quantization error in the backward path. For RL, where mismatch compounds across weight updates, that trade-off is worth it.