NVIDIA's NVFP4 Makes Llama 3.1 405B Train 1.73x Faster on Blackwell
NVIDIA's NVFP4 training recipe for JAX and MaxText delivers up to 1.73x faster pretraining than FP8 on Blackwell GPUs, with no measurable accuracy loss.
- 1.31x to 1.73x speedup: NVFP4 training in JAX/MaxText beats FP8 on Blackwell GPUs with zero measurable accuracy loss.
- Largest gains at scale: Llama 3.1 405B on GB300 hits 1.73x faster throughput vs FP8, where GEMM compute dominates over communication overhead.
- Five-technique recipe: Micro block scaling, E4M3 scale factors, Random Hadamard Transform, 2D weight scaling, and stochastic rounding work together to stabilize 4-bit training.
- Attention layers stay at higher precision: Only MLP layers are quantized to NVFP4; attention blocks remain in higher precision to avoid softmax noise amplification.
- Available now: Recipe is live in the JAX-Toolbox GitHub repo via the public
ghcr.io/nvidia/jax:maxtextcontainer. - Blackwell-only: NVFP4 requires 5th-gen Tensor Cores with native FP4 silicon support; Hopper and older architectures are not supported.
Pre-training a frontier LLM is a race against the clock. When training spans trillions of tokens across thousands of accelerators, every percentage point of step time can add up to days of training and substantial compute costs. NVIDIA just published a full recipe and benchmarks showing that switching from 8-bit (FP8) to 4-bit (NVFP4) precision during JAX-based pretraining delivers 1.31x to 1.73x faster throughput, with loss curves that are statistically indistinguishable from the FP8 baseline.
What just happened
NVIDIA released an end-to-end NVFP4 pretraining recipe for JAX, implemented in MaxText, NVIDIA's scalable LLM training library. The recipe is available now via the public ghcr.io/nvidia/jax:maxtext container, and it targets NVIDIA Blackwell hardware (GB200 and GB300). The benchmarks were run on Llama 3 8B and Llama 3.1 405B, covering both the GB200 Grace Blackwell Superchip and the newer GB300 Grace Blackwell Ultra Superchip.
The format itself: what is NVFP4?
NVFP4 is a purpose-built 4-bit floating point format developed for the Blackwell GPU architecture, aimed at improving efficiency of both training and inference tasks. The challenge with 4-bit representations is that you only have 16 unique values to work with, so a single outlier in a weight block can corrupt the entire quantization range. NVFP4 solves this with a two-level scaling strategy.
NVFP4 uses a two-level scaling trick: each value packs into an E2M1 format (1 sign bit + 2 exponent + 1 mantissa = 4 bits), groups of 16 values share one FP8 (E4M3) scale factor, and a single FP32 scale then applies across the whole tensor. This micro-scaling restores lost magnitude info that plain 4-bit formats discard, giving much better accuracy than standard FP4 or INT4 while staying very compact at roughly 4.5 bits effective per value.
This hardware-software co-design makes NVFP4 available only on Blackwell-based GPUs like the B300, B200, RTX Pro 6000/5000/4000, and RTX 5000 series. Previous architectures like Hopper or Ampere can emulate low-precision formats but lack dedicated datapaths, resulting in lower efficiency and higher effective latency. On the GB300, native hardware support of NVFP4 delivers 7x GEMM throughput compared to native FP8 precision on the NVIDIA Hopper.
The five-ingredient training recipe
Getting 4-bit training to converge stably at scale is not trivial. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision poses challenges to training stability, convergence, and implementation, notably for large-scale models trained on long token horizons. NVIDIA's recipe combines five techniques that together make it work:
- Micro block scaling (16-element blocks): Smaller blocks mean a single outlier value has less influence over the shared scale factor. NVFP4 uses 16-element blocks, half the size of the competing MXFP4 format's 32-element blocks. In an 8B-parameter, 1T token experiment, MXFP4 required ~36% more tokens to match NVFP4's final loss.
- E4M3 block scale factors: Unlike MXFP4's power-of-two E8M0 scaling, NVFP4 uses mantissa bits in its scale factors, layered under a per-tensor FP32 scale. This gives finer-grained control over the representable range.
- Random Hadamard Transform (RHT) on WGRAD inputs: A Hadamard transform is a mathematical rotation that spreads outlier values across all elements of a block before quantization, making them easier to represent in 4 bits. The method integrates Random Hadamard transforms to bound block-level outliers, employs a two-dimensional quantization scheme for consistent representations across both the forward and backward passes, and utilizes stochastic rounding for unbiased gradient estimation. RHT is only applied to the weight gradient (WGRAD) pass, not forward or backward activation passes, to avoid breaking 2D-scale consistency.
- 2D weight scaling: One FP8 scale per 16x16 weight block ensures the forward pass (FPROP) and its transposed backward pass (DGRAD) use the same scale, keeping the two passes numerically consistent.
- Stochastic rounding on gradient quantizers: Standard round-to-nearest can silently crush tiny gradient updates to zero. Stochastic rounding (SR) randomly rounds up or down with probability proportional to the fractional part, keeping small updates statistically unbiased. Both modes are native to Blackwell FP4 conversion instructions.
Crucially, GEMM operations in transformer MLP layers are quantized to NVFP4, while attention blocks remain at higher precision to avoid softmax quantization noise amplification. Because MLPs account for the majority of training FLOPs, this captures most of the speedup without risking convergence in the numerically sensitive attention layers.
The numbers
The benchmarks were run using MaxText pretraining on Llama 3 8B and Llama 3.1 405B, with a sequence length of 8,192 and 50 steps inside the public MaxText container. Here are the results:
| Model | Hardware | FP8 TFLOPs/GPU | NVFP4 TFLOPs/GPU | Speedup |
|---|---|---|---|---|
| Llama 3 8B | GB200 | 1497 | 2017 | 1.35x |
| Llama 3 8B | GB300 | 1759 | 2301 | 1.31x |
| Llama 3.1 405B | GB200 | 1557 | 2241 | 1.44x |
| Llama 3.1 405B | GB300 | 2103 | 3633 | 1.73x |
The largest relative gains are with the 405B configurations (1.44x on GB200, 1.73x on GB300), where the per-step GEMM mass dominates FSDP collective overhead and a precision-level speedup translates directly into wall-clock savings. On the accuracy side, the Llama 3 8B loss curves for NVFP4 and FP8 were tracked across 10,000 pretraining steps. Both descend the same curve from ~12.2 nats to ~3.9 nats, with a converged-regime mean gap of just +0.026 nats, well inside step-to-step noise.
The broader research backing this up is substantial. NVIDIA validated the approach by training a 12-billion-parameter model on 10 trillion tokens, the longest publicly documented training run in 4-bit precision to date. The model achieves an MMLU-pro accuracy of 62.58%, nearly matching the 62.62% accuracy achieved through FP8 pretraining.
How to use it
The MaxText NVFP4 recipe is available in the JAX-Toolbox GitHub repository. The launch script trains Llama 3 8B with NVFP4 on Blackwell. To enable it, set the quantization flag in MaxText to switch into the NVFP4 path. Two modes are exposed:
quantization=te_nvfp4: NVFP4 with Random Hadamard Transform. Recommended when convergence quality matters most.quantization=te_nvfp4_no_rht: NVFP4 without RHT. Lowest overhead, but may degrade convergence quality on some workloads.
A minimal example of the Llama 3 8B MaxText NVFP4 training script looks like this:
RUN_SETTINGS="-m maxtext.trainers.pre_train.train maxtext/configs/base.yml \
run_name=debug_run \
model_name=llama3-8b \
dtype=bfloat16 \
max_target_length=8192 \
per_device_batch_size=4 \
quantization=te_nvfp4_no_rht"To produce the FP8 baseline for comparison, swap quantization=te_nvfp4_no_rht for quantization=te_fp8_delayedscaling. The recipe requires JAX, NVIDIA Transformer Engine, and the required CUDA/cuDNN libraries, all bundled in the public ghcr.io/nvidia/jax:maxtext container.
The bigger picture
NVIDIA's achievement marks the first known example of stable, large-scale training with FP4 precision and shows that NVFP4 can substantially reduce memory usage and computational cost without degrading model quality. The field has been on a steady march: BF16 gave way to FP8 as the standard training format, and now FP4 is proving viable for full pretraining runs at scale.
The implications for AI infrastructure are significant. NVIDIA Blackwell Ultra GPUs provide peak dense NVFP4 throughput up to 15 petaFLOPS, 3x that of FP8 on the same GPUs. And looking forward, the NVIDIA Rubin platform delivers large leaps in NVFP4 capability, offering 35 petaFLOPS of NVFP4 training compute and 50 petaFLOPS of NVFP4 Transformer Engine inference compute, a 3.5x and 5x leap compared to Blackwell, respectively.
There is one hard constraint to keep in mind: ablation studies show that fully NVFP4 models diverge. Stable training requires keeping some layers in BF16, particularly near the end of the network, to mitigate NVFP4 quantization error. Maintaining the final four transformer layers in BF16 proved sufficient. And the entire approach is gated on Blackwell hardware. Anyone still running Hopper or older will need to wait for hardware upgrades before this recipe applies to them. But for those already operating on Blackwell at scale, the math is compelling: the same cluster, the same model, the same hyperparameters, and up to 73% faster training, for free.