NVIDIA Shrinks Qwen3.5-122B to Run on a Single Blackwell GPU

NVIDIA has published a pre-quantized NVFP4 checkpoint of Alibaba's Qwen3.5-122B-A10B, a Mixture-of-Experts model with 122B total parameters and roughly 10B activated per token. The release targets Blackwell owners who want a 4-bit checkpoint they can serve directly through vLLM without running their own calibration pipeline. It has crossed 900K downloads on Hugging Face.

How NVFP4 works

NVFP4 is a 4-bit floating-point format tied to Blackwell tensor cores. Each value is stored in E2M1 (one sign bit, two exponent bits, one mantissa bit) with symmetric per-group quantization at a group size of 16, finer than MXFP4's group size of 32. Each 16-value block stores one 4-bit number per element, one shared FP8 scaling factor, and one FP32 per-tensor scaling factor, averaging around 4.5 bits per value. The two-level scaling preserves dynamic range in weights that mix very large and very small values, which is where earlier FP4 schemes tended to fall apart.

Compared to FP16, memory footprint drops by roughly 3.5x; compared to FP8, by about 1.8x. Both figures are higher overhead than typical INT4 at block size 128, but NVFP4 earns it back in hardware: Blackwell tensor cores run matmuls directly in FP4, so there's no mid-inference dequantization step to bloat latency. INT4 inference requires unpacking weights back to 16-bit before computation; NVFP4 skips that entirely.