NVIDIA Shrinks GLM-5.2 Memory by 1.8x With NVFP4 Without Losing Accuracy

GLM-5.2-NVFP4 is now ready to serve in vLLM. NVIDIA just dropped the official NVFP4 checkpoint of Z.ai's GLM-5.2, the 744B-parameter MoE model built for long-horizon coding and agentic tasks, and it's already deployable with a single vllm serve command. The headline promise: smaller memory footprint than FP8, same accuracy.

The model underneath

GLM-5.2 is an open-weights model from Z.ai (formerly Zhipu AI), tuned heavily for software engineering, multi-step reasoning, and tool-augmented agent work. It builds on the Mixture-of-Experts (MoE) foundation introduced with GLM-5 and GLM-5.1, extending the context window to a usable 1 million tokens while preserving strong coding performance.

It uses a MoE design with approximately 753B total parameters and roughly 40B active per token. That last number is what actually matters for compute cost: only 40B parameters fire per forward pass, not 753B. The headline change over GLM-5 / 5.1 is that Multi-Token Prediction (MTP) is extended from 3 to 5 draft tokens, lifting end-to-end throughput on reasoning, coding, and agentic workloads. MTP is a speculative decoding technique where the model predicts multiple future tokens in parallel, then verifies them, effectively getting more output per GPU cycle.

GLM-5.2 also introduces IndexShare, which reuses the same attention indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9x at a 1M context length. This is what makes a 1M-token context window practical rather than just a marketing number.

What NVFP4 actually is

NVFP4 is not your typical INT4 quantization. NVIDIA Blackwell's NVFP4 is a 4-bit floating point format designed to improve model accuracy at ultra-low precision using a two-level scaling strategy. It reduces quantization error by using a smaller block size of 16 values, compared to its predecessor MXFP4 which used 32, allowing for more localized adaptation to the data's dynamic range.

The practical payoff: NVFP4 results in a 3.5x reduction in model memory footprint relative to FP16 and a 1.8x reduction compared to FP8, while maintaining model accuracy with less than 1% degradation on key language modeling tasks. And unlike INT4, which requires dequantizing weights back to 16-bit before compute, NVFP4 avoids this bottleneck entirely. Because Blackwell Tensor Cores are designed to handle NVFP4 operations directly, tensors remain in their compact 4-bit format throughout inference.

The raw compute numbers back this up. On the B200, FP4 achieves 7,700 TFLOPS at 96.2% of theoretical peak, compared to 3,850 TFLOPS in FP8 mode. That is a 2x throughput advantage at the hardware level, before any software-level optimizations.

What NVIDIA quantized and what they left alone

Not everything in the model gets compressed. The NVFP4 checkpoint was obtained by quantizing the weights and activations of the linear operators within transformer blocks in MoE experts only. The shared expert is not quantized. Attention layers, norms, embeddings, and the router stay in higher precision. This is a deliberate tradeoff: since the MoE expert weights constitute the vast majority of model parameters in an MoE architecture, this still yields significant memory savings.

The model is NVFP4 version 1.0, quantized with nvidia-modelopt v0.46.0. NVIDIA calibrated the quantization scales using a curated dataset and benchmarked against GLM-5.2-FP8 as the baseline across reasoning, coding, and long-context tasks.

The hardware constraint you need to know

This is a Blackwell-only checkpoint. NVIDIA Blackwell GPUs (B200, GB200, or RTX PRO 6000 Blackwell) are required. NVFP4 tensor cores are Blackwell-only. Running it on Hopper (H100/H200) is not an option for the NVFP4 format itself.

Even compressed, the model is large. VRAM for weights alone is approximately 459 GB, requiring a minimum of 6x 96 GB GPUs just to hold the weights, with 8 GPUs recommended for KV cache headroom. The tested configuration is a single node with 8x RTX PRO 6000 Blackwell GPUs using tensor-parallel 8.

Serving it today

The vLLM command is straightforward:

vllm serve nvidia/GLM-5.2-NVFP4 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 131072 \
  --gpu-memory-utilization 0.80

SGLang is also supported. After installing transformers>=5.3.0, you can launch with SGLang using the modelopt_fp4 quantization flag and a chunked prefill size of 131072. Both frameworks load the checkpoint natively with trust_remote_code=True.

Why this matters for production

The FP8 vs NVFP4 tradeoff is fundamentally about what you can do with a fixed GPU budget. Quantizing the MoE expert weights to 4-bit cuts the footprint from 755 GB to around 368 GB, which frees enough memory for the complete 1M-token KV cache. The FP8 release reaches only about 500K tokens on the same hardware. That is not a marginal win: it is the difference between fitting the full context window or not.

The accuracy story holds up under scrutiny. There is no measurable quality loss. Reasoning, instruction-following, long-context, and tool-calling benchmarks all land within measurement noise of the FP8 model. NVIDIA benchmarked against GLM-5.2-FP8 as the baseline across GPQA Diamond, coding, and the AA-LCR long-context retrieval benchmark.

Where NVFP4 falls short is portability. FP8 has more production history. If your application cannot tolerate even small accuracy degradation, FP8 is the safer first checkpoint. And if your infrastructure is not Blackwell, NVFP4 simply does not apply. On any pre-Blackwell GPU, NVFP4 would require emulation and offer no benefit.

The bigger picture

This release is part of a pattern. NVIDIA has been shipping official NVFP4 checkpoints for the GLM family (GLM-5, GLM-5.1, now GLM-5.2) in close succession, each time targeting Blackwell hardware specifically. The message is clear: NVFP4 is NVIDIA's preferred inference format for Blackwell, and they are investing in making frontier open-weight models run well on it out of the box.

GLM-5.2 trades blows with Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro on benchmarks, winning some rows and losing others. The real story is that an open-weight model you can download and self-host is now genuinely competitive on agentic engineering, and it ships with the architectural tricks needed to make a 1M context practical. The NVFP4 checkpoint makes that self-hosting story more tractable on Blackwell hardware by cutting the memory bill nearly in half versus FP8, without giving up accuracy.

If you are running Blackwell GPUs and need a frontier-class coding or agentic model, the nvidia/GLM-5.2-NVFP4 checkpoint is the most memory-efficient path to full 1M-context inference available today. The NVFP4 format documentation and the base model card are good starting points if you want to understand the precision tradeoffs before committing to a deployment.