NVIDIA Shrinks GLM-5.3-Flash to 166 GiB so You Can Run It Locally
NVIDIA shipped an FP4-quantized build of Z.ai's 320B GLM-5.3-Flash that fits on two DGX Sparks while retaining nearly all BF16 accuracy.
- NVIDIA released an FP4 quantized GLM-5.3-Flash, a 320B MoE with 18B active parameters.
- Uses quantization-aware distillation (QAD) with NVFP4 experts; fits in ~166 GiB VRAM.
- Runs on 2x DGX Spark or 2x RTX PRO 6000 Blackwell with full 1M-token context.
- Accuracy: 91.3% GPQA Diamond vs 92.1% BF16; actually beats BF16 on ToolEvalBench.
- Requires community vLLM containers from Local Inference Lab until upstream support lands.
- Governed by NVIDIA Open Model Agreement on top of base GLM-5.3-Flash MIT license, commercial use allowed.
NVIDIA has published an NVFP4 checkpoint of Z.ai’s GLM-5.3-Flash. It reduces the 320-billion-parameter mixture-of-experts model to a weight footprint of roughly 166 GiB, enabling local inference on two validated Blackwell configurations. NVIDIA reports accuracy within 0.8 percentage points of the BF16 model on two reasoning and tool-use benchmarks.
166 GiB, layer by layer
The base model, available through the GLM-5 repository, contains 320 billion total parameters and activates 18 billion for each token. Mixture-of-experts routing selects a small subset of the model for each inference step, reducing computation. Serving still requires the full set of expert weights to remain available in memory, which makes quantization central to local deployment.
GLM-5.3-Flash supports a 1,048,576-token context window plus image and video input. Its hybrid architecture combines sparse and linear attention, which reduce the cost of processing long sequences compared with full attention. Manifold-Constrained Hyper-Connections, or mHC, modify residual connections to improve training stability and scaling efficiency.
NVIDIA assigns different numerical formats according to each component’s sensitivity:
| Component | Format | Purpose |
|---|---|---|
| Routed experts | NVFP4 | Stores 4-bit E2M1 values with E4M3 scales applied to small blocks. |
| Attention projections | MXFP8 | Retains more range and precision in attention-sensitive paths. |
| Normalization and selected weights | BF16 | Preserves higher precision where 4-bit errors are more disruptive. |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
