Skip to main content
Back

Z.ai's GLM-5.3-Flash 320B Model Now Runs on Just Two Consumer GPUs

An independent quantizer squeezed Z.ai's 320B-parameter GLM-5.3-Flash into a 4-bit checkpoint that runs multimodal inference on two workstation GPUs.

Model, Full Access
  • Independent quantizer brandonmusic released GLM-5.3-Flash-tr3-4bpw, a 4-bit EXL3 quant of Z.ai's 320B MoE.
  • Runs on two RTX PRO 6000 Blackwell 96GB cards using a custom vLLM/B12X build for SM120.
  • Language-only MTP3 profile exposes 1.37M KV tokens and 10.5x concurrency at the tested ceiling.
  • Measured 145 to 151 tok/s single-stream decode from empty to 64K context with DFlash2 speculation.
  • NVFP4 MLA KV divergence of 0.055 nats vs BF16 teacher; fails LAVD-low quality gate at 3/10.
  • Ships with hash-verified reproducibility bundle, embedded provenance manifest, and custom ShapleyMCG 1.0 license.

Community quantizer brandonmusic has published a 4-bit checkpoint of Z.ai’s GLM-5.3-Flash, a 320-billion-parameter multimodal mixture-of-experts model. The Hugging Face release has drawn close to 10,000 downloads and includes measured benchmarks, runtime containers, launch configurations, and reproducibility bundles.

GLM-5.3-Flash contains 320 billion total parameters but activates about 18 billion for each token. Its 45-layer language model routes every token through 8 of 288 experts and combines KDA linear attention with sparse MLA attention, which compresses the key-value state used during inference. The native FP8 model supports text, images, video, a declared 1,048,576-token context window, and one multi-token-prediction draft layer.

Two GPUs, three memory budgets

The release uses uniform K4 EXL3/TR3 quantization and targets SM120 Blackwell hardware. Tests ran on two RTX PRO 6000 Blackwell GPUs with 96 GB of memory each. Serving requires the project’s custom vLLM fork because stock upstream vLLM lacks the necessary kernels and runtime changes.

Three supplied profiles divide GPU memory among model weights, vision components, speculative decoding, and the KV cache. The KV cache stores attention state as prompts and generated sequences grow, so its capacity determines practical context length and concurrency.

Profile Vision Speculator KV capacity Reported concurrency
Multimodal DFlash2-7 Enabled External 7-layer draft 129,473 tokens 1.32x
Language-only DFlash2-7 Disabled External 7-layer draft 184,619 tokens 1.88x
Language-only MTP3 Disabled Built-in MTP head 1,376,256 tokens 10.50x

Disabling vision increases the DFlash KV pool by roughly 43%. Removing the external draft model and using the built-in multi-token-prediction head raises aggregate KV capacity above 1.37 million tokens. That figure covers the cache pool across requests; individual requests remain subject to the model and profile limits.