Mia AI Lab Runs GLM-5.3-Flash Across Two DGX Sparks at 146 tok/s
A hobbyist lab shipped a two-node vLLM stack that runs GLM-5.3-Flash at 4bpw across a pair of DGX Sparks with 900k context.
- Two-node vLLM overlay runs GLM-5.3-Flash 4bpw on 2x DGX Spark with TP=2.
- Decode hits 62.9 tok/s single-stream and 146.5 tok/s aggregate at concurrency 4.
- DFlash2 k=7 speculator lands 0.918 accept on structured prompts, much lower on prose.
- 4bpw EXL3 weights match official FP8 KLD at 54% of bytes (176 GB vs 328 GB).
- Full 900k context with a 982,612-token fp8_ds_mla KV pool at util 0.87.
- Overlay patches sparse MLA, EAGLE3 hooks, and MLA page-sharing that stock vLLM lacks.
Mia AI Lab has published an updated deployment recipe that serves GLM-5.3-Flash across two NVIDIA GB10 DGX Spark desktops through an OpenAI-compatible vLLM API. The release targets concurrent decoding, the earlier stack’s weak point, and packages the Docker overlay, launch scripts, custom kernels, and version-pinned configuration in a project repository.
GLM-5.3-Flash uses a mixture-of-experts architecture, which routes each token through a subset of the model’s expert layers. That design reduces active computation, although the full expert weights must remain available in memory. Splitting the quantized checkpoint across two Spark systems provides enough capacity for the model, its key-value cache, and speculative-decoding components.
Two Sparks become one node
The deployment serves brandonmusic/GLM-5.3-Flash-tr3-4bpw, an EXL3 checkpoint whose routed experts use uniform K4 quantization at four bits per weight. The weights occupy about 164 GiB across 120 shards. Tensor parallelism splits inference across both GB10 systems over ConnectX-7, while native sm_121a cubins provide GPU code compiled for the hardware. The head node exposes the API on port 8888.
| Component | Implementation | Purpose |
|---|---|---|
| Expert execution | exllamav3_ext.exl3_moe |
Keeps routed experts in packed trellis form and launches one fused EXL3 MoE kernel per layer, reducing per-expert launch overhead. |
| Attention | FLASHINFER_MLA_SPARSE_SM120 |
Provides sparse multi-head latent attention, or MLA, on the GB10’s SM12x architecture. |
| KV cache | fp8_ds_mla |
Stores attention state in packed FP8 because the stack lacks a sparse BF16 KV kernel for this architecture. |
| Speculative decoding | DFlash2 at k=7 |
Proposes seven future tokens for the main model to verify together. MTP at k=2 remains available as a fallback. |
The deployment depends on these custom kernels and hooks, so its image tags and package versions form part of the working configuration. Upgrading vLLM, FlashInfer, Triton, or the EXL3 extension independently can break compatibility.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
