NVIDIA Squeezes 125B Qwen3.8 Flash Next onto a Single DGX Spark
NVIDIA shipped an FP4 quant of Qwen3.8-Flash-Next that fits 1M tokens of context on a single DGX Spark, with only a 0.7-point accuracy drop.
- NVIDIA released Qwen3.8-Flash-Next-FP4-Local, a 96 GiB FP4 quant of Alibaba's 125B hybrid MoE.
- Runs 1M-token context on a single DGX Spark with FP8 KV cache and PLE disk offload.
- Accuracy drops only 0.7 points on GPQA Diamond (92.2% to 91.5%) versus BF16 teacher.
- Built via quantization-aware distillation with mixed NVFP4, MXFP8, and BF16 precisions per layer.
- Requires Blackwell GPUs, Linux, vLLM, and a community container, upstream vLLM support pending.
- Community recipes report 37-65 tok/s decode and 1500-2000 tok/s prefill on single Spark.
NVIDIA has published a pre-quantized version of Alibaba’s Qwen3.8 Flash Next on Hugging Face. The FP4-Local checkpoint packages a 125-billion-parameter mixture-of-experts model with vision support into roughly 96 GiB, allowing one DGX Spark to host a 1,048,576-token context with the documented offload configuration.
Reported performance includes an 8.5-second time to first token for an SSD-streamed KV-cache path and roughly 65 tokens per second during decode in the cited configuration. Community measurements vary by workload, reaching about 37 tokens per second for single-stream prose and 86 aggregate tokens per second across four streams. These figures make local, offline deployment practical for developers who already have Blackwell hardware, though the million-token mode extends beyond the model’s trained context range.
- Checkpoint size: approximately 96 GiB
- Total parameters: 125 billion
- Activated parameters: 6 billion per token, as reported
- Native context: 262,144 tokens
- Extended context: 1,048,576 tokens with YaRN scaling
- Supported stack: Blackwell GPU, Linux, and a customized vLLM build
How 125B parameters fit in 96 GiB
The BF16 base checkpoint combines a causal language model, vision encoder, Gated DeltaNet layers, Qwen Sparse Attention, gated residual streams, n-gram embeddings, and routed experts. Its mixture-of-experts architecture stores every expert but sends each token through a subset, reducing per-token computation while retaining the storage cost of the full model.
NVIDIA reports 125 billion total parameters with 6 billion activated, alongside 51 billion parameters in the n-gram embedding system and 4 billion in the multi-token prediction head. The prediction head proposes several tokens at once for speculative decoding, which can increase generation speed when the main model accepts those drafts.
NVIDIA assigns precision by component instead of forcing the entire network into one numerical format:
| Component | Format | Implementation detail |
|---|---|---|
| Decoder routed experts | NVFP4 | E2M1 values with E4M3 block scales |
| Decoder attention | MXFP8 | Eight-bit microscaled projections |
| Vision attention linears | MXFP8 | UE8M0 scales for each group of 32 weights |
| Vision MLP and merger linears | NVFP4 | BF16 activations |
| Normalization, biases, and position embeddings |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
