AutoTrust's Qwen3.8 Drafter Hits 48 tok/s, Nearly 4x Faster on DGX Spark
A community-tuned drafter for Qwen3.8-27B's NVFP4 build pushes DGX Spark decode from 12.5 to 48.2 tokens per second, bit-exact lossless.
- AutoTrust ships a lossless MTP speculative-decoding recipe for Qwen3.8-27B-NVFP4, taking DGX Spark from 12.5 to 48.2 tok/s.
- Weights are bit-exact identical to NVIDIA's upstream NVFP4 release; only the drafter changes.
- Three tricks cut draft-step bytes-read 8x: 64k truncated vocab, Marlin NVFP4 draft head, quantized MTP layers.
- Enables k=8 speculation; mean acceptance length 5.29 at temperature 0.
- On B200 the recipe adds only ~4% over native MTP; benefit is bandwidth-bound hardware.
- Ships as a vLLM 0.31.0 plugin plus in-process patch, verified on sm_120/121 Blackwell.
AutoTrust AI Lab has published a third-party AutoTrust release that pairs NVIDIA’s existing Qwen3.8-27B NVFP4 checkpoint with a faster Multi-Token Prediction drafter. The 21.9 GB target-model safetensors remain unchanged; the gains come from a smaller draft vocabulary, a quantized draft head, quantized MTP layers, and vLLM integration code.
The reported single-stream decode rate on a DGX Spark GB10 rises from 12.5 to 48.2 tokens per second at temperature 0. That 3.9x headline compares tuned MTP with speculative decoding disabled. The incremental gain over Qwen’s native MTP configuration, which plateaus around 37 tok/s, is approximately 30%.
Spark jumps from 12.5 to 48.2 tok/s
AutoTrust measured one active request at a time across 24 prompts, each containing 1,024 input tokens. Decoding used greedy sampling at temperature 0.
| Configuration | Draft length | Throughput | Gain over no MTP |
|---|---|---|---|
| MTP disabled | 0 | 12.5 tok/s | 1.0x |
| Native MTP | Higher values plateau | About 37 tok/s | About 3.0x |
| AutoTrust drafter | k=8 |
48.2 tok/s | 3.9x |
Workload-specific results at k=8 range from 44.4 to 56.9 tok/s. Here, k is the maximum number of speculative tokens proposed before the target model verifies them.
| Workload | Throughput |
|---|---|
| Code | 52.9 tok/s |
| Math | 56.9 tok/s |
| Chat | 44.4 tok/s |
| Tool use | 45.2 tok/s |
A 248k-token head taxes every draft
Memory bandwidth sets the DGX Spark’s decode ceiling. Its 128 GB of unified LPDDR5X supplies 273 GB/s of raw bandwidth to the entire system-on-chip, roughly one-seventh the bandwidth available on high-end HBM accelerators. Autoregressive decoding repeatedly streams the active model weights from memory, leaving a dense 27B model in the low tens of tokens per second even with 4-bit quantization.
Speculative decoding reduces that traffic by letting a cheaper model propose several tokens. The target model checks those proposals together, accepting correct tokens and replacing rejected ones. More accepted tokens per verification pass means fewer full-weight scans for each generated token.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
