Mia AI Lab Squeezes a 99 GB Qwen3.8-Flash-Next onto One GPU
A single 121 GiB DGX Spark serves a 99 GB vision-language model at up to 512k context, thanks to PLE offload and FP8 KV cache tricks.
- Launcher and patch set for serving Qwen3.8-Flash-Next NVFP4 on one DGX Spark at TP=1.
- PLE table is memory-mapped with MADV_RANDOM, cutting disk read per token 24x and freeing ~2 GiB.
- FP8 KV cache patch roughly doubles KV pool to ~1.43M tokens with ~3-6% speed cost.
- Prefill peaks at 2,073 tok/s around 32k context; decode 27-37 tok/s depending on content.
- Vision and video work out of the box, but MTP speculative decode falls back for multimodal.
- Reasoning is on by default via
--reasoning-parser qwen3; disable per request for short answers.
Serving a 99 GB Qwen checkpoint on one DGX Spark
Mia AI Lab has published a self-contained recipe for serving the 99 GB Qwen3.8-Flash-Next NVFP4 checkpoint on a single NVIDIA DGX Spark. The project repository runs vLLM at tensor parallelism 1, meaning one accelerator handles the full model, and adds disk-backed per-layer embeddings, an FP8 KV cache and hardware-specific patches.
A 99 GB checkpoint occupies about 92 GiB, leaving limited space within the Spark’s 121 GiB of usable unified memory. That pool is shared by the CPU, GPU, operating system, runtime buffers and KV cache. The launcher automates that memory balance and documents several configurations that caused host lockups during development.
Configuration at a glance
| Setting | Default |
|---|---|
| Checkpoint | 99 GB NVFP4 |
| Usable unified memory | 121 GiB |
| Tensor parallelism | 1 |
| Context window | 262,144 tokens |
| YaRN context scaling | Disabled |
| Drafting | MTP 3 |
| KV cache | FP8 E4M3, 22 GiB target |
| Maximum sequences | 4 |
| API | OpenAI-compatible server on port 8888 |
| Reported startup | 10–12 minutes to /health |
All benchmark figures are author-reported from one DGX Spark. They characterize this hardware, container image, patch set and request mix.
One launcher coordinates bring-up
The repository supplies a launcher and patch generators for the vLLM files inside the container. Its start.sh script performs four main tasks:
- Calculates the GPU memory budget from current system availability.
- Builds the packed per-layer embedding table on the first run.
- Regenerates the patched vLLM files.
- Starts the serving container and a memory watchdog.
The reported 10–12 minute startup includes the work required before the health endpoint responds. Once ready, the service exposes an OpenAI-compatible API on port 8888.
PLE makes disk access selective
Per-layer embedding offload keeps the 27 GB packed PLE table in a memory-mapped file. The operating system loads requested pages into memory as inference accesses them, while the complete table remains outside the process’s resident allocation.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
