DeepSeek-V4.1-Flash's 748B Model Now Runs on a Single RTX 5090
A community quantization shrinks DeepSeek's 748B mixture-of-experts to 178GB, letting Project Maya run it on a single consumer GPU by streaming experts from SSD.
- New GGUF quant of DeepSeek-V4.1-Flash-1017 runs the 748B MoE locally via Project Maya.
- Maya-S matches the original's next token 92.6% of the time (KL 0.120).
- 339.5 GB total: 178 GB model weights plus 161 GB Engram n-gram tables streamed from SSD.
- Experts quantized to 2-3 bits with GPTQ-style per-expert error feedback; attention stays at 8-bit.
- Confirmed running on a single RTX 5090 at ~21 tok/s with a 262K context.
- MIT licensed, OpenAI- and Anthropic-compatible API, works on NVIDIA and AMD, Linux and Windows.
A new 339.5 GB GGUF quantization of DeepSeek-V4.1-Flash-1017 can run on a single gaming GPU through Project Maya. The runtime treats a fast NVMe SSD and system RAM as extensions of GPU memory, making the model accessible on workstation hardware at the cost of storage, I/O sensitivity, and long prompt-processing times.
The model files package a 2-to-3-bit Maya-S quantization built for Maya’s expert-streaming engine. The author reports 92.6% top-token agreement with DeepSeek’s reference implementation on held-out text, with a KL divergence of 0.120.
The release in numbers
| Component | Size |
|---|---|
| Quantized model weights | 178.2 GB |
| Engram tables | 161.3 GB |
| Total | 339.5 GB |
- Model size: 748 billion parameters
- Architecture: mixture of experts, with 384 routed experts per layer
- Experts used per token: six routed experts plus one shared expert
- Reported hardware: one RTX 5090, 128 GB of system RAM, and fast NVMe storage
- Reported decode speed: about 21 tokens per second in one community test
A 748B model, served from storage
Mixture-of-experts models route each token through a small subset of their weights. DeepSeek-V4.1-Flash-1017 contains 384 routed experts per layer, yet activates only six for each token, along with a shared expert. Project Maya keeps frequently used experts in VRAM, places others in system RAM, and loads colder experts from the SSD as needed.
This cache hierarchy lowers the amount of GPU memory required to serve the model. Performance depends on how often requested experts are already cached, how quickly the SSD can supply missing weights, and how much RAM is available between storage and the GPU.
Precision where it pays
Quantization reduces storage and memory use by representing weights with fewer bits. Maya-S applies its heaviest compression to the routed experts, which account for most of the model, while retaining higher precision for components that are more sensitive to rounding error:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
