ISTA-DASLab Squeezes DeepSeek's 763B Model onto a 12 GB GPU
ISTA-DASLab shrinks DeepSeek-V4.1-Flash-1017 to under 2 bits per weight, letting a 763B MoE run locally on a single 12GB consumer GPU.
- ISTA-DASLab released GSQ-RCO GGUF quants of DeepSeek-V4.1-Flash-1017, a 763B 384-expert MoE.
- Three sizes available: IQ1_M (1.75 bpw), IQ2_XXS (2.00) and IQ2_XS (2.25).
- Runs on a single 12GB GPU when paired with the Strata inference engine.
- GSQ handles per-tensor scalar quantization; RCO assigns formats per tensor under a size budget.
- Same lab previously built GPTQ, now standard across the quantization ecosystem.
- Best for private local inference on workstations with large system RAM.
ISTA-DASLab, the research group behind GPTQ, has released three GGUF quantizations of DeepSeek-V4.1-Flash-1017. The smallest averages 1.75 bits per weight, reducing storage enough for hybrid inference with a 12 GB GPU and a much larger pool of system memory.
DeepSeek-V4.1-Flash-1017 is a 197-billion-parameter model. Sparse routing activates only a subset of those experts for each token, reducing computation. Every expert’s weights still require storage, so the deployment depends on system RAM, disk capacity, memory bandwidth, and the Strata inference engine.
| Release detail | Value |
|---|---|
| Architecture | 763B-parameter mixture of experts |
| Expert count | 384 |
| Quantizations | IQ1_M, IQ2_XXS, and IQ2_XS |
| Container | GGUF with per-tensor quantization types |
| GPU claim | 12 GB of VRAM with Strata and sufficient system RAM |
One optimizer shapes weights, another spends bits
The compression pipeline combines Gumbel-Softmax Quantization, or GSQ, with Riemannian Constrained Optimization, or RCO. GSQ quantizes each tensor at a selected precision, while RCO decides how to distribute the available bit budget across the model.
GSQ treats the quantization bucket assigned to each weight as an optimization variable. A Gumbel-Softmax relaxation makes that discrete choice differentiable during calibration, allowing the method to optimize grid assignments and group scales instead of applying ordinary nearest-value rounding. The resulting weights remain compatible with scalar formats supported by GGUF runtimes.
RCO searches across candidate formats for every tensor and selects a combination that meets the target size while limiting estimated quality loss. That allocation matters for mixture-of-experts models because attention tensors, shared layers, embeddings, and routed experts can respond differently to aggressive compression. A uniform two-bit conversion spends the same budget everywhere, regardless of sensitivity.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
