Qwen3.8 Squeezed to 11.8 GB Runs at 150 Tokens per Second
A community remix packs an uncensored 27B Qwen into 11.8 GB by giving each of 851 tensors its own quantization type, hitting 100+ tokens per second on a 16 GB card.
- Non-uniform GGUF quant of Qwen3.8-27B-Heretic assigns a different precision to each of 851 tensors.
- IQ3_S build: 11.80 GB, only 0.95% worse perplexity than F16, fits on a 16 GB GPU.
- Uniform IQ3_S at 12.42 GB shows 2.9% regression, so non-uniform wins on both size and quality.
- Reproduces IST-DASLab's RCO allocation locally via llama.cpp and the official imatrix.
- Optional MTP speculative-decoding twin hits 150 tokens/sec on RTX Pro 6000; 100+ t/s on a 4080.
- Apache 2.0, works in llama.cpp, Ollama, LM Studio and vLLM out of the box.
A 27B Qwen quantization targets 16 GB GPUs
A new Hugging Face release packages a refusal-modified Qwen3.8-27B checkpoint as a set of non-uniform GGUF quantizations. The 11.80 GB IQ3_S variant is small enough to leave some memory available on a 16 GB GPU while keeping WikiText-2 perplexity within 0.95% of the F16 reference, according to the author’s benchmarks.
The model repository uses a per-tensor allocation map derived from IST-DASLab’s GSQ and RCO work. Every weight tensor can receive a different quantization type based on its sensitivity to compression and the target file-size budget. The resulting files use the standard GGUF container and load through compatible llama.cpp-based runtimes.
Four memory and quality tradeoffs
| Variant | File size | Perplexity change | Reported speed |
|---|---|---|---|
| IQ2_XS | 8.45 GB | +10.7% | 105 tokens/s |
| IQ2_S | 9.28 GB | +6.3% | 101 tokens/s |
| IQ3_XXS | 10.12 GB | +3.7% | 97 tokens/s |
| IQ3_S | 11.80 GB | +0.95% | 90 tokens/s |
Perplexity measures how well a model predicts the next token, with lower scores indicating better predictions. The IQ3_S build records 6.1778 on WikiText-2, compared with 6.1197 for F16. Its average precision is 3.50 bits per weight.
A uniform IQ3_S quantization of the same checkpoint occupies 12.42 GB and increases perplexity by 2.9%. The per-tensor allocation therefore produces a file that is 0.62 GB smaller while preserving more of the F16 result. Perplexity remains a narrow proxy and does not measure coding accuracy, instruction following, long-context behavior, or output safety.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
