Qwen3.5-9B Squeezed to 4 GB While Beating Standard Compression by 10%
A community quantization recipe shrinks Qwen3.5-9B to roughly 4 GB at under 4 bits per weight while beating standard low-bit GGUFs on benchmarks.
Takeaways
- MoQ GGUFs compress Qwen3.5-9B to about 4 GB at under 4 bits per weight.
- Independent eval shows roughly 10% benchmark gain and 2x token efficiency over Unsloth Dynamic quants.
- Reported FP16 performance recovery sits between 86% and 97% on low-bit MoQ builds.
- Reverse engineering shows MoQ uses a uniform IQ3_S/IQ4_XS allocation, not a heavy per-tensor mixture.
- The surprise: low-bit SSM modules work here without runaway generation, per Kaitchup analysis.
- Quantization method is not yet public; author plans to share code, applicability beyond Qwen3.5 unknown.
A new community quantization recipe called Mixture of Quants (MoQ) is generating buzz on Hugging Face for an unusual reason: it claims to shrink Qwen3.5-9B down to roughly 4 GB at under 4 bits per weight while holding onto most of the original model's accuracy. Independent benchmarks by Benjamin Marie back up at least part of the claim, and a reverse-engineering pass on the GGUF files reveals a quantization recipe that does not match what the model card advertises.
The author, w-ahmad, has published roughly twenty GGUF variants ranging from full BF16 at about 16.7 GB all the way down to a 2.10 bpw build at 2.23 GB. The sweet spot most people are talking about sits between MoQ-3.65 (3.79 GB) and MoQ-4.85 (5.05 GB), with the 4.00 bpw variant landing at exactly 4.20 GB. The weights ship under an MIT license and run with stock llama.cpp using a one-line command.
What the benchmarks actually show
Benjamin Marie, who runs the Kaitchup newsletter, evaluated the low-bit MoQ files against Unsloth's Dynamic (UD) quants in the same bit range. He focused on the low-bit versions, below 4 bits on average, since that's where GGUF models are usually much weaker, and found that MoQ models are much better than UD quants on benchmarks, and also more token-efficient. Independent summaries put the gap at roughly 10% on benchmark scores and about 2x better token efficiency at matched bits-per-weight.