IST Austria Shrinks a 177B Qwen3.8-Flash-Next Model to 66 GB
IST Austria's DASLab shrinks Qwen3.8-Flash-Next, a 177B mixture-of-experts model, down to 66 GB with almost no quality loss using non-uniform GGUF quantization.
Takeaways
- IST Austria DASLab released non-uniform GGUF quantizations of the 177B-parameter Qwen3.8-Flash-Next MoE model.
- The IQ3_XXS build hits 99.4% of BF16 task average at 4.7x smaller size (75.8 GB vs 354 GB).
- IQ3_S matches or exceeds base on AIME25, GPQA-Diamond, and LiveCodeBench v6 at 83.6 GB.
- Two methods power the release: GSQ for scalar quantization and RCO for budget-constrained per-tensor type allocation.
- Q2_0 build gives 3.4x higher prompt throughput than IQ2_XS by avoiding lookup-table formats.
- Sparse n-gram lookup shard (28.8 GB) can stay memory-mapped on disk, cutting resident VRAM needs significantly.
IST Austria’s DAS Lab has released four quantized GGUF builds of Qwen3.8-Flash-Next, a 177-billion-parameter mixture-of-experts model. The smallest builds occupy roughly 66 to 68 GB, compared with 354 GB for the BF16 weights. The 75.8 GB IQ3_XXS build retains 99.4% of the base model’s reported task average.
Flash-Next uses 512 routed experts in each of its 48 layers, with 10 experts active for each token. Sparse routing reduces per-token computation, although every expert still contributes weights that must be stored and loaded. Expert matrices account for 95% of the weights considered by the quantization search, creating substantial room for mixed-precision compression.
GSQ Chooses Values; RCO Places the Bits
GSQ, or Gumbel-Softmax Quantization, compresses trained weights into scalar values without retraining the full model. It jointly learns which quantization bucket each weight should use and the scale applied to each group. A Gumbel-Softmax relaxation makes those discrete choices differentiable, allowing gradient descent to optimize them. The lab designed the method to approach vector-quantization quality at average widths of 2 to 3 bits while remaining compatible with scalar GGUF formats.
RCO, or Riemannian Constrained Optimization, decides how much precision each tensor receives. It selects among several quantization types for every tensor while enforcing a fixed total file-size budget. The method represents that budget as a smooth geometric constraint, allowing the optimizer to minimize task loss without exceeding the target size.
Together, GSQ and RCO produce a mixed-precision GGUF in which sensitive tensors receive more bits and tolerant tensors receive fewer. GGUF is the model container used by llama.cpp and compatible applications such as Ollama and LM Studio, so the resulting files require no new quantization format at inference time.