AngelSlim Shrinks Tencent's 1.5TB Hy4-Preview to 214 GiB at 6x Compression
Tencent's AngelSlim team shrinks the 770B Hy4-preview MoE from 1.5TB down to 213GB using a custom 1.31-bit quantization strategy with minimal accuracy loss.
Takeaways
- AngelSlim released three GGUF builds of Tencent's Hy4-preview 770B MoE, compressing 1.5TB down to 214 GiB.
- New MIX-STQ1_0 recipe uses 1.31-bit ternary weights with 3:4 sparsity on selected expert layers.
- Custom encoder swaps amax scaling for weighted least-squares, cutting weighted quantization error by 89.7%.
- Accuracy vs BF16 drops only marginally: SWE-Bench multi 82.9 to 81.3, MRCR 81.3 to 81.1.
- Requires patched llama.cpp since the hyv4 architecture is not upstream yet.
- Benchmarks on 8x H20 show 204 tokens/sec prefill and 20 tokens/sec decode for STQ1_0.
Tencent's Hy4-preview is a 770B parameter Mixture-of-Experts model. In its native form it weighs 1.5TB and requires a rack of B200s to run. The AngelSlim team just released three GGUF builds that bring it within reach of a single well-specced node, including one that compresses the whole thing to roughly 213 GiB while keeping benchmark scores within a couple of points of the BF16 original.
That compression, from 1.5TB to about 214 GiB, relies on a new mixed-precision recipe called MIX-STQ1_0. Some layers drop to 1.31 bits per weight while higher-sensitivity layers keep more bits, with the allocation driven by an importance matrix rather than uniform rounding.
Three builds, three tradeoffs
The repo ships three quantizations of the same Hy4-preview checkpoint:
| File | Size | Bits/weight | Use case |
|---|---|---|---|
| Q4_K_M | 435 GiB | 4.86 | Safe default when VRAM is available |
| UD-IQ1_M | 220 GiB | 2.44 | Unsloth's dynamic 1-bit strategy |
| STQ1_0 | 214 GiB | 2.38 | MIX-STQ1_0 recipe, smallest |
None of these run on stock llama.cpp because the hyv4 architecture is not yet upstream. You need to apply patches from the hy4-preview-patch directory, after which standard commands like llama-cli and llama-bench work as usual. On an 8x H20 rig, the STQ1_0 build hits roughly 204 tokens/sec prefill and 20 tokens/sec decode.
How STQ1_0 works
STQ1_0 originates from llama.cpp PR #22836. Each weight takes one of three values (negative d, zero, or positive d), and exactly one lane out of every four is forced to zero, producing 3:4 sparsity. Each group of 4 weights is stored as a 4-bit code plus a 1-bit table-select that indexes into a 32-entry codebook, with a single fp16 scale shared across 256 weights. That works out to 42 bytes per 256 weights, or 1.3125 bits per weight.