AngelSlim Shrinks Tencent's 1.5TB Hy4-Preview to 214 GiB at 6x Compression
Tencent's AngelSlim team shrinks the 770B Hy4-preview MoE from 1.5TB down to 213GB using a custom 1.31-bit quantization strategy with minimal accuracy loss.
PRO- AngelSlim released three GGUF builds of Tencent's Hy4-preview 770B MoE, compressing 1.5TB down to 214 GiB.
- New MIX-STQ1_0 recipe uses 1.31-bit ternary weights with 3:4 sparsity on selected expert layers.
- Custom encoder swaps amax scaling for weighted least-squares, cutting weighted quantization error by 89.7%.
- Accuracy vs BF16 drops only marginally: SWE-Bench multi 82.9 to 81.3, MRCR 81.3 to 81.1.
- Requires patched llama.cpp since the hyv4 architecture is not upstream yet.
- Benchmarks on 8x H20 show 204 tokens/sec prefill and 20 tokens/sec decode for STQ1_0.
Tencent's Hy4-preview is a 770B parameter Mixture-of-Experts model. In its native form it weighs 1.5TB and requires a rack of B200s to run. The AngelSlim team just released three GGUF builds that bring it within reach of a single well-specced node, including one that compresses the whole thing to roughly 213 GiB while keeping benchmark scores within a couple of points of the BF16 original.
That compression, from 1.5TB to about 214 GiB, relies on a new mixed-precision recipe called MIX-STQ1_0. Some layers drop to 1.31 bits per weight while higher-sensitivity layers keep more bits, with the allocation driven by an importance matrix rather than uniform rounding.
Three builds, three tradeoffs
The repo ships three quantizations of the same Hy4-preview checkpoint:
| File | Size | Bits/weight | Use case |
|---|---|---|---|
| Q4_K_M | 435 GiB | 4.86 | Safe default when VRAM is available |
| UD-IQ1_M | 220 GiB | 2.44 | Unsloth's dynamic 1-bit strategy |
| STQ1_0 | 214 GiB | 2.38 | MIX-STQ1_0 recipe, smallest |
None of these run on stock llama.cpp because the hyv4 architecture is not yet upstream. You need to apply patches from the hy4-preview-patch directory, after which standard commands like llama-cli and llama-bench work as usual. On an 8x H20 rig, the STQ1_0 build hits roughly 204 tokens/sec prefill and 20 tokens/sec decode.
How STQ1_0 works
STQ1_0 originates from llama.cpp PR #22836. Each weight takes one of three values (negative d, zero, or positive d), and exactly one lane out of every four is forced to zero, producing 3:4 sparsity. Each group of 4 weights is stored as a 4-bit code plus a 1-bit table-select that indexes into a 32-entry codebook, with a single fp16 scale shared across 256 weights. That works out to 42 bytes per 256 weights, or 1.3125 bits per weight.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
