Skip to main content
Back

AngelSlim Shrinks Tencent's 1.5TB Hy4-Preview to 214 GiB at 6x Compression

Tencent's AngelSlim team shrinks the 770B Hy4-preview MoE from 1.5TB down to 213GB using a custom 1.31-bit quantization strategy with minimal accuracy loss.

  • AngelSlim released three GGUF builds of Tencent's Hy4-preview 770B MoE, compressing 1.5TB down to 214 GiB.
  • New MIX-STQ1_0 recipe uses 1.31-bit ternary weights with 3:4 sparsity on selected expert layers.
  • Custom encoder swaps amax scaling for weighted least-squares, cutting weighted quantization error by 89.7%.
  • Accuracy vs BF16 drops only marginally: SWE-Bench multi 82.9 to 81.3, MRCR 81.3 to 81.1.
  • Requires patched llama.cpp since the hyv4 architecture is not upstream yet.
  • Benchmarks on 8x H20 show 204 tokens/sec prefill and 20 tokens/sec decode for STQ1_0.

Tencent's Hy4-preview is a 770B parameter Mixture-of-Experts model. In its native form it weighs 1.5TB and requires a rack of B200s to run. The AngelSlim team just released three GGUF builds that bring it within reach of a single well-specced node, including one that compresses the whole thing to roughly 213 GiB while keeping benchmark scores within a couple of points of the BF16 original.

That compression, from 1.5TB to about 214 GiB, relies on a new mixed-precision recipe called MIX-STQ1_0. Some layers drop to 1.31 bits per weight while higher-sensitivity layers keep more bits, with the allocation driven by an importance matrix rather than uniform rounding.

Three builds, three tradeoffs

The repo ships three quantizations of the same Hy4-preview checkpoint:

FileSizeBits/weightUse case
Q4_K_M435 GiB4.86Safe default when VRAM is available
UD-IQ1_M220 GiB2.44Unsloth's dynamic 1-bit strategy
STQ1_0214 GiB2.38MIX-STQ1_0 recipe, smallest

None of these run on stock llama.cpp because the hyv4 architecture is not yet upstream. You need to apply patches from the hy4-preview-patch directory, after which standard commands like llama-cli and llama-bench work as usual. On an 8x H20 rig, the STQ1_0 build hits roughly 204 tokens/sec prefill and 20 tokens/sec decode.

How STQ1_0 works

STQ1_0 originates from llama.cpp PR #22836. Each weight takes one of three values (negative d, zero, or positive d), and exactly one lane out of every four is forced to zero, producing 3:4 sparsity. Each group of 4 weights is stored as a 4-bit code plus a 1-bit table-select that indexes into a 32-entry codebook, with a single fp16 scale shared across 256 weights. That works out to 42 bytes per 256 weights, or 1.3125 bits per weight.