Skip to main content
Back

Tufa Labs' tufazip Beats Hybrid AI Compressors Using a Single Transformer

Tufa Labs' tufazip squeezes a billion bytes of Wikipedia into roughly 105 MB using a Transformer as its only predictor, taking #2 on a 20-year-old benchmark.

Repo, Full Access
  • Tufa Labs' tufazip compresses enwik9 to 105.39 MB, taking #2 on the Large Text Compression Benchmark.
  • It is the best entry using a neural network as the sole predictor; all entries above it mix Transformers with hand-built statistical models.
  • Uses a 12-layer 163M-parameter Transformer initialized from a 4.33 MB offline-trained prior with two shared blocks.
  • Encoder and decoder run identical deterministic online training, so online weights never need to be shipped.
  • Replay helps substantially, smaller vocabularies win, and bigger models stop helping past ~77M parameters on enwik8.
  • Decoder requires two NVIDIA B200 GPUs and about a week of wall time to decompress; full writeup here.

A researcher at Tufa Labs has compressed enwik9, the billion-byte Wikipedia snapshot, to 105,388,353 bytes with tufazip v2. The system uses a single Transformer as its only predictor. At the time of the report, every benchmark entry with a smaller archive combined a Transformer with hand-crafted statistical models such as PPM or context mixing.

The final predictor has 12 layers and 163 million parameters. It starts from a 34-million-parameter offline model quantized to 4.33 MB, with two pretrained blocks copied into 12 independent layers. Those layers continue learning during compression, and the decoder repeats the same updates during decompression. The archive therefore carries the compact initial model without storing the much larger set of learned weights.

A leaderboard with a model-size tax

Matt Mahoney’s compression benchmark has measured enwik9 results for more than two decades. A submission’s score includes the compressed text, zipped decoder, model weights, dictionaries, and any other data required to reconstruct the source. A large pretrained model may reduce the encoded payload and still increase the final score.

Information theory provides the link between prediction and compression. When a model assigns probability p to the next token, that token’s ideal encoding cost is -log2(p) bits. An arithmetic coder can convert a sequence of predicted probabilities into a payload whose size approaches the model’s cumulative cross-entropy. More accurate predictions generally produce fewer bits.

The decoder learns in lockstep

tufazip sends the Transformer’s next-token probabilities to an arithmetic coder and runs an identical training loop at both ends. Its architecture uses rotary position embeddings to represent token order, SwiGLU feed-forward blocks, RMSNorm around attention and feed-forward sublayers, and query-key normalization.

The encoder and decoder process every token through the same three stages: