Skip to main content
Back

TurboQuant+ Squeezes AI Memory 5x So Giant Models Fit Anywhere

TurboQuant+ brings Google's ICLR 2026 KV cache compression to Apple Silicon and vLLM, squeezing 4.9x more context into the same memory with near-zero quality loss

Repo, Full Access
  • TurboQuant+ is an open-source implementation of Google's ICLR 2026 TurboQuant algorithm, compressing LLM KV caches up to 4.9x with near-zero quality loss.
  • Works end-to-end on Apple Silicon via a llama.cpp fork with Metal GPU kernels; use --cache-type-k turbo3 flags.
  • vLLM upstream support is already merged: --kv-cache-dtype turboquant_3bit_nc works on stock vLLM for Qwen, Llama, Mistral, and Gemma.
  • Community finding: value cache compression is essentially free — compressing V down to 2 bits has zero measurable effect when key precision is maintained.
  • Current limitation: decode speed is 3-8x slower than q8_0 baseline due to unoptimized Metal Walsh-Hadamard shaders (issue #23).
  • Repo is free under Apache 2.0; Python prototype installs with pip install -e ".[dev]" and requires no model download to demo.

TurboQuant+ is an open-source Python implementation of Google Research's TurboQuant algorithm, which landed at ICLR 2026 with a striking claim: compress the KV cache of any transformer model down to 3 bits per coordinate with zero measurable accuracy loss. The repo, built by independent developer Tom Turney, goes well beyond a paper reproduction. It ships a working C port integrated into llama.cpp, Metal GPU kernels for Apple Silicon, and a growing body of follow-on research findings that the community has been independently validating.

The KV cache problem, in plain terms

Every time a transformer generates a token, it stores key and value vectors for all previous tokens in what's called the KV cache. This is the model's working memory for the current context. As context windows push toward 128K tokens and beyond, the KV cache becomes the primary memory bottleneck. For example, Llama 3.1 70B at 128K context requires roughly 40 GB just for KV cache storage. That's before you've loaded a single model weight.

TurboQuant is a vector quantization algorithm, not a model training technique and not a weight compressor. It operates exclusively on the KV cache during inference, compressing key and value vectors as they are written to the cache and decompressing them when the attention mechanism reads them back. The big win: it is data-oblivious, requiring no calibration data, no preprocessing, no codebook training. The same quantizer works on any input distribution.

How the math actually works

The core trick is a two-stage pipeline. First, each KV vector gets multiplied by a random orthogonal matrix. After rotation, each coordinate's distribution becomes a known Beta distribution, concentrated near zero with a predictable shape that depends only on the vector dimension. Because the distribution is known analytically, you can precompute the optimal scalar quantizer (a Lloyd-Max quantizer) once and reuse it for every vector. No training data needed.