TurboQuant Doubles LLM Context Length by Cutting KV Cache Memory in Half
A community vLLM integration of Google's ICLR 2026 paper cuts KV cache memory in half, doubling effective context on consumer GPUs with no retraining required.
Takeaways
- 2x context on same hardware: The TurboQuant vLLM integration freed 30 GB of KV cache across 4x RTX 3090s, doubling token capacity to 914K.
- Based on ICLR 2026 Google Research paper: arXiv:2504.19874 proves near-optimal compression within ~2.7x of the Shannon information-theoretic limit.
- No retraining or calibration needed: The algorithm is data-oblivious — a fixed random rotation + precomputed codebook works on any transformer architecture.
- Zero accuracy loss at 3.5 bits: LongBench scores match FP16 exactly; GPQA Diamond and HumanEval show no significant degradation.
- Key limitation — hybrid models only partially benefit: Only full-attention layers are compressed; Mamba/linear-attention layers are untouched, limiting savings on models like Qwen3.5.
- vLLM PR not yet merged: Upstream support (#38479) is unstable with known memory leaks; the community repo works around this via monkey-patching.
The KV cache is the silent tax on every LLM inference run. Every token you generate forces the model to store a key vector and a value vector for that token across every attention layer, and that memory bill scales linearly with context length. At 128K tokens, a 70B model's KV cache alone consumes around 40 GB , nearly enough to crowd out the model weights themselves on high-end hardware. Now a community implementation of Google's TurboQuant paper is making it practical to slash that cost in half, running directly inside vLLM on consumer-grade GPUs.
What just shipped
TurboQuant by 0xSero is an open-source Python library that integrates the TurboQuant KV cache compression algorithm directly into vLLM via monkey-patching. It implements KV cache compression from the ICLR 2026 paper (arXiv:2504.19874) with vLLM integration. The headline result: on Qwen3.5-27B running across 4x RTX 3090s, the implementation freed 30 GB of KV cache memory across the four GPUs and doubled the effective token capacity from ~457K to ~914K tokens.
TurboQuant is a KV cache compression method, not a model weight quantization technique. This distinction matters. The model weights still require the same VRAM at FP16 regardless of TurboQuant. What changes is the memory footprint of the attention key-value cache generated during inference. You can combine it with weight quantization methods like AWQ or GPTQ for a compounding effect.
The algorithm behind the compression
TurboQuant is a two-stage vector quantization algorithm from Google Research (ICLR 2026) that compresses high-dimensional vectors , particularly LLM key-value caches , to as low as 3 bits per coordinate with zero accuracy loss. The core insight is elegant: raw KV vectors are hard to quantize efficiently because their coordinates are unevenly distributed. The fix is a rotation.
TurboQuant is a training-free way to turn each key/value vector from a chunk of FP16 numbers into a much smaller code, while trying to preserve the geometry that attention cares about. Its core trick is to first apply a fixed random rotation that makes the vector's coordinates look much more uniform and predictable, then quantize those rotated coordinates with a precomputed scalar codebook. After rotation, each coordinate independently follows a Beta distribution that converges to Gaussian N(0, 1/d) as dimensionality increases , which is exactly the shape that an optimal Lloyd-Max quantizer is designed to handle.