Skip to main content
Back

TU Munich's KV Packet Cuts RAG Prefill Time by 19x on Llama

A new caching trick lets language models reuse precomputed document caches with almost zero compute overhead, cutting first-token latency by up to 19x.

  • KV Packet wraps cached documents with learned header and trailer tokens, letting precomputed caches be concatenated without recomputation.
  • Achieves 5 to 6 orders of magnitude fewer FLOPs than full recompute while matching F1 on retrieval and reasoning tasks.
  • Time-to-first-token drops up to 19.45x on Needle-in-a-Haystack versus recomputation-based baselines like CacheBlend and EPIC.
  • Trained via self-supervised KL distillation using the model as its own teacher, base weights stay frozen, no labeled data needed.
  • Only 8 header plus 8 trailer tokens per packet, works with Llama-3.1-8B and Qwen-3-4B, compatible with KV compression.
  • Code at github.com/ChuangtaoChen-TUM/KVPacket, paper at arxiv.org/abs/2604.13226.

Retrieval-augmented generation has an ugly little secret: every time your system pulls the same document into a new prompt, the model recomputes its key-value cache from scratch. A team from TU Munich, TU Darmstadt, Zhejiang University, and TU Ilmenau just proposed a way to skip that step almost entirely, wrapping cached documents in small trainable "adapter" tokens that make them safe to stitch together at serving time.

Their method, called KV Packet, treats each document's KV cache as an immutable packet with a learned header and trailer. The adapters are trained via self-supervised distillation to bridge context discontinuities, and the whole thing plugs into Llama-3.1 and Qwen-3 without touching the base model weights.

The context-dependency tax

The KV cache in a transformer is fast because it saves the key and value projections for every token so you do not have to reprocess them on each generation step. The catch is that those projections were computed with a specific prefix in mind. Because the KV states for each token are conditioned on its entire preceding context, a cache precomputed for a document in isolation is no longer valid when that document is placed alongside other retrieved chunks at inference time.

Positional dependency is the easy half of this problem. Modern LLMs use Rotary Positional Embeddings, which encode position by rotating Query and Key vectors by a position-dependent angle, so you can cheaply rotate a cached key to a new absolute position. Context dependency is the hard half. Naive concatenation creates isolated attention blocks because each document's KV cache was generated independently, so its internal representations were computed using an incomplete attention map that lacks the semantic influence of the global prefix.