---
title: "TU Munich's KV Packet Cuts RAG Prefill Time by 19x on Llama"
subtitle: "A new caching trick lets language models reuse precomputed document caches with almost zero compute overhead, cutting first-token latency by up to 19x."
url: "https://alphasignal.ai/news/tu-munich-s-kv-packet-cuts-rag-prefill-time-by-19x-on-llama"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T23:10:58.305Z"
type: "Paper"
topics:
  - "Infra"
  - "Post Training"
subtopics:
  - "Inference Optimization"
  - "Model Serving"
  - "Distillation"
upvotes: 1969
source_url: "https://arxiv.org/abs/2604.13226"
comments: 0
truncated: true
---

# TU Munich's KV Packet Cuts RAG Prefill Time by 19x on Llama

> A new caching trick lets language models reuse precomputed document caches with almost zero compute overhead, cutting first-token latency by up to 19x.

Retrieval-augmented generation has an ugly little secret: every time your system pulls the same document into a new prompt, the model recomputes its key-value cache from scratch. A team from TU Munich, TU Darmstadt, Zhejiang University, and TU Ilmenau just proposed a way to skip that step almost entirely, wrapping cached documents in small trainable "adapter" tokens that make them safe to stitch together at serving time.

Their method, called [KV Packet](https://arxiv.org/abs/2604.13226), treats each document's KV cache as an immutable packet with a learned header and trailer. The adapters are trained via self-supervised distillation to bridge context discontinuities, and the whole thing plugs into Llama-3.1 and Qwen-3 without touching the base model weights.

## The context-dependency tax

The KV cache in a transformer is fast because it saves the key and value projections for every token so you do not have to reprocess them on each generation step. The catch is that those projections were computed with a specific prefix in mind. Because the KV states for each token are conditioned on its entire preceding context, a cache precomputed for a document in isolation is no longer valid when that document is placed alongside other retrieved chunks at inference time.

Positional dependency is the easy half of this problem. Modern LLMs use Rotary Positional Embeddings, which encode position by rotating Query and Key vectors by a position-dependent angle, so you can cheaply rotate a cached key to a new absolute position. Context dependency is the hard half. Naive concatenation creates isolated attention blocks because each document's KV cache was generated independently, so its internal representations were computed using an incomplete attention map that lacks the semantic influence of the global prefix.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs](https://arxiv.org/abs/2604.13226)
- [KV Packet GitHub Repository](https://github.com/ChuangtaoChen-TUM/KVPacket)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444)
- [KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse](https://arxiv.org/abs/2502.16002)