QCFuse Slashes RAG Response Time by 1.7x Without Sacrificing Answer Quality
QCFuse cuts RAG prefill time by 1.7x over full prefill and 1.5x over the best baseline, matching full-quality answers without stalling the serving pipeline.
PRO- QCFuse cuts RAG prefill time by 1.7x over full prefill and 1.5x over ProphetKV while matching full answer quality.
- The core problem: RAG cache fusion systems must pick which tokens to recompute, and existing methods either ignore the user query or stall the serving pipeline.
- QCFuse uses chunk-anchor query probing to get query-aware token scores cheaply from compact offline-precomputed chunk summaries.
- Critical-layer profiling identifies a single transformer layer whose attention pattern predicts full-context rankings, avoiding all-layer inspection.
- Implemented in SGLang with a Triton kernel; evaluated on Llama3.1-8B, Qwen3-8B, Qwen3-14B, Mistral-7B across six datasets.
- Best suited for high-concurrency RAG over stable knowledge bases using open-weight models; closed-API providers are not supported.
Every RAG system hides a dirty secret: the moment you retrieve context and stuff it into an LLM prompt, you've handed the model a massive, expensive prefill job. The model has to process every retrieved token before it can generate a single word of response. In RAG systems, the prefill stage is computationally expensive due to long input contexts, which are essential because they contain retrieved context that often provides crucial background knowledge for generating accurate answers. Real-world RAG pipelines typically process 4K to 16K tokens per query, with the majority coming from retrieved text chunks. The result is a sluggish time-to-first-token (TTFT) that makes interactive RAG feel broken.
A new paper introduces QCFuse, a system that attacks this problem at its root. It achieves full-prefill-level answer quality while cutting prefill time by an average of 1.7x, and does so without introducing the pipeline bottlenecks that have plagued previous approaches.
The Cache Fusion Landscape
The core insight behind a family of recent systems is that RAG workloads are highly repetitive. Context chunks retrieved for different queries can overlap by over 70%, yet strict prefix-matching policies prevent traditional prefix caching from reusing them, forcing LLMs to fully prefill redundant contexts. Cache fusion systems like CacheBlend exploit this by precomputing and storing KV caches (the key-value attention matrices) for each document chunk offline, then reusing them at query time. CacheBlend reuses pre-computed KV caches regardless of prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache.
The challenge is deciding which tokens to recompute. Reusing a KV cache naively is fast but inaccurate: the cached keys and values were computed without seeing the surrounding context or the user query, so cross-attention is broken. You need to recompute a small fraction of tokens to fix this, but picking the wrong tokens wastes your budget.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
