KV cache techniques for faster, cheaper multi-model AI agents
Nvidia's Qwen3 14B to 32B mapper is 25.1x vs re-prefill at 32,768 tokens. Cache-to-Cache reports about 2.5x against text
- Nvidia's mapper moves a 32,768-token Qwen3 14B cache to Qwen3 32B in 278 ms, against 6,975 ms of re-prefill, 25.1x at that length.
- That Qwen3 pair keeps 97.6% of the 32B model's accuracy, and two Ministral 3 pairs fall to 42% to 44%.
- Cache-to-Cache reports about 3.1 to 5.4 points over text for a Qwen3-0.6B receiver, and the paper's stated average speedup is 2.5x (the three published pairs are 1.51x, 3.46x, and 14.41x).
- Decoding one 80-token MMLU-Redux reply from Qwen2.5-0.5B took 1,312 ms, and cache fusion took 90 ms.
- On a preliminary GSM8K flow, text plus cache fusion scored 78.01, against 62.55 for cache fusion alone and 61.18 for a text handoff.
Multi-model agents promise the best of both worlds: strong models for difficult tasks, smaller models for routine work, and specialists for everything in between.
The catch? Every time models hand off work, they can end up repeating expensive computations or translating their knowledge into text. As agents take on longer tasks, these costs start adding up.
New research and frameworks suggest there are ways around this bottleneck, with some impressive performance gains.
Today we look at what this means for building faster, more efficient multi-model AI agents.
How sharing KV caches could make multi-model AI agents faster
Consider a coding agent that has spent an hour exploring a repository, running tests and tracing a difficult bug. Now it must either switch to a stronger model to solve the problem or delegate part of the investigation to a specialist model working in parallel.
Both options introduce severe compute overhead. Researchers are exploring transfers through key-value (KV) caches, representations models build when reading input.
Nvidia has developed a way to reuse a cache across models, speeding up the handoff by up to 25 times. A separate technique, Cache-to-Cache (C2C), lets models share information without generating text tokens, reporting an average 2.5x speedup.
For developers building longer-running agents, these techniques address the growing cost of moving work between models.
Why AI model handoffs become expensive
There are two common reasons to hand off work in a multi-model system:
- Model switching: An existing session is transferred to a more capable model for difficult reasoning, or to a smaller model to cut costs on easier tasks.
- Task delegation: A subtask is given to a specialist model, such as reviewing a component or writing tests.
The handoff tax is different between different modes. In model switching, the new model needs to reprocess the accumulated source code, test results, conversation and earlier decisions. In delegation, the main tax becomes expensive when specialists exchange long reports or overlapping contexts.
The costs arise from how language models process input. During prefill, a model reads the prompt and computes the keys and values used by its attention mechanism. It stores these representations in a KV cache. During
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves
Comments
