Nvidia Shows How to Cut AI Context Without Sacrificing Accuracy
Nvidia research shows compacting AI agent context to 16,000 tokens boosts throughput up to 7x on Qwen 3 Coder Next while preserving SWE-bench accuracy
- Nvidia researchers found that reducing the configured context window from
256,000to16,000tokens on NVIDIA H100 GPUs increased output-token throughput by3.6xfor GLM 5.2,5.6xfor Kimi K2.6, and7xfor Qwen 3 Coder Next. - Across
13benchmarks (including SWE-bench Verified, Terminal-Bench, LongBench v2, and LOONG), threshold-based compaction (triggered at thresholds like16Ktokens) kept accuracy close to full-history baselines on long-horizon tasks. - On SWE-bench Verified, Claude Opus 4.5 improved from
70%to74.8%accuracy with its best compaction setting, while GPT-5.6 Sol rose from84.4%to88.8%. - With single-action compaction on SWE-bench Verified, both models nearly matched full-history performance, and across two evaluated benchmarks the median peak context was just
7,000-9,000tokens. - Model sensitivity varied sharply: Qwen 3 Coder Next's accuracy on SWE-bench Verified collapsed from
53%to3.6%under a16K-token limit, and GPT 5.6 Sol's API cost rose from$167.68without compaction to$203.12at the16Klimit due to prompt-cache invalidation.
AI agents are getting better at tackling complex, multi-step tasks. But there's a hidden cost: every tool call, file read, and intermediate decision adds to their growing context, making each subsequent step more expensive.
New research from Nvidia challenges the assumption that agents need all that history. And the findings have important implications for how we build, optimize, and serve AI applications.
Today we break down what developers should know.
Why AI agents don't always need long contexts
In agentic tasks, the context of the underlying LLM might grow into hundreds of thousands of tokens. The agent spends many steps searching repositories, modifying files, and running tests.
Each of these interactions becomes part of the next request to the model. The agent’s context continues to grow, even when much of that history is no longer useful.
In a new study, researchers at Nvidia tested whether AI models really need those long contexts. Across different long-context tasks, they found that LLMs can often work with much shorter contexts while maintaining their accuracy.
For developers, the findings suggest that instead of stuffing everything into the model’s context, only keep the minimal information the agent needs for the next step.
Why long contexts are expensive
When an LLM generates a token, it must attend to all the tokens that precede it. Even when storing attention values in the KV cache, the attention tax increases as the context grows because attending to previous tokens takes more computation and memory.
The researchers measured the impact on three open-weight models running on NVIDIA H100 GPUs. Reducing the configured context window from 256,000 to 16,000 tokens increased output-token throughput by 3.6 times for GLM 5.2, 5.6 times for Kimi K2.6 and seven times for Qwen 3 Coder Next.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves