Nvidia Shows How to Cut AI Context Without Sacrificing Accuracy

Nvidia research shows compacting AI agent context to 16,000 tokens boosts throughput up to 7x on Qwen 3 Coder Next while preserving SWE-bench accuracy

·
·
·
Nvidia Shows How to Cut AI Context Without Sacrificing Accuracy
AuthorBy Ben Dickson
Read2 min
TopicAgents · Infra
  • Nvidia researchers found that reducing the configured context window from 256,000 to 16,000 tokens on NVIDIA H100 GPUs increased output-token throughput by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next.
  • Across 13 benchmarks (including SWE-bench Verified, Terminal-Bench, LongBench v2, and LOONG), threshold-based compaction (triggered at thresholds like 16K tokens) kept accuracy close to full-history baselines on long-horizon tasks.
  • On SWE-bench Verified, Claude Opus 4.5 improved from 70% to 74.8% accuracy with its best compaction setting, while GPT-5.6 Sol rose from 84.4% to 88.8%.
  • With single-action compaction on SWE-bench Verified, both models nearly matched full-history performance, and across two evaluated benchmarks the median peak context was just 7,000-9,000 tokens.
  • Model sensitivity varied sharply: Qwen 3 Coder Next's accuracy on SWE-bench Verified collapsed from 53% to 3.6% under a 16K-token limit, and GPT 5.6 Sol's API cost rose from $167.68 without compaction to $203.12 at the 16K limit due to prompt-cache invalidation.

AI agents are getting better at tackling complex, multi-step tasks. But there's a hidden cost: every tool call, file read, and intermediate decision adds to their growing context, making each subsequent step more expensive.

New research from Nvidia challenges the assumption that agents need all that history. And the findings have important implications for how we build, optimize, and serve AI applications.

Today we break down what developers should know.

Why AI agents don't always need long contexts

In agentic tasks, the context of the underlying LLM might grow into hundreds of thousands of tokens. The agent spends many steps searching repositories, modifying files, and running tests.

Each of these interactions becomes part of the next request to the model. The agent’s context continues to grow, even when much of that history is no longer useful.

In a new study, researchers at Nvidia tested whether AI models really need those long contexts. Across different long-context tasks, they found that LLMs can often work with much shorter contexts while maintaining their accuracy.

For developers, the findings suggest that instead of stuffing everything into the model’s context, only keep the minimal information the agent needs for the next step.

Why long contexts are expensive

When an LLM generates a token, it must attend to all the tokens that precede it. Even when storing attention values in the KV cache, the attention tax increases as the context grows because attending to previous tokens takes more computation and memory.

The researchers measured the impact on three open-weight models running on NVIDIA H100 GPUs. Reducing the configured context window from 256,000 to 16,000 tokens increased output-token throughput by 3.6 times for GLM 5.2, 5.6 times for Kimi K2.6 and seven times for Qwen 3 Coder Next.

Keep reading

Don't miss what's next in AI

Join 300,000+ engineers and researchers who get the signal, not the noise. Create a free account to read the rest of this story.

  • Full access to in-depth AI research breakdowns
  • Be the first to know what's trending before it hits mainstream
  • Daily curated papers, repos, and industry moves
Trending
  • No trending articles

Comments

avatar

Next Reads