Together AI's ThunderAgent Doubles GPU Throughput Running 192 Agents at Once
ThunderAgent fixes KV cache thrashing in agentic inference, delivering 2.5x higher throughput and ~10x lower latency -- accepted as an ICML 2026 Spotlight paper.
- ThunderAgent is a program-aware scheduler for agentic LLM inference, accepted as an ICML 2026 Spotlight paper (top 2.2%).
- It solves KV cache thrashing -- the cascade of cache evictions that cripples engines like vLLM and SGLang when hundreds of agents run concurrently.
- On a single 8xH100 node, it delivers 2x throughput (803 vs 390 tok/s) and 6x lower latency (10.6s vs 65s) over SGLang at batch 192.
- Multi-node scaling is near-linear: 671 to 2,248 steps/min from 16 to 64 GPUs, with a lead that widens from 1.79x to 2.39x over SGLang Gateway.
- Integration requires only adding a
program_idfield to existing OpenAI-compatible API calls -- works with vLLM, SGLang, offloading, and speculative decoding. - Already adopted by NVIDIA Dynamo and SkyRL; open-source on GitHub under MIT license.
Running hundreds of AI agents concurrently is quietly one of the hardest infrastructure problems in ML right now. Every agent alternates between heavy GPU work -- generating tokens -- and idle waiting while a tool runs. That rhythm is brutal for inference engines, and Together AI's new ThunderAgent is the first system to fix it at the scheduler level. It was accepted to ICML 2026 as a Spotlight paper, placing it in the top 2.2% of submissions.
The GPU memory war nobody talks about
To understand why this matters, you need to know about the KV cache. When a model generates tokens, it stores intermediate attention computations (keys and values) in GPU memory so it doesn't have to recompute them for every new token. This cache is precious -- and finite.
In a single-agent setup, this is fine. But run 192 agents at once and you have a problem. An agentic workflow alternates between two phases: the GPU-heavy reasoning phase where the model is generating tokens, and the GPU-idle acting phase where the agent is waiting for a tool like a compiler to return. While Agent A waits on a tool call, its KV cache sits in memory doing nothing. The engine, seeing memory pressure, evicts it using a simple least-recently-used (LRU) policy. When Agent A's tool returns and it needs to resume, the engine has to recompute the entire conversation history from scratch -- which in turn evicts Agent C's cache. Cascade.
This is KV cache thrashing. At high concurrency, this cascade of evictions and recomputations can lead to severe throughput and latency degradation.
Why the obvious fixes don't work
The intuitive solutions all fall short:
- More GPU nodes: Existing multi-node routers such as SGLang Gateway pin each agent to a fixed node to preserve cache locality. But because agentic context lengths grow unpredictably, some nodes are assigned agents with long contexts that exhaust memory, while others sit idle with capacity to spare. The overloaded nodes still thrash.