Prefix Sliding Cuts Reasoning AI Memory Costs With a 3x Speedup

Reasoning models like o1-style systems get smarter by thinking longer, but that thinking is not free. Every extra token in a chain of thought sits in the key-value cache, and full attention grows quadratically with sequence length. On the hardest problems, where a model may want to reason for tens of thousands of tokens, GPUs run out of memory long before the answer arrives. A new paper called Prefix Sliding for efficient test-time scaling proposes a blunt fix: throw most of the reasoning trace away as the model generates it.

The author list is unusually stacked. Niklas Muennighoff leads the work, joined by Mike Lewis, Jason Wei, Percy Liang, Luke Zettlemoyer, Andrew Ng, Yejin Choi and others. Muennighoff was also first author on s1, the widely cited paper showing that budget forcing can control test-time compute by cutting a model's thinking short or extending it with an appended "Wait" that prompts the model to double-check its answer. Prefix Sliding is the natural sequel: if we are going to let models think for a hundred thousand tokens, we need a way to make that affordable.

Why long reasoning breaks the cache

Test-time scaling has become one of the dominant ways to squeeze more capability out of frozen weights, using extra compute at inference to improve performance. The approach was popularized by OpenAI's o1 model, which triggered many replication efforts. The catch is that a transformer attending over its full reasoning trace pays for every token it has already produced. Double the thinking, quadruple the attention cost. On tasks that need genuine deliberation, such as competition math or hard science questions, that quickly becomes prohibitive.