UC San Diego's BeNice Slashes AI Memory Use by 66% in Looped Models

Looped language models reuse the same transformer block across multiple passes, trading additional computation for better predictions at a fixed parameter count. Reusing weights keeps parameter memory stable as the recurrence count rises, but inference still stores separate attention keys and values for every pass. An eight-pass model can therefore require eight KV-caches for each layer in the repeated block.

The BeNice! paper from UC San Diego and Together AI presents a training recipe that lets every recurrence use keys and values produced by the final pass for earlier tokens. A server can then retain one cache per repeated layer, removing the recurrence multiplier from KV-cache memory while maintaining the model’s reported quality.

One block, many caches

During autoregressive generation, self-attention caches the keys and values of completed tokens so the model does not recompute them for every new token. Standard looped transformers maintain this history separately for each recurrence because every pass produces different hidden states and, consequently, different keys and values.

Earlier research found that some looped models could reuse the final recurrence’s cache across earlier passes with little quality loss, a property the BeNice authors call KV-shareability. The behavior remained unreliable across models trained with similar procedures. Some checkpoints tolerated sharing; others suffered sharp quality degradation.