WhiteMatter Lets Any Transformer Layer Attend Across All Depths While Shrinking Memory
A brain-inspired routing scheme lets every transformer layer attend to every past layer, cutting perplexity 8.2% and shrinking the KV cache.
Takeaways
- WhiteMatter lets every attention layer attend to every past layer via routed KV channels.
- Beats same-depth vanilla by 8.2% perplexity and outperforms a 50% deeper baseline.
- Half-cache variant retains 6.3% gain; single-channel version gives 16x compression and still beats vanilla.
- Cyclic Gauss-Seidel iteration converges 11.2x faster than Jacobi during prefill.
- Training and prefill cost 2.5x and 3.3x vanilla FLOPs; decoding cost is unchanged.
- Full paper from USC, tested on Qwen3 architecture with FineWeb-Edu.
Transformers waste information. When a model generates a token, it produces a full stack of hidden states, one per layer, each capturing something different about that position. When the next token arrives, each layer can only look back at keys and values produced at its own depth. Everything else the model computed for prior tokens sits in memory, unreachable. A new architecture called WhiteMatter tears down that wall, letting any layer attend to any past layer through a small pool of shared KV channels, and it does so while shrinking the KV cache rather than growing it.
At 16 layers with a full KV cache, WhiteMatter reaches 19.968 held-out perplexity, an 8.2% reduction over a vanilla 16-layer baseline at 21.747, and even edges out a 24-layer vanilla model at 20.181. Halving the cache keeps most of that gain.
The dead end in every attention layer
In an autoregressive transformer, each layer at the current token can only attend to KV produced from the hidden state at the same depth, so the model cannot use all the information it has already generated. Prior work tried to relax this. Feedback Transformer mixed all past layers into one shared pool, but every consumer layer got the exact same mixture. LCKV pushed only the top layer's state down to every consumer. FusedKV added static per-layer fusions but only in the feedforward direction, so shallow layers still could not reach deeper past representations.
None of these allowed different consumer layers to pull from different source depths in a way that also adapts to the token's content