Skip to main content
Back

AIST Researchers Prove Softmax Attention Breaks a Long-Assumed Math Law

A new random matrix theory result pins down the exact spectrum of softmax attention at initialization, overturning a previously assumed Marchenko-Pastur law.

Paper, Full Access
  • First rigorous Gaussian equivalence result for softmax attention at random initialization, in the proportional-limit regime with fixed inverse temperature.
  • Bulk singular value distribution of the attention matrix matches a two-parameter linear Gaussian model.
  • Corrects prior claim: the squared-singular-value law departs from Marchenko-Pastur, with a strictly wider bulk.
  • Proof uses normalizer concentration, Taylor expansion of degree log(d)/log log(d), and polynomial linearization.
  • Suggests breakdown scale beta ~ sqrt(log d), aligning with LogN-Scaling, YaRN, and SSMax heuristics.
  • Accepted as an oral at AISTATS 2026; no code or model released.

A trio of researchers from AIST, Kyoto University, and RIKEN AIP has produced the first rigorous Gaussian equivalence result for self-attention, giving a precise mathematical description of how the singular values of a softmax attention matrix are distributed at random initialization. The paper, accepted as an oral at AISTATS 2026, also corrects a prior claim in the literature about what that distribution looks like.

When researchers reason about rank collapse, entropy collapse, or long-context behavior in transformers using spectral arguments, the baseline they should be comparing against is now known, and it differs from the one previously assumed.

Why attention resists standard tools

Random matrix theory (RMT) has already given deep learning a lot of mileage on feedforward networks. Pennington and Worah initiated nonlinear random matrix theory for deep learning by deriving the spectral distribution of random-feature Gram matrices, and later work extended this to neural kernels, Jacobian spectra, and Fisher information. A recurring principle in that line of work is Gaussian equivalence: a high-dimensional nonlinear model behaves, in the limit, like a linear Gaussian model determined by only a handful of statistics.

Attention breaks the standard machinery. Softmax over pairwise scores produces a full matrix with row-wise normalization, coupling all tokens and destroying i.i.d. structure. Because softmax is not an entrywise operation, the usual proof strategies for nonlinear random matrices do not apply. Prior spectral analyses either worked on a surrogate model or made assumptions that turn out to be too strong.

The setup, in one paragraph

The authors study a single attention layer with query and key weights drawn as independent Gaussians. The score matrix is