Imperial College Researcher Shows Self-Attention Solves Quantum Pair Correlations Where Neural Nets Fail

A new paper argues that self-attention works like the Dyson equation from many-body physics, which would explain why sum-pooling models break down on correlated quantum states.

·
·
·
Imperial College Researcher Shows Self-Attention Solves Quantum Pair Correlations Where Neural Nets FailPRO
Read2 min
TypePaper
TopicLlms · Reasoning
  • New paper argues self-attention is a real-space Dyson iteration, with softmax acting as a two-body self-energy.
  • DeepSets sum-pooling is shown to be mean-field (Hartree), so sensitivity to any single other particle shrinks as 1/N.
  • Because Hamiltonians are pairwise, iterating a learned two-body kernel can in principle capture all correlated energetics.
  • On a six-particle fractal backflow target, attention recovers the nested nodal pockets while DeepSets smooths them away.
  • Attention tracks the exact pairwise Jacobian over four decades of separation, but uses 116,226 parameters versus DeepSets' 50,690.
  • Julia code is available in Halcyon.jl; there are no real-molecule or quantum Monte Carlo benchmarks yet.

Many machine-learned interatomic potentials achieve sub-meV-per-atom errors on standard benchmarks by predicting local atomic contributions and summing them into a total energy. Electron wavefunctions and excited states pose a harder problem because their values can depend on specific, widely separated particle pairs. A paper by Jarvist Moore Frost proposes a physical explanation for the difference: self-attention behaves like a real-space Dyson iteration, with its softmax kernel acting as a learned two-body self-energy. The interpretation offers an architecture criterion based on the range and pair specificity of the target correlations.

Where uniform pooling loses the pair

Frost compares two architecture families used in quantum machine learning. Interatomic potentials commonly combine finite-cutoff equivariant message passing with a sum over predicted atomic energies. Neural wavefunction models such as FermiNet instead represent correlated electrons through one-electron and pairwise feature streams. PsiFormer simplifies that design by applying multi-head self-attention to one-electron features, and it has improved correlation estimates for molecular dimers.

An atomic-energy sum readout and a normalized feature mean have different derivatives, so the paper’s argument requires a precise boundary. Summing local energies is the natural way to produce an extensive total energy, and interactions already encoded upstream survive that operation. Frost’s criticism targets finite-cutoff features combined with uniform, permutation-invariant mixing when the desired dependence remains nonlocal and pair-specific.

For a normalized mean update, the derivative of particle i’s output with respect to particle j scales as 1/N. This off-diagonal Jacobian measures how strongly moving one particle changes another particle’s representation. The scaling can suffice for nearsighted ground-state energies, where nearby atoms dominate local contributions. A sharply correlated wavefunction may require one distant particle to exert an order-one effect on another.

The 1/N scaling appears directly in a small NumPy sketch. This example illustrates the distinction and does not reproduce the paper’s implementation:

makefile
import numpy as np

rng = np.random.default_rng(0)
n, d = 6, 8
x = rng.normal(size=(n, d))

# Uniform mixing fixes every pair weight at 1/n.
mean_weights = np.full((n, n), 1.0 / n)

# Attention assigns input-dependent weights to particle pairs.
q = x @ rng.normal(size=(d, d))
k = x @ rng.normal(size=(d, d))
scores = q @ k.T / np.sqrt(d)

shifted = scores - scores.max(axis=1, keepdims=True)
weights = np.exp(shifted)
attention = weights / weights.sum(axis=1, keepdims=True)

# attention[i, j] may approach 1 for a strongly coupled pair.

Pro article

This story is for Pro members

You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.

Trending
  • No trending articles

Comments

avatar

Next Reads