Fisher information sets a speed limit for SGD learning

Two Kyoto University researchers have applied a tool from statistical physics to neural network training. Their preprint derives an upper bound on how quickly trainable parameters can acquire information about the latent variables that generated the training data. The bound also predicts which features a model learns first under tractable linear dynamics.

Each mini-batch produces a random gradient, so repeated runs of stochastic gradient descent (SGD) induce a changing probability distribution over parameter vectors. Modeling that evolution as a stochastic process lets the authors use information-flow speed limits, which constrain how quickly probability distributions can change under noisy dynamics.

Measuring information in the weights

The analysis begins with data generated by latent variables Z, such as a signal’s amplitude, frequency, or phase. As training proceeds, the conditional parameter distribution p(θ | Z) becomes sensitive to those variables. The authors quantify that sensitivity with Fisher information.

For a scalar latent variable, Fisher information measures the expected squared sensitivity of log p(θ | Z) to changes in Z. For several latent variables, it becomes a matrix. Through the Cramér-Rao inequality, greater Fisher information corresponds to a lower bound on the uncertainty of estimates of Z derived from the trained parameters.

Earlier information-theoretic work on deep learning has often tracked signals across network layers. This framework tracks information entering the trainable parameters during optimization, including the rate and order in which specific latent factors become statistically accessible.