PC-ALM trains 1,000-layer networks with local credit signals

Sakana AI has released PC-ALM, a predictive-coding method that trained residual multilayer perceptrons as deep as 1,000 layers while exchanging learning signals only between adjacent layers. On MNIST, its width-32 models remained within about two percentage points of backpropagation at every tested depth. The result extends local credit assignment beyond standard predictive coding in the paper’s experiments, although each mini-batch requires a depth-dependent settling process.

Why local credit fades

Backpropagation computes a loss at the output, then applies the chain rule through the network in reverse. A layer’s gradient depends on downstream gradients, creating an ordered, network-wide dependency. Conventional accelerators handle that pattern efficiently. Models of biological neurons and many neuromorphic designs favor local state changes, making the reverse sweep a poor physical fit.

Predictive coding assigns each layer an activation state and a local prediction error. During training, the network repeatedly adjusts those states so neighboring predictions become consistent while the output responds to the target. Supervision applied at the output must propagate through many rounds of local updates before it affects early layers. In deep, narrow networks, the signal weakens as it diffuses. The authors report that standard predictive coding deteriorates sharply in this regime and note that earlier demonstrations reached about 128 layers with wider hidden states.

Dual neurons carry the signal

PC-ALM reformulates the network as a constrained optimization problem in which each layer’s state should match the prediction produced by the preceding layer. An augmented Lagrangian combines a quadratic penalty for violating that constraint with a Lagrange multiplier. PC-ALM represents each multiplier as a vector of dual neurons that accumulates residual prediction errors over time.

A 1988 result from Yann LeCun showed that the Lagrange multipliers of a constrained network correspond at equilibrium to the credit signals calculated by backpropagation. Sakana AI’s method combines those multipliers with predictive coding’s existing quadratic penalties. For linear networks, the authors prove that the resulting local dynamics recover exact supervised-loss gradients at equilibrium.

The method’s formal guarantee covers linear networks. Its nonlinear residual MLP and ResNet experiments provide empirical evidence rather than an exact equivalence theorem.

A proportional-integral controller offers another interpretation of the dynamics. The current prediction error supplies the proportional term, while the dual variable stores accumulated error as the integral term. A network of these local controllers distributes output credit through adjacent-layer interactions.

Inside one mini-batch

Each mini-batch runs repeated primal and dual updates before applying one local weight update:

1. Primal settling: each layer adjusts its activation state using prediction errors, neighboring states, and its current dual value.
2. Dual integration: each multiplier increases according to the remaining constraint violation, preserving an accumulated error signal.
3. Local weight change: after settling, each layer applies a Hebbian-style update based on presynaptic activity and postsynaptic error.

A layer reads only its own variables and messages from immediate neighbors. The scheme replaces the ordered backward sweep with parallel, iterative settling. The authors set the inference budget to T = 2L, where L is network depth, so the 1,000-layer experiments use 2,000 settling steps before each weight update.

A 1,000-layer stress test

The deepest experiment uses residual MLPs with a hidden width of 32, a demanding setting because narrow layers offer little spare capacity for weak credit signals.

Setting	Reported outcome
MNIST, width-32 residual MLPs through 1,000 layers	PC-ALM stays within about two percentage points of backpropagation across the tested depths. Standard predictive coding degrades as depth increases.
Fashion-MNIST, CIFAR-10, and Tiny ImageNet with ResNet-18	PC-ALM consistently reduces the performance gap between standard predictive coding and backpropagation.

Inference traces show PC-ALM’s credit signal moving from the output toward the input as a wavefront. The authors call this ballistic credit propagation. Standard predictive coding behaves more like heat diffusion, spreading supervision gradually across layers. The wavefront behavior allows PC-ALM to reach the earliest layers with a settling budget proportional to network depth.

The tradeoffs are concrete

• Benchmark scope: the study covers image classification with residual MLPs and ResNet-18. Transformers, language models, and production-scale workloads remain untested.
• Settling cost: local communication still requires many iterative updates. The 1,000-layer model performs 2,000 settling steps per mini-batch before changing its weights.
• Hardware evidence: wall-clock and energy comparisons on neuromorphic systems are still needed.
• Stability: an excessive dual step size can produce oscillations and destabilize inference.
• Learning behavior: standard predictive coding can settle on activations that differ from the initial forward pass, a property called prospective configuration that may improve sample efficiency. PC-ALM gives up that behavior at convergence to strengthen credit propagation.
• Biological interpretation: adjacent-layer communication addresses one objection to backpropagation. Evidence that brains implement equivalent dual variables and update rules remains absent.

Where PC-ALM could fit

For neuroscience, PC-ALM supplies a computational model in which adjacent-layer dynamics recover backpropagation-equivalent credit variables for linear networks. For hardware research, its local communication and parallel settling offer a candidate mapping to analog and neuromorphic systems, pending physical benchmarks. Developers can inspect the paper and reproduce the experiments with the public source code.