Skip to main content
Back

ITMO Builds a Tool That Shows What Your Optimizer Misses About Curvature

A new formalism cracks open the Hessian of arbitrary neural architectures, exposing per-layer curvature interactions that standard autodiff hides.

Pro

  • New formalism decomposes the neural network Hessian into DAG-indexed blocks, exposing inter-layer curvature interactions hidden by standard autodiff.
  • Canonical split H = H^GN + H^T isolates the convex Gauss-Newton part from the residual curvature responsible for saddle points.
  • Four diagnostics (resonance, coupling, stable rank, GN-Gap) computable in O(P) via Hessian-vector products.
  • Proves exponential resonance decay in vanilla nets and its preservation under skip connections, quantifying why ResNets train deeper.
  • Softmax attention creates dense Q-K cross-blocks with GN-Gap around 69, versus 10^-8 for ReLU MLPs.
  • PyTorch code released at github.com/comiam/dag-hesse, validated on ResNet-18 with 11M parameters.

If you have ever tried to understand why second-order optimizers work well on some networks and fall apart on others, you have probably hit the same wall: modern autodiff frameworks give you the Hessian as one giant opaque matrix. A paper from researchers at ITMO University proposes an analytical decomposition that indexes the Hessian by the computation graph itself, turning that monolithic object into a set of interpretable inter-layer blocks with quantitative diagnostics.

The blind spot in every autodiff framework

Modern autodiff frameworks such as JAX and PyTorch return the Hessian of the loss as a single tensor, without exposing the internal structure of inter-layer interactions. The paper presents an analytical formalism that decomposes the full matrix into blocks indexed by the DAG of an arbitrary architecture. A lot of what we care about, including saddle points, curvature coupling between layers, and why skip connections stabilize training, lives in the cross-terms between layers rather than on the diagonal.

Existing tools work around this rather than solving it. K-FAC and Shampoo approximate block-diagonal pieces. Pearlmutter-style HVP gives you exact Hessian-vector products but reveals nothing about which layers are interacting. Per-layer spectral analyses look at diagonal blocks in isolation. None of these expose the off-diagonal structure between arbitrary node pairs in a DAG.

Splitting curvature into convex and residual parts

The core contribution is a canonical decomposition: H = H^GN + H^T. This separates the Gauss-Newton component, the convex part, from the tensor component, which is the residual curvature responsible for saddle points. The GN component always sits inside the positive semi-definite cone, which is exactly why optimizers built on it, with K-FAC as the poster child, are structurally blind to negative curvature.

The tensor component