---
title: "ITMO Builds a Tool That Shows What Your Optimizer Misses About Curvature"
subtitle: "A new formalism cracks open the Hessian of arbitrary neural architectures, exposing per-layer curvature interactions that standard autodiff hides."
url: "https://alphasignal.ai/news/itmo-builds-a-tool-that-shows-what-your-optimizer-misses-about-curvature"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-13T00:00:00.000Z"
updated: "2026-08-24T03:10:22.478Z"
type: "Paper"
topics:
  - "Training Infra"
  - "Benchmarks"
subtopics:
  - "Scaling Laws"
  - "Distributed Training"
upvotes: 907
source_url: "https://arxiv.org/abs/2604.11639"
comments: 0
truncated: true
---

# ITMO Builds a Tool That Shows What Your Optimizer Misses About Curvature

> A new formalism cracks open the Hessian of arbitrary neural architectures, exposing per-layer curvature interactions that standard autodiff hides.

If you have ever tried to understand why second-order optimizers work well on some networks and fall apart on others, you have probably hit the same wall: modern autodiff frameworks give you the Hessian as one giant opaque matrix. A paper from researchers at ITMO University proposes an [analytical decomposition](https://arxiv.org/abs/2604.11639) that indexes the Hessian by the computation graph itself, turning that monolithic object into a set of interpretable inter-layer blocks with quantitative diagnostics.

## The blind spot in every autodiff framework

Modern autodiff frameworks such as JAX and PyTorch return the Hessian of the loss as a single tensor, without exposing the internal structure of inter-layer interactions. The paper presents an analytical formalism that decomposes the full matrix into blocks indexed by the DAG of an arbitrary architecture. A lot of what we care about, including saddle points, curvature coupling between layers, and why skip connections stabilize training, lives in the cross-terms between layers rather than on the diagonal.

Existing tools work around this rather than solving it. K-FAC and Shampoo approximate block-diagonal pieces. Pearlmutter-style HVP gives you exact Hessian-vector products but reveals nothing about which layers are interacting. Per-layer spectral analyses look at diagonal blocks in isolation. None of these expose the off-diagonal structure between arbitrary node pairs in a DAG.

## Splitting curvature into convex and residual parts

The core contribution is a canonical decomposition: `H = H^GN + H^T`. This separates the Gauss-Newton component, the convex part, from the tensor component, which is the residual curvature responsible for saddle points. The GN component always sits inside the positive semi-definite cone, which is exactly why optimizers built on it, with K-FAC as the poster child, are structurally blind to negative curvature.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures (arXiv abstract)](https://arxiv.org/abs/2604.11639)
- [Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures (full HTML)](https://arxiv.org/html/2604.11639)
- [dag-hesse GitHub repository](https://github.com/comiam/dag-hesse)
- [Local properties of neural networks through the lens of layer-wise Hessians](https://arxiv.org/abs/2510.17486)