Skip to main content
Back

NTNU Proves Sparsity Fails AI Interpretability and Builds a Geometric Fix

A new geometric regularizer called Coh forces neural network features to tile data manifolds like brain cells, yielding provably interpretable latent spaces accepted at ICML 2026.

Paper, Full Access
  • Researchers introduce coherence, a geometric property requiring neural network features to activate on spatially contiguous regions of the data manifold, not just rare ones.
  • A differentiable loss function Coh, based on Fréchet variance, enforces coherence during training and works with any architecture using non-negative activations.
  • The paper proves coherent matrices guarantee matching topology between sample and feature spaces via bounded Vietoris-Rips interleaving.
  • On rotated MNIST, Coh achieves 100% of features tuned to rotation angle vs 63% for L1 sparsity, with no reconstruction cost.
  • Applied to BERT token embeddings on WikiText-2, Coh yields 87.6/256 interpretable features (categories like years, kinship terms, verb types) vs 0 for non-negative baseline alone.
  • Accepted at ICML 2026; full paper at arxiv.org/abs/2606.02841.

The interpretability problem in deep learning has a dirty secret: even when individual neurons activate rarely, they can still fire for semantically scattered, unrelated inputs. A sparse feature that activates on only 2% of data is not automatically meaningful if those inputs are spread across disconnected regions of the data manifold. A new paper accepted at ICML 2026 from researchers at NTNU argues that the field has been optimizing the wrong geometric property, and proposes a provably better alternative.

The Gap Sparsity Cannot Fill

The dominant approach to interpretability in representation learning is sparsity regularization. Sparse autoencoders (SAEs), now a cornerstone of mechanistic interpretability research, work by forcing features to activate on as few samples as possible. The idea is that rare activations are easier to label and understand. But sparsity only constrains how many samples a feature activates on, not which ones.

For unsupervised approaches such as auto-encoders, without class labels to guide separation, networks can learn representations where semantically related samples are spread across the latent space, and where individual features lack any coherent meaning. Sparsity regularization encourages features to activate on few samples, but does not ensure those samples are geometrically related; a sparse feature may fire on disconnected regions of the data manifold.

The paper introduces a new concept called coherence, a geometric constraint inspired by how neurons actually work in biological brains.

Lessons From Grid Cells

The key insight comes from neuroscience. Grid cells in the entorhinal cortex tile physical space with periodic firing fields. Head direction cells activate for specific orientations, with each neuron covering a contiguous arc of angles. These neural codes exhibit locality: each neuron's activity is concentrated on a coherent region of the underlying state space. This locality is precisely what makes these cells interpretable; one can read off an animal's position or heading from the concurrent neural activity because the mapping between state and response is geometrically organized.

The authors ask: can we formalize this locality principle and impose it on artificial neural networks? Their answer is the Coh objective function.

What Coherence Actually Means

Think of the latent space of an autoencoder as a matrix M where rows are samples and columns are features. Coherence is a property of this matrix with two components:

  • Locality (epsilon-local):