Skip to main content
Back

Researchers Prove Sharp Minima Beat Flat Ones, Upending 30 Years of Deep Learning

New empirical work argues sharp minima are not inherently bad. They often reflect a model learning a more complex function, and can beat flat minima on generalization and reliability.

  • Paper argues sharpness is a function-dependent property, not an indicator of poor generalization in neural networks.
  • Regularizers like weight decay, augmentation, and SAM often produce sharper minima that generalize better than flat baselines.
  • On ResNet-18 CIFAR-10, augmentation plus SAM raises Relative Flatness 140x while boosting test accuracy from 72% to 91%.
  • SAM increases global sharpness under reparameterization-invariant metrics despite being designed to promote flatness.
  • Toy experiments show target functions like Rosenbrock have intrinsically sharp minima regardless of optimizer choice.
  • No universal Goldilocks zone exists for sharpness across architectures and datasets, warning against aggregate claims.

For nearly three decades, deep learning folklore has held that flat minima generalize better than sharp ones. The intuition, borrowed from Occam's Razor and Minimum Description Length, is that a wide basin of low loss corresponds to a simpler solution that tolerates parameter noise. A new empirical study, A Function-Centric Perspective on Flat and Sharp Minima, pushes back hard on that story, arguing that sharpness is a property of the function being learned rather than a warning sign about the model.

Cracks in the flatness dogma

The flatness-equals-generalization view has driven concrete engineering decisions, most visibly the design of Sharpness Aware Minimization (SAM), which was explicitly motivated by biasing training toward flatter regions of the loss landscape. The connection has always been fragile though. Dinh et al. showed that flat minima, under Hessian-based definitions, can be arbitrarily sharpened through reparameterisation without changing the model's learned function or generalisation behaviour. That triggered work on reparameterisation-invariant metrics like Fisher-Rao norm and Relative Flatness, which supposedly restored the link.

The authors ask a sharper question: even under those invariant metrics, does high sharpness really mean poor generalization, or does it just mean the model is fitting a more intricate function?

Toy problems break the story first

The paper starts with single-objective optimization on classical test functions. Some functions like Sphere have flat global minima, while others like Rosenbrock have inherently sharp ones. The geometry of the solution space is tied to function complexity, not optimality alone. An MLP fit to each objective converges to a sharpness that matches the target function itself. Chasing flatness on an intrinsically sharp objective is just fighting the geometry.