Modulation Networks Beat ReLU on Derivative Accuracy for Physics AI
A new approximation theory shows neural networks with Gaussian time-frequency windows on each neuron beat plain ReLU nets at learning both functions and their derivatives.
ProTakeaways
- New shallow architecture multiplies each ReLU unit by two Gaussian windows for spatial and frequency localization.
- Proves dimension-free
N^(-1/2)approximation rate in general Sobolev normsW^{n,r}on bounded and unbounded domains. - Recovers Barron, Feichtinger, Shubin-Sobolev, and Fourier-Lebesgue approximation results as corollaries with explicit constants.
- Extends Siegel and Xu's
H^nBarron bounds to arbitrary Sobolev norms and any dimensiond. - 1D and 2D experiments over 100k epochs show markedly lower
H^1loss than parameter-matched plain ReLU networks. - Read the full paper on arXiv.
Shallow ReLU networks are excellent function fitters, but they behave poorly when you also need their derivatives to be accurate. That gap matters for scientific machine learning: physics-informed networks, neural PDE solvers, and variational methods all measure error in Sobolev norms, which penalize errors in both the function and its partial derivatives. A new paper by Ahmed Abdeljawad and Elena Cordero, forthcoming in Applied and Computational Harmonic Analysis, builds a full approximation theory for a modified shallow architecture that provably closes that gap and, in numerical tests, does so by a wide margin.
Why standard theory falls short for PDEs
Most quantitative approximation results for neural networks are stated in L^p norms, which only measure pointwise fitting error. Scientific computing needs an approximant that faithfully captures both the target function and its derivatives up to a given order, which shifts the focus from Lebesgue-type error measurements to Sobolev norms.
The classical workaround for the curse of dimensionality is the Barron space, which characterizes a function by a spectral moment of its Fourier transform and yields a dimension-free N^(-1/2) approximation rate. This view is purely frequency-based. The Fourier-only viewpoint underlying spectral Barron spaces struggles with functions that have nontrivial time-frequency localization, meaning functions whose behavior is constrained in both space and frequency. Results on unbounded domains with derivative control are also scarce.