CMU and Google's FM-SIREN Cuts Hidden Redundancy by 50% Without Extra Parameters
A CMU and Google team fixes a hidden bottleneck in periodic-activation INRs by giving each neuron its own Nyquist-informed frequency, cutting feature redundancy roughly in half.
ProTakeaways
- CMU and Google propose FM-SIREN and FM-FINER, giving each neuron a Nyquist-derived frequency multiplier.
- The change cuts hidden feature redundancy by roughly 50% versus SIREN and FINER baselines.
- Image fitting gains reach 9.44 dB PSNR on BSDS500 with only two-layer networks.
- 3D shape fitting reduces Chamfer Distance by up to 55.15% versus the strongest baseline.
- No extra parameters, no hyperparameter tuning, and training time matches the original baselines.
- Limitation: assumes Cartesian sampling, so NeRF-style irregular coordinates see smaller gains.
Implicit neural representations have quietly become one of the most versatile tools in modern signal modeling, powering everything from NeRF-style view synthesis to neural image and video codecs. The workhorse architectures behind them, SIREN and FINER, have carried a subtle inefficiency for years: every neuron in a layer is scaled by the same frequency multiplier, so neurons end up learning near-duplicate views of the signal. A new paper from Carnegie Mellon and Google proposes an almost embarrassingly simple fix, and the reconstruction gains are large.
The authors introduce FM-SIREN and FM-FINER, drop-in variants that assign each neuron its own frequency multiplier derived from the Nyquist sampling theorem. Hidden feature redundancy drops by 49.92% and 50.43% respectively, as measured by the Frobenius norm of the hidden embedding covariance matrix. That translates to consistent quality gains on audio, images, 3D shapes, and video, without adding parameters, depth, or hyperparameter tuning.
The redundancy hiding inside SIREN
Understanding the fix requires seeing the failure mode first. In a SIREN layer, every sinusoidal activation uses the same frequency scale (typically written as omega_0 or beta, often 30). All neurons therefore behave as replicas of the same basis function, differing only through learned weights and biases. The optimizer is expected to sort out diversity through gradient descent alone.
That expectation is unrealistic. The authors point out a striking baseline: a single-layer SIREN, which has the same parameter count and architecture as a Discrete Sine Transform, cannot match the DST's closed-form reconstruction of a simple image. The DST assigns each basis function a distinct, harmonically spaced frequency by construction, so its components are mutually orthogonal. SIREN has to hope its optimizer will discover that structure, and in a non-convex loss landscape, it usually does not.