Skip to main content
Back

PSN Framework Proves Fan-In Profiles Are Basically Free Parameters at 90% Sparsity

New research shows heterogeneous sparse connectivity only helps when placed by gradients, and RigL's equilibrium topology can be predicted and used as initialization.

Pro

Paper, Full Access
  • New Profiled Sparse Networks paper tests heterogeneous fan-in distributions across four datasets at 80-99.9% sparsity.
  • Static heterogeneous connectivity gives no accuracy advantage: all profiles land within 0.2-0.6% of dense baselines.
  • Structured profiles create 2-5x gradient concentration at hub neurons, predicted by fan-in CV (r=0.93).
  • Lognormal initialization matched to RigL's equilibrium fan-in beats ERK by up to +0.49% on Forest Cover.
  • RigL converges to a characteristic fan-in distribution regardless of initialization, enabling analytical starting points.
  • ERK badly underperforms on low-dimensional inputs like Forest Cover due to first-layer fan-in starvation.

A new arXiv paper takes a systematic swing at a question that has quietly haunted sparse neural network research for years: does it matter how you distribute the surviving connections across neurons, or just how many you keep? After four datasets, eight profile shapes, and sparsity levels from 80% up to 99.9%, the answer is nuanced. Random placement of hub neurons buys you nothing. But if you know where the hubs should go, you can skip a lot of expensive topology search.

The work introduces Profiled Sparse Networks (PSN), a framework that treats per-neuron fan-in as a controllable architectural variable rather than a byproduct of training. Instead of the uniform random sparsity used by most methods, PSN assigns fan-in deterministically via a profile function that maps neuron index to connectivity density, producing a mix of dense hub neurons and sparse specialist neurons within each layer. Exponential decay, bell curves, lognormal tails, power laws, all with the same total parameter budget, just distributed differently.

A negative result worth publishing

At 90% sparsity, every profile the authors tried, including the plain random baseline, landed within 0.2 to 0.6% of the dense model on every dataset. The spread across all model variants (8 profiles by 2 spreading patterns, plus random) was only around 0.2%, with fully overlapping confidence intervals and no pairwise comparison reaching statistical significance.