D2MoE Cuts Graph AI Memory 73% by Giving Hard Nodes More Compute

A new mixture-of-experts framework for graph neural networks argues that not every node deserves the same amount of compute. D2MoE (Difficulty-Aware Dynamic Mixture-of-Experts) replaces the fixed top-k routing that dominates current Graph MoEs with an entropy-guided top-p mechanism, giving hard nodes a bigger expert ensemble while letting easy ones coast through with a single expert. The result: state-of-the-art accuracy across 13 benchmarks, up to 7.92% gains on heterophilous graphs, 73.07% less memory and 46.53% less training time on large graphs compared to the best-performing Graph MoE.

The one-size-fits-all problem in Graph MoEs

Mixture-of-Experts has become a popular way to scale GNN capacity without paying the full dense-compute bill, but existing designs are surprisingly rigid. Current Graph MoEs typically rely on static routing strategies that enforce a uniform expert budget or coarse-grained toggles across every node. Frameworks like GMoE use a fixed budget for every node, while Mowst applies binary strong/weak toggles.

The authors ran a motivational study using a vanilla top-k router with six GCN experts, stratifying test nodes into deciles by predictive entropy from an independent teacher. Two patterns emerged consistently:

• Capacity saturation on easy nodes. Adding experts to homophilous, low-entropy nodes did nothing except waste compute and inject noise. A single expert matched or beat larger ensembles.
• Capacity starvation on hard nodes. Nodes sitting at heterophilous boundaries needed larger ensembles (k=3, k=5) to disentangle conflicting neighborhood signals. A small fixed budget capped their accuracy.