FedInit Fixes Federated AI Drift 2x Faster Without Extra Communication Cost
A new federated learning trick called FedInit nudges local models backwards before training, cutting client drift with zero extra communication cost.
Takeaways
- FedInit fixes federated client drift by initializing each local round with a backward nudge from the previous local state.
- Formula:
w_i = w_global + beta * (w_global - w_i_previous), controlled by one scalar coefficient beta. - Zero extra communication or storage overhead versus FedAvg, unlike SCAFFOLD or FedCM which double bandwidth.
- Theory shows client drift mainly hurts generalization error, not optimization error, reframing prior FL analysis.
- Beats FedAvg by 3.42 points on non-IID CIFAR-10; SCAFFOLD plus RI gains 2 to 3 points, reaching state-of-the-art.
- Works as a drop-in plug-in for FedSAM, SCAFFOLD, FedAdam and other advanced federated optimizers.
Federated learning has a stubborn problem. Send a shared model out to hundreds of phones, hospitals, or edge devices, let each one train on its own local data, and the copies drift apart. Averaging them back together produces a global model worse than what centralized training would give you. A new paper introduces FedInit, a simple fix that changes only how each round of local training is initialized, along with the first tight theoretical account of why the fix helps generalization more than it helps optimization.
The drift that quietly wrecks federated models
Prior work has documented that FL suffers from client drift, caused by inconsistent optima across local clients, but solid theoretical analysis of that inconsistency has been missing. In plain terms, each client optimizes toward its own local best point, and those points sit in different places because the data is heterogeneous. When the server averages the models, you get something that is nobody's optimum.
Existing fixes throw more machinery at the problem. Methods like SCAFFOLD, FedProx, and FedDyn modify local training to align client updates with the global objective, but many of them double communication bandwidth or memory by shipping extra control variates alongside the model weights. For deployments with millions of edge devices, that is a serious tax.
Initialize backwards, then train forwards
FedInit's idea fits on one line. At the start of each round, instead of initializing every selected client to the current global model w^t, initialize it to:
w_{i,0}^t = w^t + beta * (w^t - w_{i,K}^{t-1})
Take the global model and push it further away from where this client landed last time. FedInit calls this relaxed initialization (RI): each active client's starting point moves away from the current global state in the reverse direction of its most recent local state. A single scalar