FedMTFI Fixes Federated Learning's Biggest Flaw Without Touching Raw Data
FedMTFI combines multi-teacher knowledge distillation with SHAP feature importance to tackle the hardest problem in federated learning: heterogeneous devices with skewed data
Takeaways
- FedMTFI is a new federated learning framework that clusters devices by hardware, builds per-cluster prototype models, then distills them into one global student model.
- It uses SHAP (Shapley) values to weight the distillation loss, making the student focus on the most informative features across all teacher models.
- The approach directly targets the two hardest FL problems: non-IID data skew and device heterogeneity (different compute/memory across clients).
- Experiments on MNIST and Fashion-MNIST show FedMTFI outperforms standard FL algorithms like FedAvg under non-IID conditions, converging within 30 rounds.
- The SHAP integration adds interpretability on top of accuracy gains, a rare combination in federated learning research.
- No code or model weights have been publicly released; the paper is available at arxiv.org/abs/2606.01607.
Federated learning (FL) was supposed to be the answer to privacy-preserving AI: train models across thousands of devices without ever moving raw data. But the real world is messy. Devices differ wildly in compute power, memory, and the data they hold is almost never evenly distributed. A new paper introduces FedMTFI, a framework that attacks both problems at once by combining multi-teacher knowledge distillation with SHAP-based feature importance scoring.
The wall that stops federated learning from working
The standard FL playbook, FedAvg, works by having each device train a local model and then averaging all those model weights on a central server. Simple, elegant, and fragile under real-world conditions. FedAvg aggregates client models based on the number of local samples, but exhibits suboptimal performance in scenarios involving non-IID data or heterogeneous devices.
Non-IID (non-independent and identically distributed) data is the core villain here. It means that each device's local dataset looks nothing like the global distribution. A hospital in one city sees mostly elderly patients; one in another sees mostly children. The global model in FL often suffers from performance degradation due to global knowledge forgetting caused by highly non-IID client data. This knowledge forgetting is asymmetric, with locally under-represented classes suffering significantly more severe performance degradation than over-represented ones.
On top of that, system heterogeneity in FL causes the straggler issue, where resource-limited edge devices delay global model aggregation. You can't force a Raspberry Pi to run the same model as a server-grade GPU. Existing approaches either ignore device differences or require a shared public dataset on the server, which is often unavailable in practice.