McMahan et al. (2016) — ARXIV-1602.05629
Key takeaways
Introduces Federated Learning and the FederatedAveraging (FedAvg) algorithm, which trains a shared deep network model by aggregating locally-computed SGD updates from mobile devices without centralizing raw data. Demonstrates 10-100x reduction in communication rounds compared to synchronized SGD across multiple model architectures and datasets.
- Communication round reduction (empirical, Sections 3–4). Increasing local epochs E from 1 to 20 reduces required communication rounds by up to 45x on MNIST 2NN and up to 28x on CNN, with comparable or better final accuracy. Holds when: IID data, C=0.1, B=10; gains are smaller but still present under non-IID partitions.
- IID convergence equivalence (informal, Section 3). Under IID data, FedAvg with sufficiently many local steps per round converges to a solution of comparable quality to centralized SGD, because the local gradient estimates are unbiased with respect to the global loss. Holds when: IID data distribution; no formal rate derived in this paper — tight rates established by later work.
What the evidence does not cover
- No theoretical convergence guarantees for non-convex objectives; analysis is purely empirical.
- Performance degrades under highly non-IID data; speedups in the pathological non-IID case are much smaller than IID.
- Does not address client dropout, asynchrony, or communication failures common in real deployments.
- Privacy guarantees are limited; model updates can still leak information, especially for sparse gradient models.
Standing in the anthology
Read — the reading is NOTE-109. Arrived in the imported batch, which brought in the federated branch, where the workers hold different data.