Tailin Zhou, Jun Zhang and Danny H. K. Tsang (2023), arXiv preprint — ARXIV-2309.16923
Key takeaways
- Heterogeneity shrinks the overlap between client and global solutions. On VGG11 with CIFAR-10 and PACS, the region where a client model and the global model both have low loss is smaller under α = 0.1 than under α = 0.5 (Dirichlet label skew), and smaller on PACS (§IV, Figs. 2–3). Client models stay near the global model throughout training but move in different directions within a round (Fig. 4).
- Global models are distinct modes in one region. FedAvg models from one initialisation trained under α = 0.1, 0.2 and 0.5 disagree on more than 20% of predictions, are closer to each other than to their initialisation, show an error barrier on the line that grows with heterogeneity and over rounds, and are joined without barrier by a one-bend polygonal chain (§V, Figs. 5–7; accuracy drops of about 10% on the line, App. A-B).
- A bound in heterogeneity and width. Modelling heterogeneity as Gaussian noise of variance βh(α) on each neuron’s FedAvg update (Assumption 5), the global model of a two-layer network is ε_D-dropout stable and two such models are ε_C-connected by a 7-segment path, with ε_D, ε_C ∝ √h(α)·((√log N + z)/√N + √ε(√(D + log N) + z)) (Theorems 1–2). Dropout error and the linear barrier fall with width, from N = 200 to N = 6400, in MNIST runs (§VII, Fig. 9).
Standing in the record
Filed with the owner’s batch on mode connectivity and model merging, under
loss-landscapes (ADR-027). Federated averaging is weight averaging done
repeatedly, so this is the batch’s view of merging from inside training:
the global model is the average of client models, and the question is
whether the average lies where the clients’ losses are low.
It extends Kuditipudi et al. (LIT-671). Theorem 2 is their result that dropout-stable networks are connected by a short piecewise-linear path, applied to FedAvg models (“Following [20], we demonstrate that dropout-stable NNs in FL have mode connectivity”), with the dropout stability supplied by a mean-field analysis after Shevchenko and Mondelli (2020) and Mei et al., none of which the record holds. The paper uses Garipov et al.’s path objective and polygonal chain (LIT-673) for its non-linear paths, and cites Entezari et al. (LIT-652) for permutation as a possible source of the linear barrier, which its function-dissimilarity measurement argues against for these models.
The anthology holds FedAvg (ANTH-LIT-307). The batch’s GNN reading (LIT-676) makes a parallel argument, with graph homophily in place of client heterogeneity as the data property that sets the barrier. Mirzadeh et al. (LIT-651) report the opposite outcome for a shared start: their multitask and continual solutions are linearly connected, while these global models, also from one start, are not.