nucleation

LIT-654: Linear Mode Connectivity and the Lottery Ticket Hypothesis

3 min read

Properties6
StatusActive — read in full 2026-10-03 (NOTE-514); the source of linear mode connectivity as a measurement and of instability analysis. Two copies of a network trained from the same state under different SGD noise end up linearly connected once that state is a little way into training (3% for ResNet-20, 20% for ResNet-50 on ImageNet), and iterative-magnitude-pruning subnetworks are matching only when they are stable in this sense. The evidence is correlational: stability and matching coincide across the settings tested, and nothing shows that one causes the other.
Tagsloss-landscapes, anthology-candidate
ExtendsLIT-673 — Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, LIT-653 — Essentially No Barriers in Neural Network Energy Landscape
Extended byLIT-651 — Linear Mode Connectivity in Multitask and Continual Learning, LIT-652 — The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, LIT-669 — Mechanistic Mode Connectivity, LIT-674 — Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Cited as “paper” byNOTE-514 — Linear Mode Connectivity and the Lottery Ticket Hypothesis
Cited as “source” byTHEORY-114 — In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training, THEORY-1172 — Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually

Graph View

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy and Michael Carbin (2020), Proceedings of the 37th International Conference on Machine Learning, PMLR 119 — ARXIV-1912.05671

Key takeaways

  • Linear mode connectivity as a measurement. The error barrier on the straight line between W₁ and W₂ is the highest error along it minus the mean of the endpoint errors. Two networks are linearly mode connected when that is about zero; the paper takes under 2% as zero (§2).
  • Networks become stable to SGD noise early in training. Train two copies from step k with different data orders and augmentation. At k = 0 only LeNet on MNIST is stable; for the rest, error on the line reaches chance. ResNet-20 and VGG-16 on CIFAR-10 become stable by iteration 2000 (3%) and 1000 (1.5%); ResNet-50 and Inception-v3 on ImageNet by epoch 18 (20%) and 28 (16%) (§3, Fig. 3). Restarting the learning-rate schedule so that the copies train for the full T steps changes nothing (Fig. 4).
  • The linearly connected region is large. At the point of stability, ResNet-20 is still at about 25% test error against 8.3% at the end, and the two copies end up more than half as far apart as the whole distance the network travels in training, with every point between them at full accuracy (App. B). Once stable at the end, copies are stable at every epoch of training (App. C).
  • Lottery tickets are matching only when stable. At extreme sparsities, IMP subnetworks rewound to initialization match full accuracy only where they are stable (LeNet, and the low-learning-rate and warmup variants of ResNet-20 and VGG-16). Rewound to an early iterate instead, the IMP subnetworks of standard ResNet-20, VGG-16, ResNet-50 and Inception-v3 become stable, and become matching at about the same rewinding point (§4.3, Figs. 6–7). Randomly pruned or reinitialized subnetworks are neither.

Standing in the record

Filed with the batch ADR-027 opened loss-landscapes for. It extends mode connectivity as Garipov et al. (LIT-673) and Draxler et al. (LIT-653) found it. Those papers joined independently trained networks by curved or piecewise paths and found the straight line between them high; this paper asks when the straight line is enough, and answers: between networks that share the early part of their trajectory. It takes the two papers’ definition of mode connectivity, and calibrates its 2% threshold on the error increases along their paths (Draxler et al.’s Table B.1, Garipov et al.’s Table 2).

The line then splits. Entezari et al. (LIT-652) adopt this paper’s definition of LMC, with a barrier measured against the interpolated rather than the mean endpoint loss, and ask whether networks that share nothing become linearly connected once their hidden units are permuted. Git Re-Basin (LIT-661) finds that such connectivity, too, appears only after some training.

This record already leaned on the paper before holding it. LIT-370 (Zhou et al. 2025, read 2026-09-30) found an early chaotic phase followed by a stable one, and its reading noted that the chaos half “largely re-measures Frankle et al.’s (2020) spawning / linear-mode-connectivity result” with a parameter perturbation in place of fresh SGD noise.

In the anthology, the original lottery-ticket paper is ANTH-LIT-019, and ANTH-THEORY-004 holds that a rewound ticket wins by landing back in the basin its dense run had already found. This paper’s result that IMP subnetworks become matching exactly when they become stable to SGD noise fits that account, and its rewinding procedure is what makes the ImageNet tickets exist. The paper bears on pruning practice, so it carries the anthology-candidate flag.

Backlinks