nucleation

THEORY-1172: Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually

2 min read

Properties5
StatusProposed
Tagsloss-landscapes, anthology-candidate
SourceLIT-654 — Linear Mode Connectivity and the Lottery Ticket Hypothesis, LIT-661 — Git Re-Basin: Merging Models modulo Permutation Symmetries, LIT-680 — Proving Linear Mode Connectivity of Neural Networks via Optimal Transport
ExtendsTHEORY-114 — In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training
Bears onQUESTION-312 — Does linear mode connectivity emerge during training rather than at initialization outside image classification and beyond MLPs: in a language model or transformer, for copies spawned from a shared state and for independent runs aligned under the architecture's symmetry group, and for standard-width convnets or ResNets after alignment?

Graph View

Source

Frankle, Dziugaite, Roy & Carbin (2020), LIT-654, read in NOTE-514; and Ainsworth, Hayase & Srinivasa (2023), Git Re-Basin, LIT-661, read in NOTE-532. Ferbach et al. (2024), LIT-680, read in NOTE-543, reconciles the aligned half with connectivity at initialization in principle. What the networks measured showed is THEORY-114, which this entry extends.

The claim

Both papers state the result as a general property of training, not of the networks they ran. Frankle et al. conclude that standard networks become stable to SGD noise early in training, after which copies end in one linearly connected minimum, and that, except in small settings, networks are unstable at initialization (NOTE-514, C1–C2). LeNet on MNIST is their small setting: it is stable from the start. Git Re-Basin concludes that linear mode connectivity modulo permutation “is an emergent property of training, and we were unable to uncover it early in training” (§5.2; NOTE-532, C3). Ferbach et al.’s width bound explains why the two agree with Entezari et al.’s theorem of connectivity at initialization: that needs widths far past any trained in practice.

The evidence is six vision networks for the shared-state half and MLPs on MNIST and CIFAR-10 for the aligned half. The claim is that training, not initialization, makes the straight-line connection in general, in other architectures and tasks as well.

What would settle it

The promote_when, which is QUESTION-312: the onset tracked against training time, with seeds, in a language model or transformer, for both halves, or the aligned onset reproduced for standard-width convnets or ResNets. Connectivity at step 0 under an alignment computed from the initial weights alone, at practical width, would refute it. More vision networks with the same onset curve cannot settle it.

Backlinks