nucleation

THEORY-114: In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training

5 min read

Properties9
StatusActive
Version2
Tagsloss-landscapes, anthology-candidate
SourceLIT-654 — Linear Mode Connectivity and the Lottery Ticket Hypothesis, LIT-661 — Git Re-Basin: Merging Models modulo Permutation Symmetries, LIT-680 — Proving Linear Mode Connectivity of Neural Networks via Optimal Transport
ExtendsTHEORY-109 — The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum
Presupposed byTHEORY-115 — Linear connectivity from a shared, already-stable start holds only when the objectives the runs optimise share structure, and breaks as that structure is removed
Bears onQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?, QUESTION-312 — Does linear mode connectivity emerge during training rather than at initialization outside image classification and beyond MLPs: in a language model or transformer, for copies spawned from a shared state and for independent runs aligned under the architecture's symmetry group, and for standard-width convnets or ResNets after alignment?
Cited as “extends” byTHEORY-1172 — Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually
Cited as “grounds” byCLAIM-524 — Whether functionally equivalent realizations are connected depends on which paths are admissible: for f = u₁v₁ + u₂v₂ with nonnegative weights, (1, 1, 0, 0) and (0, 0, 1, 1) are joined by a curve along which f stays 1 though the straight line between them falls to ½, and under any resource budget below √2 times the endpoints' cost no path with f = 1 joins them

Graph View

Source

  • Frankle, Dziugaite, Roy & Carbin (2020), LIT-654, read in NOTE-514: §§2–3, Figs. 2–4 and Appendices B–C.
  • Ainsworth, Hayase & Srinivasa (2023), Git Re-Basin, LIT-661, read in NOTE-532: §5.2 and Fig. 3.
  • Ferbach, Goujaud, Gidel & Dieuleveut (2024), LIT-680, read in NOTE-543: Lemma 5.1, Theorems 5.2–5.3 and the remark that follows them in §5.1.

The claim, in two halves

From a shared state, after a short time. Frankle et al., LIT-654, train two copies of a network from the same state at step k, with different data order and augmentation. They measure the error barrier on the straight line between the results: the highest error along it minus the mean of the endpoints’. Under 2% counts as none. At k = 0 only LeNet on MNIST is stable. For the other networks, error on the line reaches chance (Fig. 2). They become stable early: VGG-16 on CIFAR-10 by iteration 1000 (1.5% of training), ResNet-20 by 2000 (3%), Inception-v3 on ImageNet by epoch 28 (16%) and ResNet-50 by epoch 18 (20%) (Fig. 3). Resetting the schedule so that the copies train for the full run changes nothing (Fig. 4). At that point the network is far from done: ResNet-20 is still at about 25% test error against 8.3% at the end. Once stable at the end, the copies are linearly connected at every epoch (Appendices B–C). This could have come out otherwise: the copies end more than half as far apart as the whole distance ResNet-20 travels in training, and every point between them is at full accuracy.

Zhou et al. (2025), LIT-370, an earlier reading in this record, found the same two phases with a small parameter perturbation in place of fresh SGD noise. VGG-16 and ResNet-20 on CIFAR-10 sent to different basins before an early inflection point (about 2,500 and 100–500 iterations) and stayed linearly connected after it. That is one run each with no error bars, and its reading notes that it largely re-measures Frankle et al.’s result, so it corroborates this half and is not a source.

Between independent networks, after alignment, gradually. Git Re-Basin, LIT-661, aligns the hidden units of independently initialized networks by permutation before interpolating. It tracks the barrier after alignment through training for MLPs on MNIST and on CIFAR-10. It is high at initialization and falls over training: “LMC manifests gradually throughout training” (Fig. 3). The authors conclude that linear mode connectivity “is an emergent property of training, and we were unable to uncover it early in training” (§5.2). Here there is no onset step as sharp as Frankle et al.’s, and nothing but alignment links the two endpoints.

The tension, and its reconciliation. Entezari et al. (LIT-652, Theorem 3.1) prove that two wide one-hidden-layer ReLU networks at uniform random initialization can be permuted so that the network on the line computes nearly the line between their outputs. That is connectivity at initialization, the opposite of Git Re-Basin’s finding. Ferbach et al., LIT-680, resolve it. They generalize the theorem to deep networks with i.i.d. neuron weights, which holds exactly at initialization (Theorem 5.2). The width of layer ℓ must then exceed Õ((T_ℓ/ε)^{m̃_{ℓ−1}}), exponential in the width of the layer before. A matching lower bound shows the rate is tight (Theorem 5.3). In their words, the condition “appears excessive as compared to the typical width of neural networks used in practice”, and they cite Git Re-Basin’s finding as consistent with it. So both hold: connectivity at initialization exists in principle, at widths no one trains. At practical widths it is training that produces it.

What this does not say

  • The two halves are different measurements. Frankle et al.’s copies share their initialization and their first k steps, and need no alignment. Git Re-Basin’s networks share nothing and are aligned. That both find connectivity appearing in training does not make the onset one event, and neither paper measures one against the other.
  • The aligned half rests on MLPs. Git Re-Basin’s onset curve (Fig. 3) is for MLPs only. The authors say the CIFAR-10 MLP is under-powered and noisy.
  • It does not say what sets the onset time. Frankle et al. connect it to the noisy early phase without measuring the link.
  • It does not exclude connectivity at initialization under some alignment. Git Re-Basin cites concurrent work (Benzing et al. 2022, not held) that finds LMC at initialization using a permutation found at the end of training. The claim is that the barrier under alignment computed from the weights falls in training, not that no permutation could ever join two initializations.
  • Error, not loss. Frankle et al.’s barriers are on classification error, which saturates at chance.
  • Image classification only. Neither half has been measured on a language model or a transformer in the sources.

Connections

  • THEORY-109. This account extends it. That account says the solutions training finds are joined by curves, and that it is a property of found solutions, not of every minimum. This one says the straight-line version is also made by training, and says when: early for a shared stem, gradually for aligned strangers.
  • THEORY-112 says that once symmetries are removed most of the linear barrier between trained solutions disappears. This account is the time axis of its trained-solution restriction. No relation is declared: the shared-stem half involves no symmetry, and the symmetry account is about where training ends.
  • The anthology’s ANTH-THEORY-004, that a rewound lottery ticket wins by landing back in the basin its dense run found, rests on Frankle et al.’s rewinding. This account is the landscape fact under it.

Backlinks