nucleation

QUESTION-312: Does linear mode connectivity emerge during training rather than at initialization outside image classification and beyond MLPs: in a language model or transformer, for copies spawned from a shared state and for independent runs aligned under the architecture's symmetry group, and for standard-width convnets or ResNets after alignment?

2 min read

Properties5
StatusOpen
Tagsloss-landscapes, learning-theory
Linelearned-structure
RefinesQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?
Cited as “bears on” byTHEORY-114 — In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training, THEORY-1172 — Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually

Graph View

Why it is a question

THEORY-114 holds that linear mode connectivity appeared during training and not at initialization, in the settings its sources measured. Frankle et al.’s onset covers six vision networks, and Git Re-Basin’s aligned onset covers MLPs on MNIST and CIFAR-10 only. Whether the same holds for other architectures and tasks is not shown.

What would answer it

THEORY-114’s former promote_when, unchanged:

The onset measured outside image classification, and the aligned half measured on more than MLPs. Frankle et al.’s onset covers six vision networks, and Git Re-Basin’s aligned onset covers MLPs on MNIST and CIFAR-10 only. What would promote: in a language model or a transformer, the barrier tracked against training time, with seeds, both between copies spawned from a shared state and between independently initialized runs aligned under the architecture’s symmetry group, with each high at step 0 and falling to near zero later. Or the aligned onset reproduced for standard-width convnets or ResNets. What would refute it: independently initialized networks of practical width that are linearly connected after alignment at step 0, under an alignment computed from the initial weights alone. Or copies spawned at initialization of a standard deep network that end up linearly connected. More vision networks with the same onset curve cannot settle it.

Backlinks