nucleation

THEORY-1188: Once the architecture's symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width

2 min read

Properties5
StatusProposed
Tagsloss-landscapes, anthology-candidate
SourceLIT-652 — The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, LIT-661 — Git Re-Basin: Merging Models modulo Permutation Symmetries, LIT-680 — Proving Linear Mode Connectivity of Neural Networks via Optimal Transport, LIT-664 — Generalized Linear Mode Connectivity for Transformers, LIT-655 — Understanding Mode Connectivity via Parameter Space Symmetry
ExtendsTHEORY-112 — Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42)
Bears onQUESTION-317 — Does aligning independently trained networks over the architecture's full symmetry group remove most of the linear barrier at standard width, on ImageNet or a language corpus, with an alignment computed without data; is the barrier that remains a difference in what the endpoints compute; and does it rise with depth?

Graph View

Source

  • Entezari, Sedghi, Saukh & Neyshabur (2022), LIT-652, read in NOTE-537.
  • Ainsworth, Hayase & Srinivasa (2023), Git Re-Basin, LIT-661, read in NOTE-532.
  • Ferbach, Goujaud, Gidel & Dieuleveut (2024), LIT-680, read in NOTE-543.
  • Theus, Cabodi, Anagnostidis, Orvieto, Singh & Boeva (2025), LIT-664, read in NOTE-540.
  • Bo Zhao, Dehmamy, Walters & Yu (2025), LIT-655, read in NOTE-528.

What alignment removed in each network these sources measured is THEORY-112, which this entry extends.

The claim

Entezari et al. conjecture that for wide enough networks most SGD solutions can each be permuted so that the straight line between any two has about zero barrier: one basin, seen through different labellings of the hidden units (Conjecture 1; NOTE-537, C1). Git Re-Basin holds that wide enough networks trained independently can be permuted into zero-barrier connectivity, and that permutation alone does not connect thin ones (NOTE-532, C1–C2). Ferbach et al. prove the conjecture for wide networks with independent neurons, with a width requirement that is a bound on width. Bo Zhao et al. place permutations inside the architecture’s full symmetry group, and Theus et al. carry the group to transformers: feed- forward units take permutations, attention heads a semi-permutation, and the residual stream one orthogonal map, without which a barrier remains.

So the claim is general: what separates independently trained solutions on a straight line is mostly the architecture’s symmetry, the full group and not only permutations, and less of it is left the wider the network. The measurements reach zero only for MNIST MLPs, 32×-wide ResNet-20s and small transformers whose alignment was learned against the midpoint loss.

What would settle it

The promote_when, which is QUESTION-317: standard-width models on ImageNet or a language corpus brought to near-zero held-out barrier by a data-free alignment over the full group, or the residual barrier shown to be a difference in what the endpoints compute. Another permutation-finding algorithm on vision models at moderate scale cannot settle it.

Backlinks