nucleation

LIT-661: Git Re-Basin: Merging Models modulo Permutation Symmetries

4 min read

Properties7
StatusActive — read in full 2026-10-03 (NOTE-532); the strongest evidence for one basin modulo permutation. A weight-matching algorithm that needs no data and runs in seconds aligns independently trained networks so that the straight line between them has no loss barrier, for MLPs on MNIST and for ResNet-20 on CIFAR-10 at large width, the first such result for ResNets. It also finds where the picture stops: thin models are not linearly connected after any permutation it finds, connectivity appears only after some training, ImageNet ResNet-50 keeps a barrier, and a constructed counterexample shows that the property belongs to SGD solutions, not to architectures. Second reading under ADR-013; the anthology holds it as ANTH-LIT-333, and that entry misdescribes the algorithm and overstates the results.
Tagsloss-landscapes, anthology-candidate
ExtendsLIT-652 — The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
Extended byLIT-659 — Do Deep Neural Network Solutions Form a Star Domain?, LIT-662 — Geodesic Mode Connectivity, LIT-664 — Generalized Linear Mode Connectivity for Transformers, LIT-670 — On Linear Mode Connectivity of Mixture-of-Experts Architectures, LIT-674 — Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Compared againstLIT-665 — ZipIt! Merging Models from Different Tasks without Training, LIT-680 — Proving Linear Mode Connectivity of Neural Networks via Optimal Transport
Cited as “paper” byNOTE-532 — Git Re-Basin: Merging Models modulo Permutation Symmetries
Cited as “source” byTHEORY-112 — Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42), THEORY-114 — In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training, THEORY-1172 — Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually, THEORY-1188 — Once the architecture's symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width

Graph View

Samuel K. Ainsworth, Jonathan Hayase and Siddhartha Srinivasa (2023), International Conference on Learning Representations (ICLR 2023) — ARXIV-2209.04836

Key takeaways

  • Alignment removes barriers, where the network is wide enough. Weight matching chooses permutation matrices P_ℓ to maximize Σ_ℓ ⟨W_ℓ^A, P_ℓ W_ℓ^B P_{ℓ−1}ᵀ⟩_F. That problem is NP-hard with no polynomial-time constant-factor approximation for L > 2 (Lemma 1). Solving for one P_ℓ at a time is a linear assignment problem, and cycling through the layers until nothing changes terminates (Algorithm 1, Lemma 2). It uses no data and takes seconds to minutes: 3 s for an MLP, 33 s for ResNet-50, 194 s for a 32×-wide ResNet-20 (App. A.5). After matching, MNIST MLPs and CIFAR-10 ResNet-20s at large width show zero barrier, the first zero-barrier connection between independently trained ResNets (Figs. 2, 4).
  • Where it fails. At 1× width neither VGG-16 nor ResNet-20 becomes linearly connected (Fig. 4). ImageNet ResNet-50 keeps a barrier, reduced 67% from naive interpolation (§5.1). Connectivity is absent early in training and appears gradually (Fig. 3). The authors conclude that permutations are “a necessary piece, though not a complete picture”, and guess that general linear relationships between activations are at work.
  • A counterexample (§4, App. A.6). A two-dimensional classification task with two hidden layers of two ReLU units has two perfect solutions that no permutation connects linearly: one tests x₂ first and x₁ second, the other the reverse. LMC is a property of the solutions training reaches, not of architectures. The authors note this does not contradict Entezari et al.’s conjecture, whose restriction to SGD solutions is “load-bearing”.
  • Merging. ResNet-20s trained on disjoint, biased halves of CIFAR-100 merge into a model with better test loss than either and better calibration, though not better accuracy (§5.4). Merging five MNIST MLPs at once with Algorithm 3 lowers test loss by 43% (Table 3).

Standing in the record

Second reading under ADR-013. The anthology holds this paper as ANTH-LIT-333 (read 2026-09-15, its ANTH-NOTE-116), for practice: a toolkit for aligning networks before averaging them. This record reads it for the landscape question: how far it confirms the single-basin conjecture, and what its negative results (thin models, early training, the counterexample, the ImageNet barrier) say about the shape of the solution set.

The anthology entry misdescribes the paper, as this reading finds it. Per ADR-013 this is reported here and in NOTE-532, and not edited across the boundary:

  • ANTH-LIT-333 calls weight matching a “Hungarian algorithm” applied “layer by layer” and lists among its limits that “greedy layer-by-layer is not globally optimal”. The paper’s weight matching is coordinate descent over all layers, repeated until convergence (Algorithm 1). Appendix A.7 argues against greedy one-pass layer-by-layer matching, with a counterexample, and shows that a single pass does much worse.
  • It says multi-model merging “requires pairwise anchoring”. The paper gives a merge-many algorithm (Algorithm 3, App. A.10).
  • It reports near-zero barriers “across architectures (MLPs, CNNs, VGGs, ResNets)”, and merged models “within 1–2%” of the individual ones on CIFAR-10. The paper gets zero barriers for VGG-16 and ResNet-20 only at large width, none at 1×, and a remaining barrier on ImageNet. This reading found no 1–2% figure.

The paper extends Entezari et al. (LIT-652). It states their conjecture as its own Conjecture 1. Where their simulated annealing lowered barriers only for shallow nets, its weight matching removes them for wide ResNets, and its counterexample sharpens their restriction to SGD solutions. It reads Benton et al. (LIT-656) as showing that SGD solutions form a connected volume, and Entezari et al. as conjecturing that this volume is convex modulo permutation.

Its finding that LMC appears only after training sits uneasily with Entezari et al.’s Theorem 3.1, which proves it at random initialization. Ferbach et al. (LIT-680) reconcile the two: the width their bound needs at initialization is far beyond practical widths. Its picture of connectivity emerging during training parallels Frankle et al. (LIT-654), who saw the same for copies sharing an initialization.

The paper anticipates using merge-many “along the lines of ‘model soups’”; the anthology holds model soups as ANTH-LIT-675. The paper carries merging instructions, hence the anthology-candidate flag (ADR-005), although the anthology already holds it.

Backlinks