nucleation

LIT-664: Generalized Linear Mode Connectivity for Transformers

3 min read

Properties5
StatusActive — read 2026-10-03 (NOTE-540); worth reading as the result that independently trained transformers become linearly mode connected once alignment uses their real symmetries: permutations of feed-forward units, semi-permutations of attention heads, and an orthogonal map on the residual stream after RMSNorm reparameterisation. Data-free weight matching cuts test-loss barriers from 1.69–4.34 to 0.34–1.56, and learned matching, which optimises the alignment against task loss at the midpoint, reaches 0.00 on three vision datasets and 0.02 and 0.42 on two language ones. Most of the gap is in the orthogonal map. Small ViTs and GPT-2s only.
Tagsloss-landscapes, anthology-candidate
ExtendsLIT-661 — Git Re-Basin: Merging Models modulo Permutation Symmetries
Cited as “paper” byNOTE-540 — Generalized Linear Mode Connectivity for Transformers
Cited as “source” byTHEORY-112 — Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42), THEORY-1188 — Once the architecture's symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width

Graph View

Alexander Theus, Alessandro Cabodi, Sotiris Anagnostidis, Antonio Orvieto, Sidak Pal Singh and Valentina Boeva (2025), 39th Conference on Neural Information Processing Systems (NeurIPS 2025) — ARXIV-2506.22712

Key takeaways

  • Four symmetry classes, nested. Permutations (elementwise activations such as GELU), semi-permutations (sparse stochastic mixing, for positively homogeneous activations, ReLU and attention heads), orthogonal maps (RMSNorm) and invertible maps (linear layers, the QK and OV circuits) each preserve the function of the component they fit (Table 1).
  • Where they sit in a transformer. Feed-forward hidden units take a permutation. Attention heads take a semi-permutation, because heads are summed and each head’s OV circuit splits linearly. Each head’s W_Q W_Kᵀ and W_V W_O are invariant under invertible maps, so the circuits themselves are matched. The residual stream takes one orthogonal map O, once LayerNorm is rewritten as RMSNorm with its centring and scale absorbed into neighbouring layers (§3.2).
  • Weight matching solves a layerwise assignment for feed-forward and head permutations and a Procrustes problem for O, iterated, with no data (§4.1). Learned matching starts there and trains the latent permutation and orthogonal matrices with the task loss at λθ_A + (1 − λ)π(θ_B), λ ∼ U(0.4, 0.6), using straight-through gradients for permutations (Algorithm 1).
  • Barriers fall to zero (Table 2, test loss). Vanilla averaging gives 1.69, 2.46 and 2.84 on CIFAR-10, CIFAR-100 and Tiny ImageNet, and 2.02 and 4.34 on Tiny Shakespeare and BookCorpus. Weight matching gives 0.36, 0.69, 0.47, 0.34 and 1.56. Learned matching gives 0.00, 0.00, 0.00, 0.02 and 0.42. Learned matching restricted to permutations gives 0.45, 0.53, 0.29, 0.63 and 1.60.
  • The gap is the residual rotation. Learned matching leaves weight matching’s permutations essentially unchanged and makes small corrections to O (Fig. 5). Iterating orthogonal weight matching keeps lowering the barrier; iterating permutation matching does not (Fig. 7).

Standing in the record

Filed on 2026-10-03 with the owner’s mode-connectivity batch (ADR-027). It is the batch’s transformer case, and its strongest evidence that what looks like a barrier between two transformers is mostly a choice of coordinates.

It extends Git Re-Basin (LIT-661). Its weight matching is Git Re-Basin’s sum-of-bilinear-assignments objective, adapted to carry an orthogonal map on the residual stream. Its learned matching is Git Re-Basin’s straight-through estimator, widened from permutations to orthogonal maps. Its permutation-only learned-matching row in Table 2 is, in the paper’s words, “akin to the STE-based approach in Git Rebasin”, and that row does not reach zero. The anthology holds Git Re-Basin as ANTH-LIT-333. The conjecture its results bear on is Entezari et al.’s (LIT-652), which its related work restates: once symmetries are resolved, SGD solutions lie in one basin.

Across the boundary it bears on two Proposed anthology documents. ANTH-THEORY-010 (the barrier is mostly permutation) names “the account demonstrated on a transformer” as a condition for promotion. This paper demonstrates it, but with the symmetry group widened beyond permutations, and with permutations alone falling short. So it supports the account’s spirit and corrects its letter. ANTH-SOTA-217 (align the hidden-unit permutation before averaging) would, for transformers, need the orthogonal residual alignment as well.

Like Zhou et al. (LIT-674), whom it cites for layerwise linearity, it reads mode connectivity as a fact about coordinates. Unlike Bo Zhao et al. (LIT-655), it uses the invertible symmetries of attention products to align, not to show that the straight line can fail.

It is flagged for the anthology: it carries a procedure for merging transformers.

Backlinks