Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan and Wei Hu (2023), 37th Conference on Neural Information Processing Systems (NeurIPS 2023) — ARXIV-2307.08286
Key takeaways
- LLFC co-occurs with LMC. For pairs that are linearly mode connected, 1 − cos between f⁽ℓ⁾(αθ_A + (1 − α)θ_B) and αf⁽ℓ⁾(θ_A) + (1 − α)f⁽ℓ⁾(θ_B) is near zero in nearly every layer, for α ∈ {0.25, 0.5, 0.75}, across an MLP on MNIST and CIFAR-10, VGG-16 and ResNet-20 on CIFAR-10, and ResNet-50 on Tiny-ImageNet (Figs. 2–3). It is far from zero for independently trained pairs (Fig. 4).
- LLFC is strictly stronger than LMC. Applied to the output layer, it gives an error at most 2ε on the whole linear path if both endpoints have error at most ε (Lemma 1).
- Two conditions imply exact LLFC. One is weak additivity: ReLU(αH̃_A + (1 − α)H̃_B) = αReLU(H̃_A) + (1 − α)ReLU(H̃_B) on the two networks’ pre-activations. The other is commutativity: W_A H_A + W_B H_B = W_A H_B + W_B H_A in every layer. Together they give LLFC with scale c = 1 (Theorem 1). Both hold approximately, and far better for linearly connected pairs than for independent ones (Figs. 5–7).
- Git Re-Basin, re-read. Commutativity can be rewritten as (W_A − PW_BPᵀ)(H_A − PH_B) = 0. Weight matching minimises the first factor and activation matching the second (§5.2). Enforcing the product directly is a quadratic assignment problem (Appendix A.3).
- Why matching fails for narrow or early networks. The appendix argues that low-rank weights and features make commutativity easy to satisfy, and that narrow or barely trained networks lack that low rank (Appendix B.5, Figs. 20–22). This is an argument, not a theorem.
Standing in the record
Filed on 2026-10-03 with the owner’s mode-connectivity batch (ADR-027). It is the batch’s reading of linear mode connectivity from inside the network. When the path between two minima is flat, the network along it behaves like a linear blend of the two networks layer by layer, so averaging weights is roughly averaging features.
It extends two foundations held here. From Frankle et al. (LIT-654) it takes the spawning method, training a network for a few epochs and then splitting it into two runs, with their training recipes, as the first of its two ways to get linearly connected pairs. From Git Re-Basin (LIT-661) it takes the permutation method and the 32×-wide LayerNorm ResNet-20 checkpoints, reused directly. It then gives Git Re-Basin’s weight and activation matching objectives a reason to work: each makes one factor of its commutativity residual small.
Its related work credits Lubana et al. (LIT-669) with showing that mechanistically dissimilar models cannot be linearly connected. LLFC is the positive counterpart: linearly connected models carry the same features up to a linear blend. The anthology’s model soups (ANTH-LIT-675) and SWA (ANTH-LIT-673) average weights within a basin; this paper is a candidate explanation of why that averages features too, but it measures interpolation between two endpoints, not those methods.
No instruction for practice. It is flagged for the anthology because the anthology holds the permutation and averaging lines it explains.