Rahim Entezari, Hanie Sedghi, Olga Saukh and Behnam Neyshabur (2022), International Conference on Learning Representations (ICLR 2022) — ARXIV-2110.06296
Key takeaways
- The conjecture (Conjecture 1, §3.2). For networks wide enough, there is a set S of solutions, containing SGD solutions with high probability, and a permutation for each member, such that the barrier on the straight line between any two permuted members is about zero. Most SGD solutions end in one basin once permutation invariance is taken into account.
- Barriers, measured (§2). With the barrier defined against the linear interpolation of endpoint losses (Eq. 1), barriers between networks from different initializations first rise then fall with width. They peak at the width that just fits the training data, which the authors liken to double descent. They grow with depth, and for VGG and ResNet families they saturate at a high value that width does not change (Figs. 2–4).
- A narrow proof (Theorem 3.1). For a one-hidden-layer ReLU network f(x) = vᵀσ(Ux) with h hidden units and weights drawn uniformly at initialization, there is with probability 1 − δ a permutation such that, for any input with ‖x‖₂ = √d, the network on the line differs from the line between the two networks’ outputs by Õ(h^{−1/(2d+4)}). Nothing is proved about trained networks.
- Evidence by model (§4). Let S′ be all permutations of one SGD solution; S′ satisfies the conjecture by construction. If independent solutions S behave like S′, the conjecture plausibly holds. Across width, depth, architecture and dataset, barriers in S and S′ are “strikingly similar” before and after a permutation search (Figs. 1, 5, 7; over 3,000 trained networks).
- The search mostly fails (§4.2). Simulated annealing barely improves barriers when aligning five networks at once. Restricted to two, it finds zero-barrier permutations for some MLPs on MNIST and lowers barriers for shallow MLPs and CNNs on easy data, but does nothing for VGG and ResNet. It fails on S′ exactly where it fails on S, which the authors count as more evidence.
Standing in the record
Second reading under ADR-013. The anthology holds this paper as
ANTH-LIT-251 (read 2026-09-15, its ANTH-NOTE-131), for a practical question: why
averaging the weights of independently trained networks degrades them, and
that alignment must come first. This record reads it for a different
question: what the conjecture actually claims about the geometry of solution
sets, how much of it is proved, and how the evidence is built. That question
belongs with the rest of the loss-landscapes line (ADR-027), where the
conjecture is tested, proved in special cases, and argued against.
The anthology entry misstates the paper, as this reading finds it. Per ADR-013 the disagreement is reported here and in NOTE-537, and the anthology entry is not edited:
- ANTH-LIT-251 and its reading give the theorem as “if no neuron in either network is dead, there exists a permutation … [with] no loss barrier”, with “no dead neurons” as its condition. The paper’s only theorem (Theorem 3.1, proved in Appendix D) has no such condition. It is about a one-hidden-layer network at uniform random initialization, and it bounds an output difference at a fixed input by Õ(h^{−1/(2d+4)}), not a loss barrier.
- ANTH-LIT-251 says that “with alignment, barriers vanish”, demonstrated on “CIFAR-10, ResNets, VGGs”. The paper’s search did not reduce barriers for VGG or ResNet (§4.1, Appendix E.2); zero barriers were found only for some MLPs on MNIST.
The paper extends Frankle et al. (LIT-654). It adopts their definition of linear mode connectivity and their framing of a basin as a region SGD is stable in, and changes the barrier’s baseline from the mean of the endpoint losses to their linear interpolation, so that a loss changing linearly along the path counts as no barrier (§2.1). Frankle et al. needed a shared initialization; this paper asks whether permutations can stand in for one.
Git Re-Basin (LIT-661) states this conjecture as its Conjecture 1 and gives the first zero-barrier connection between independently trained ResNets. Ferbach et al. (LIT-680) generalize Theorem 3.1 to deep networks with a better width bound. The conjecture’s convexity claim is also what the star-domain papers in this batch push against, Sonthalia et al., “Do Deep Neural Network Solutions Form a Star Domain?” (LIT-659), and Lin et al. on star-shaped and geodesic connectivity (LIT-682). They propose that the solution set is star-shaped around a centre rather than convex modulo permutation. Those readings were filed by other readers in this batch and declare their own relations; none is declared from this side.