nucleation

LIT-662: Geodesic Mode Connectivity

2 min read

Properties5
StatusActive — read in full 2026-10-03 (NOTE-521); worth reading as the short proposal that mode-connecting paths are geodesics of the Fisher–Rao metric on the space of the networks' output distributions. It approximates a geodesic between two trained ResNet-20s (4× width, LayerNorm), after Git Re-Basin weight matching, by optimising 25 models on a chain to minimise the summed Jensen–Shannon divergence between neighbours, using unlabelled training images only, and the resulting curve has low train and test loss where the straight line does not. One architecture, one pair, one figure; the hypothesis that all geodesics between SGD solutions are mode-connecting is stated, not tested.
Tagsloss-landscapes, information-geometry, anthology-candidate
ExtendsLIT-661 — Git Re-Basin: Merging Models modulo Permutation Symmetries
Cited as “paper” byNOTE-521 — Geodesic Mode Connectivity
Cited as “source” byTHEORY-578 — Between two trained networks that compute nearly the same function, a chain of models minimising the summed Jensen–Shannon divergence between neighbours, a discretised Fisher–Rao geodesic found without labels, can be a low-loss path where the straight line in weights has a barrier

Graph View

Charlie Tan, Theodore Long, Sarah Zhao and Rudolf Laine (2023), Tiny Paper at ICLR 2023 — ARXIV-2308.12666

Key takeaways

  • Mode connectivity, recast in information geometry. A classifier with parameters θ, paired with the empirical input distribution, is a joint distribution p(x, ŷ; θ) = p(x)p(ŷ|x; θ). The Fisher–Rao metric (Eq. 3) gives that space lengths and curvature. The length of a path equals √8 times the integral of √dJSD (Eq. 1, citing Crooks 2007).
  • A discrete geodesic objective. Fix the endpoints, put N = 25 models on the straight line between them, and minimise Σᵢ JSD(pᵢ‖pᵢ₊₁) (Eq. 2). The square root is dropped, which minimises the energy rather than the length and gives the constant-speed geodesic. Only training images are used, not labels.
  • Where the straight line fails, the geodesic holds. For a ResNet-20 at 4× width on CIFAR-10, the linear path after weight matching still has a large loss barrier, while the optimised chain keeps train and test loss low throughout (Fig. 2).
  • The hypothesis is broad and untested. The paper states that all geodesics between SGD solutions are mode-connecting paths, but shows one pair of models.

Standing in the record

Filed on 2026-10-03 with the owner’s mode-connectivity batch (ADR-027). It holds one of the batch’s two meanings of “geodesic”. Here the geodesic is in the space of output distributions under the Fisher–Rao metric. In Lin, Li and Wu (LIT-682) it is the shortest path inside the minimum manifold, measured in Euclidean parameter space. The two are different objects, and only this one belongs to information-geometry.

It extends Git Re-Basin (LIT-661) directly. Its two networks are permuted by Git Re-Basin’s weight matching, its LayerNorm ResNet is the one Git Re-Basin built to make permutation work, and its baseline is Git Re-Basin’s straight line, which fails at this width. The geodesic is initialised on that line and bent away from it. The phenomenon it reframes is Garipov et al.’s (LIT-673) curved low-loss paths, but the paper leaves a comparison with Garipov’s curve-fitting to future work.

No instruction for practice. It is flagged for the anthology because the anthology holds the permutation line it builds on (Git Re-Basin is ANTH-LIT-333 there).

Backlinks