nucleation

THEORY-578: Between two trained networks that compute nearly the same function, a chain of models minimising the summed Jensen–Shannon divergence between neighbours, a discretised Fisher–Rao geodesic found without labels, can be a low-loss path where the straight line in weights has a barrier

3 min read

Properties4
StatusProposed
Tagsloss-landscapes, information-geometry, anthology-candidate
SourceLIT-662 — Geodesic Mode Connectivity
Bears onQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?

Graph View

Source

Tan, Long, Zhao & Laine (2023), LIT-662 (Tiny Paper, ICLR 2023), read in full in NOTE-521. NOTE-521 held this account back because the evidence is one figure; it is filed Proposed for that reason.

What was actually shown

The geometry. A classifier with the empirical input distribution is a joint distribution p(x, ŷ; θ) = p(x)p(ŷ|x; θ). Under the Fisher–Rao metric the length of a path is √8 times the integral of √dJSD (Eq. 1). Dropping the square root and summing over a discrete chain gives an energy, Σᵢ JSD(pᵢ‖pᵢ₊₁) (Eq. 2), whose minimiser is the constant-speed geodesic.

The experiment. Two ResNet-20s at 4× width, with LayerNorm in place of BatchNorm, trained on CIFAR-10 from two seeds and permuted by Git Re-Basin’s weight matching. 25 models are placed on the straight line between them and optimised by SGD to minimise the summed JSD, using training images only, no labels. After optimisation, train and test cross-entropy are low at every model on the chain; on the straight line both rise sharply in the middle (Fig. 2). The optimisation lowers the summed √JSD while lengthening the chain in Euclidean weight space.

What could have come out otherwise. The distribution-space geodesic could have crossed a loss barrier. On this pair it did not.

What this does not say

  • That geodesics in general connect modes. “All geodesics between SGD solutions are mode-connecting paths” is stated in §1 and not tested beyond one pair.
  • That the geodesic is special. The endpoints already have low loss, and a chain whose neighbours agree in output stays close to their outputs. The objective may be finding a path along which the function barely changes, which is mode connectivity almost by construction (NOTE-521, Limitations). The promote_when’s control is for this.
  • That it is better than Garipov’s curves. The comparison is left to future work.
  • That it works without the permutation, or between narrow networks. “Narrow” here means 4× width, narrow only beside the 32× Git Re-Basin needed for linear connectivity.
  • That 25 models approximate the continuous geodesic. Not checked.

Connections

  • THEORY-109 holds that trained solutions are joined by simple curves found by fitting low loss. This finds such a curve by shortening the path in output space instead, with no labels.
  • THEORY-112 holds that once symmetries are factored out most of the linear barrier disappears, with a residual at small width. This paper starts where weight matching’s straight line still fails.
  • THEORY-111 says that along a low-loss linear path every layer’s features interpolate the endpoints’. Here the path is curved in weights and short in outputs.
  • QUESTION-069 asks whether connectedness belongs to the solutions or to the path family searched. This is a different path family that finds a path where the line does not.
  • Lin, Li & Wu (LIT-682)‘s “geodesic” is a different object: the shortest Euclidean path inside the minimum manifold.

Backlinks