nucleation

THEORY-557: Low-loss paths between independently trained solutions are found outside supervised image classifiers: graph neural networks trained from different seeds on one graph are joined by quadratic Bezier curves where the straight line has a barrier, and one pair each of diffusion and contrastive models by chains of low-loss checkpoints

3 min read

Properties5
StatusProposed
Tagsloss-landscapes, anthology-candidate
SourceLIT-676 — Unveiling Mode Connectivity in Graph Neural Networks, LIT-660 — Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
ExtendsTHEORY-109 — The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum
Bears onQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?

Graph View

Source

  • Li, Chen, Han, Zeng, Liu & Tang (2025), LIT-676, read in full in NOTE-529: §3.1, Figs. 1–5 and the appendix figures.
  • Yao, Zhang & Tian (2026), LIT-660, read in full in NOTE-534: §5, Figs. 3–6, Appendices B and E.

What was actually shown

Graph neural networks (LIT-676). GCNs for node classification are trained on one graph with different seeds or data orders, three seeds averaged. On the straight line between two solutions there is often a loss barrier: small on Cora and Citeseer, large on Coauthor-CS and Squirrel. A quadratic Bezier curve fitted as in Garipov et al. removes most of it (Observation 1, Figs. 1–2). GCN, GraphSAGE and GAT give similar barriers on a graph, and an MLP on the same features differs markedly (Fig. 4).

Diffusion and contrastive models (LIT-660). One pair of DDPMs on Flowers102 and one pair of NanoCLIP models on Flickr30k, from different seeds, with no shared training and no permutation alignment. A chain is built by moving groups of layers toward the far mode, correcting each layer’s variance, and retraining with AdamW until the loss is under a threshold. Training loss stays below 0.04 (DDPM) and 0.09 (NanoCLIP) along it (Fig. 3). Dense interpolation between consecutive NanoCLIP checkpoints raises the maximum loss by under 5.3% (App. B). AutoNEB, adapted to these models, reaches a maximum NanoCLIP training loss of 6.69 and training accuracy as low as 0.05 (Fig. 4).

What could have come out otherwise. The curve or the chain could have failed to stay low. On these pairs it did not.

What this does not say

  • That the evidence matches the classifier evidence. LIT-676 gives no barrier numbers for the real graphs, and its proofs are missing. LIT-660 has one pair per architecture, and its path is a walk of retrained anchors, each trained back under the threshold (up to 500 AdamW rounds for DDPM), checked between anchors for NanoCLIP only.
  • That low loss along the path means a good model. Intermediate DDPM checkpoints generate worse images (higher FID) at endpoint-level training loss (Fig. 6).
  • That the straight line works. In the graph networks it often does not.
  • That latent-variable models share the property. Zhang & Blei (LIT-677, NOTE-522) claim that maxima of LDA’s ELBO from two SVI runs are joined by essentially flat paths found with the string method, and that statistical degeneracy explains it. The workshop paper reports no path, no ELBO value and none of the synthetic experiments it describes, so it is not a source here.
  • Why the paths exist. LIT-660 cites singular learning theory for expecting connected minima, and LIT-677 degeneracy; neither tests it.

Connections

  • Extends THEORY-109, whose promote_when asks for connectivity on tasks beyond CIFAR classifiers. These are such tasks, on thin evidence: the widening that account asks for, not yet the evidence it asks for.
  • THEORY-675 takes LIT-676’s other result: on one graph the linear barrier depends on the graph’s density, separability and homophily.
  • THEORY-210 gives one reason to expect connected minima: continuous parameter symmetries stretch minima into connected manifolds. LIT-677’s “statistical degeneracy” is that idea for a topic model.
  • QUESTION-069 asks whether connectedness belongs to the solutions or to the path family searched. LIT-660’s AutoNEB comparison bears on the second: on NanoCLIP AutoNEB’s path peaks at a training loss of 6.69 where the staged chain stays below 0.09, so the search matters at least to what is found.

Backlinks