nucleation

LIT-660: Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models

3 min read

Properties5
StatusProposed — read in full 2026-10-03 (NOTE-534); watching, as the first claim of mode connectivity between independently trained generative and contrastive models: one pair of DDPMs on Flowers102 and one pair of NanoCLIP models on Flickr30k, joined by a chain of checkpoints along which the training loss stays below a threshold, where AutoNEB's paths rise steeply on NanoCLIP. The path is built by moving layer groups towards the far end, rescaling each layer's variance, and retraining with AdamW until the loss is under threshold, so every anchor is a retrained model. What would settle it: more than one pair per model, a barrier measure along the whole path rather than at the anchors, and a reason to think a path made of retrained anchors says more about the landscape than that retraining works. It is a preprint of 31 August 2026.
Tagsloss-landscapes, anthology-candidate
Compared againstLIT-653 — Essentially No Barriers in Neural Network Energy Landscape
Cited as “paper” byNOTE-534 — Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
Cited as “source” byTHEORY-557 — Low-loss paths between independently trained solutions are found outside supervised image classifiers: graph neural networks trained from different seeds on one graph are joined by quadratic Bezier curves where the straight line has a barrier, and one pair each of diffusion and contrastive models by chains of low-loss checkpoints

Graph View

Chengzheyi Yao, Yongzhao Zhang and Yongding Tian (2026), arXiv preprint — ARXIV-2608.30366

Key takeaways

  • Low-loss chains exist between independently trained DDPMs and between independently trained NanoCLIP models. Two modes per architecture, from different seeds, are joined by a sequence of checkpoints whose training loss stays below 0.04 (DDPM denoising loss) and 0.09 (NanoCLIP contrastive loss), while layer-wise distance to the far mode falls by at least an order of magnitude (§5, Fig. 3). Dense linear interpolation between consecutive NanoCLIP checkpoints raises the maximum loss by under 5.3% (App. B).
  • How the chain is built matters. Moving all layers at once, or in forward-pass order, or refining with SGD instead of Adam(W), gives higher average loss along the path (Fig. 5). AutoNEB, run with 13 pivots, reaches a maximum NanoCLIP training loss of 6.69 and training accuracy as low as 0.05 (Fig. 4).
  • Low loss is not good generation, and closeness is not flatness. Along the DDPM chain, intermediate checkpoints have higher FID than the endpoints at about the same training loss (Fig. 6). Near the end mode, a checkpoint with layer-wise cosine similarity near one still shows a small bump on the straight line to it: maximum loss 0.038, against 0.034 on the path (App. E).

Standing in the record

Filed with the owner’s batch on mode connectivity and model merging, under loss-landscapes (ADR-027). Proposed, not Active: the evidence is one pair of modes per architecture, and the path is a sequence of retrained anchors. Each anchor is trained back under the loss threshold (up to 500 AdamW rounds for DDPM). So the claim shown is that one can walk from one mode to the other in small steps and retrain at each, which is weaker than the existence of a fixed low-loss curve found by optimising a path. App. B checks the straight segments between anchors for NanoCLIP only.

It is measured against Draxler et al. (LIT-653): AutoNEB, adapted to DDPM and NanoCLIP with a 14-cycle schedule (App. D, Table 3), is its only run baseline, and its Fig. 4 compares the two along the path. It leaves out Garipov et al.’s curve fitting (LIT-673), saying the released implementation “contains an error”. It builds on LLPF (Tian et al. 2026), which the record does not hold; the relation that would be declared is extends.

Its motivation cites singular learning theory (Watanabe 2009, held here as LIT-354, unread) for why non-isolated minima are to be expected; it does not use the theory. Within the batch, it is one of three works that test whether mode connectivity holds outside image classifiers, with the GNN study (LIT-676) and the ELBO study (LIT-677).

Backlinks