nucleation

QUESTION-069: Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?

2 min read

Properties7
StatusOpen
Tagsloss-landscapes, anthology-candidate, learning-theory
Linelearned-structure
Refined byQUESTION-164 — Does the law of balance hold in a deep network trained on real data during its limiting dynamics: for parameter groups related by a rescaling symmetry, does the difference of squared norms move toward the value where the groups' gradient-noise traces are equal, at a rate that scales with learning rate over batch size?
BearingTHEORY-828 — Fine-tuning from one pretrained model typically lands near the edge of a task-specific low-loss region in weight space that contains the convex hull of the fine-tuned models, and averaging their weights, as model soups and stochastic weight averaging do, works by moving inward, THEORY-675 — Without a shared stable start, the linear barrier between solutions of one objective depends on the structure of the training data: between GCNs it falls with graph density and feature separability and is highest at middling homophily, and between federated-averaging models it grows with client heterogeneity and falls with width, THEORY-578 — Between two trained networks that compute nearly the same function, a chain of models minimising the summed Jensen–Shannon divergence between neighbours, a discretised Fisher–Rao geodesic found without labels, can be a low-loss path where the straight line in weights has a barrier, THEORY-557 — Low-loss paths between independently trained solutions are found outside supervised image classifiers: graph neural networks trained from different seeds on one graph are joined by quadratic Bezier curves where the straight line has a barrier, and one pair each of diffusion and contrastive models by chains of low-loss checkpoints, THEORY-210 — Continuous parameter symmetries stretch a network's minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss, THEORY-209 — A converged network trained with SGD is not at a point and does not sample its training loss: under a quadratic approximation it samples a Gibbs distribution on a modified loss, isotropic in the Hessian's column space, with probability currents that keep it oscillating, so its speed and spread are set by the learning rate, momentum and batch size, THEORY-114 — In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training, THEORY-110 — A set of trained solutions has a centre that is linearly connected, after alignment, to the rest of them: the solution set is a star domain even where it is not convex, THEORY-109 — The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum, THEORY-108 — Across load and temperature, the connectivity and output similarity of independently trained solutions predict test accuracy where endpoint curvature does not: a flat but poorly connected landscape generalizes worse than a connected one of the same curvature, THEORY-107 — Merging models fine-tuned from one pretrained start loses accuracy through interference in parameter coordinates: redundant small updates dilute the influential ones, and influential updates of different models conflict in sign, THEORY-509 — Under genericity conditions, two dense or top-k (k ≥ 2) mixture-of-experts layers compute the same function only if they differ by a permutation of the experts with their gates and a common translation of the gate weights, and the translation leaves every interpolation barrier unchanged, so expert order is the only gate symmetry alignment must undo, THEORY-211 — In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise, THEORY-105 — For RoBERTa-base fine-tuned on English classification data, with loss measured by a linear probe refitted on the frozen encoder, points sampled from the convex hull of models fine-tuned on one dataset, one task family or twelve GLUE and SuperGLUE datasets lost no more than any model fine-tuned elsewhere and beat the group's own members in 88% to 96.7% of comparisons, and the centroid of the general-level models was a better starting point for BitFit than the pretrained model
Cited as “bears on” byCLAIM-524 — Whether functionally equivalent realizations are connected depends on which paths are admissible: for f = u₁v₁ + u₂v₂ with nonnegative weights, (1, 1, 0, 0) and (0, 0, 1, 1) are joined by a curve along which f stays 1 though the straight line between them falls to ½, and under any resource budget below √2 times the endpoints' cost no path with f = 1 joins them
Cited as “refines” byQUESTION-312 — Does linear mode connectivity emerge during training rather than at initialization outside image classification and beyond MLPs: in a language model or transformer, for copies spawned from a shared state and for independent runs aligned under the architecture's symmetry group, and for standard-width convnets or ResNets after alignment?, QUESTION-317 — Does aligning independently trained networks over the architecture's full symmetry group remove most of the linear barrier at standard width, on ImageNet or a language corpus, with an alignment computed without data; is the barrier that remains a difference in what the endpoints compute; and does it rise with depth?

Graph View

The question

Several results say that the solutions training reaches are joined at low loss. Each was measured with a particular instrument (a curve family, a straight line, a centre, an alignment), on runs with a particular history (shared start, shared stem, none), and in a particular regime. The question is which of these the connectedness belongs to. Equivalently: what does a residual barrier between two trained networks tell us about the networks themselves?

Why the record needs it

CLAIM-187 reads a barrier that survives alignment as a different mechanism. It would carry that protocol to communicative kinds. Its defeater asks whether systems that share a mechanism can stay separated after alignment. If the barrier depends on the instrument or the history, then the protocol measures those and not the mechanism. CLAIM-142 makes the same point about probes.

What the record holds

  • THEORY-109: curves join solutions where the straight segment fails, and this is “a property of the solutions training finds, not of every minimum”. Limit: “one set” is extrapolated from pairs.
  • THEORY-114: linear connectivity “emerges during training, not at initialization”. It comes early for copies that share a stem and gradually for aligned strangers.
  • THEORY-110: a centre is linearly connected to held-out solutions when pairs are not, so the relation is “a star domain even where it is not convex”. Limit: the barriers to the centre are not zero.
  • THEORY-105: from one start, the low-loss region “contains the convex hull” of same-task models, and it is graded by dataset and task family.
  • THEORY-107: sign conflicts are as frequent within one task as across tasks, yet cost far less within one. So disagreement in coordinates is not the residue.
  • THEORY-108: in phase III, flat solutions are “poorly connected” on one curve family without alignment. It is open whether this is a real barrier.
  • THEORY-209 (Kunin et al., filed 2026-10-13): a converged run trained with SGD is a nonequilibrium stationary distribution, not a point, whose spread and correlations are set by learning rate, momentum and batch size. It bears on the “regime” alternative. It measures no connectivity, and is silent on flat directions.

What would answer it

Phase III runs aligned by permutation, with the barrier measured on several path families. Alongside that, a pair of networks known to share a mechanism, tested for whether alignment always removes their barrier.

Backlinks