nucleation

CLAIM-524: Whether functionally equivalent realizations are connected depends on which paths are admissible: for f = u₁v₁ + u₂v₂ with nonnegative weights, (1, 1, 0, 0) and (0, 0, 1, 1) are joined by a curve along which f stays 1 though the straight line between them falls to ½, and under any resource budget below √2 times the endpoints' cost no path with f = 1 joins them

2 min read

Properties7
StatusActive
Rolethesis
Warrantdemonstrated
Tagsloss-landscapes, mathematics
Linelearned-structure, formal-apparatus
Bears onQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?
GroundsTHEORY-114 — In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training, THEORY-741 — A two-layer ReLU student of width at least kM, fitting an orthonormal teacher of M neurons on the sphere, has for any k global minima a global minimum joined to each by a two-piece linear path of minima; and in a deep linear network of depth L and width above 1 + r(L − 1), any r global minima in general position share a global minimum linearly connected to each

Graph View

The construction

Take a two-path network computing f = u₁v₁ + u₂v₂ with uᵢ, vᵢ ≥ 0, and the two realizations a = (1, 1, 0, 0) and b = (0, 0, 1, 1), both with f = 1.

  • The straight line has a barrier. At (1 − t)a + tb, f = (1 − t)² + t², which is ½ at t = ½.
  • A curve has none. (cos s, cos s, sin s, sin s) for s ∈ [0, π/2] keeps f = cos²s + sin²s = 1.
  • A budget removes every zero-loss path. Let a realization’s resource be R = ½ Σ (uᵢ + vᵢ), which is 1 at both endpoints. On f = 1 write p = u₂v₂, so u₁v₁ = 1 − p. By the AM–GM inequality, R ≥ √(u₁v₁) + √(u₂v₂) = √(1 − p) + √p. Any path from a to b moves p from 0 to 1 and so passes p = ½, where R ≥ √2. Under any budget below √2, the set of budget-feasible realizations with f = 1 is disconnected between a and b.

So the zero-loss set is connected while the zero-loss set under a resource bound is not. A17: “A low-path loss is not necessarily a viable transition path.”

Against the record’s readings

The owner’s guess was right: the record holds the mode-connectivity literature, and it bears on “just high-dimensional geometry?” both ways. Continuous symmetries stretch minima into connected manifolds (THEORY-210), and enough width gives piecewise-linear paths of minima (THEORY-741), which is the geometric explanation. But at practical widths linear connectivity emerges in training (THEORY-114), and connectivity at initialization needs widths far past practice (THEORY-1172), which cuts against geometry alone. QUESTION-069 asks whether connectedness belongs to the solutions or to the path family searched. This construction answers one part: for a fixed function and fixed endpoints, the answer changes with the path family admitted, and a resource bound is one way to restrict it.

What it does not say

  • That trained networks are disconnected under realistic budgets.
  • That parameter norm is the right resource. It is the simplest one to state; the point holds for any resource that grows when both paths are partly active.

Backlinks