nucleation

LIT-671: Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

3 min read

Properties6
StatusActive — read in full 2026-10-03 (NOTE-544); the first explanation of mode connectivity that applies to realistic multilayer nets. If two solutions can each lose half their units per layer at a cost of at most ε in loss (dropout stability), they are joined by a piecewise-linear path whose loss never exceeds the worse endpoint by more than ε; a stronger noise-stability condition gives 10 segments. A counterexample shows that connectivity fails for some global minima of arbitrarily overparametrized two-layer nets, so the property belongs to the solutions training finds, not to the landscape. The conditions are checked on small MNIST convnets and one VGG-11 trained with channel-wise dropout.
Tagsloss-landscapes, learning-theory, anthology-candidate
ExtendsLIT-673 — Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, LIT-653 — Essentially No Barriers in Neural Network Energy Landscape
Extended byLIT-666 — Mode Connectivity and Data Heterogeneity of Federated Learning
Cited as “paper” byNOTE-544 — Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
Cited as “source” byTHEORY-109 — The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum, THEORY-736 — Two ε-dropout-stable solutions of a multilayer ReLU net, under a loss convex in the output, are joined by a piecewise-linear path whose loss never exceeds the worse endpoint's by more than ε, and joined pairwise paths compose with no higher barrier than their worst

Graph View

Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora and Rong Ge (2019), Advances in Neural Information Processing Systems 32 (NeurIPS 2019) — ARXIV-1906.06247

Key takeaways

  • Dropout stability implies connectivity (Theorem 1). A solution is ε-dropout stable if, for each layer onward, some half of the units can be kept and rescaled and the rest zeroed with loss rising by at most ε. Two such solutions are joined by a path whose loss never exceeds the worse endpoint by more than ε. The path walks each network to its half-zeroed version (Lemma 1; the number of segments grows with depth), uses the zeroed half as scratch space to permute units freely at no loss (Lemma 4, five segments), and then crosses between the two half-networks (Lemma 2, eight segments).
  • Noise stability gives simpler paths (Theorems 2–3). Under a noise stability condition adapted from Arora et al.’s compression bounds (layer cushion, interlayer cushion, activation contraction, interlayer smoothness), a network can be interpolated straight to a randomly dropped-out version, and two ε-noise-stable networks are joined by 10 segments with barrier Õ(ε). Routing through a narrow network that already achieves low loss gives 13 segments and a barrier that shrinks with the dropout fraction.
  • Not all minima are connected (Theorem 4). For any student width h and any convex loss, a dataset labelled by a two-unit teacher exists on which the student’s global minimizers are not connected. Overparametrization alone does not give mode connectivity.
  • The conditions roughly hold in practice (§6). Three-layer, 32-filter convnets on MNIST are fairly dropout stable without any dropout in training, and a VGG-11 trained with channel-wise dropout on CIFAR-10 is noise stable; the constructed path between two such VGG-11s keeps low loss, while the straight line reaches loss 2.34 and 10% accuracy.

Standing in the record

Filed with the batch ADR-027 opened loss-landscapes for. It extends Garipov et al. (LIT-673) and Draxler et al. (LIT-653) from observation to explanation: it sets out to prove the connectivity they found, for multilayer nets and realistic widths, from properties trained networks already have. Its explanation is a formal version of Draxler et al.’s “resilience” and “redundancy” arguments. A network that can spare half its units can move the other half anywhere, including into any permutation, without a barrier.

Its counterexample is the first statement in the batch that connectivity is a property of found solutions. Git Re-Basin (LIT-661) reaches the same conclusion for linear connectivity modulo permutation with a counterexample of its own. Ferbach et al. (LIT-680) take up the link between dropout stability and linear connectivity, showing that networks with independent neuron weights become dropout stable at the same widths at which they become linearly connected.

It carries learning-theory because its conditions are the noise-stability quantities of compression-based generalization bounds: the property that explains generalization is shown to explain connectivity too.

Backlinks