nucleation

LIT-653: Essentially No Barriers in Neural Network Energy Landscape

3 min read

Properties6
StatusActive — read in full 2026-10-03 (NOTE-531); the other founding demonstration of mode connectivity, reached independently of Garipov et al. by borrowing the nudged elastic band from chemical physics. Over ten minima per architecture, the highest training loss on the best paths found is almost the minima's own for deep ResNets and DenseNets on CIFAR-10, with a small residual gap on CIFAR-100, and barriers fall as networks get wider and especially deeper. Its two explanations, resilience and redundancy, are qualitative; the redundancy toy (swapping two hidden units needs a third) is the first statement in this batch of why a permutation costs a barrier without spare capacity.
Tagsloss-landscapes, anthology-candidate
Extended byLIT-654 — Linear Mode Connectivity and the Lottery Ticket Hypothesis, LIT-671 — Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
Compared againstLIT-660 — Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
Cited as “paper” byNOTE-531 — Essentially No Barriers in Neural Network Energy Landscape
Cited as “source” byTHEORY-109 — The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum, THEORY-736 — Two ε-dropout-stable solutions of a multilayer ReLU net, under a loss convex in the output, are joined by a piecewise-linear path whose loss never exceeds the worse endpoint's by more than ε, and joined pairwise paths compose with no higher barrier than their worst

Graph View

Felix Draxler, Kambis Veschgini, Manfred Salmhofer and Fred A. Hamprecht (2018), Proceedings of the 35th International Conference on Machine Learning, PMLR 80 — ARXIV-1803.00885

Key takeaways

  • The barriers between minima are essentially zero for deep networks. The paper seeks the minimum energy path (MEP) between two minima, the path whose highest loss is lowest, and approximates it with AutoNEB (§3.1). For ResNet-20 to -56 and DenseNets on CIFAR-10, the training loss at the highest point of the best paths is almost that of the minima. On CIFAR-100 a small gap remains, which is small against the loss of an untrained network. Test error rises by at most 0.5% on CIFAR-10 and 2.2% on CIFAR-100 for all deep architectures (§4.3).
  • Depth and width remove barriers; harder data raises them. Across CNNs of varying width and depth, barriers fall as networks get wider and especially deeper (Fig. 5).
  • Paths are cheap to bound jointly. Concatenating paths gives an ultrametric inequality, L*_AC ≤ max{L_AB, L_BC}, so the best paths among ten minima form a minimum spanning tree, and bad local MEPs can be routed around (§3.2, Algorithm 3).
  • The barrier is crossed late in training. The training loss falls below the mean saddle energy only after the first learning-rate decay, and for most deep networks after the second (Fig. 6).
  • Two explanations, both qualitative (§5): resilience, many parameters can compensate for a change in one; and redundancy, illustrated by XOR. A two-unit network cannot swap its hidden units Alice and Bob without misclassifying a point on the way, but with a third unit, Charlie, standing in, the swap is free.

Standing in the record

Filed with the batch ADR-027 opened loss-landscapes for. It is the second of the two founding papers, beside Garipov et al. (LIT-673). The two cite each other as simultaneous and independent: this paper’s §2 says that after its ICML submission Garipov et al. “independently reported” the same observation, and its conclusion proposes using the paths as an ensemble “(Garipov et al., 2018)”. Garipov et al.’s §2 returns the credit. No relation is declared, because neither builds on the other.

The redundancy argument deserves keeping. It is the earliest statement in this batch that exchanging two hidden units, a permutation, costs a barrier unless the network has a unit to spare. Kuditipudi et al. (LIT-671) make that argument a proof: a network that can drop half its units can permute the rest along a zero-loss path (their Lemma 4). Entezari et al. (LIT-652) turn it around: the barriers seen on straight lines are what permutations cost, and removing the permutation removes the barrier.

Frankle et al. (LIT-654) borrow this paper’s Table B.1 to calibrate their 2% threshold for “no barrier”.

The method is from chemical physics (nudged elastic band, Jónsson et al. 1998), and the paper treats loss as energy. That is a borrowed tool, not a claim about physics, so no natural-sciences tag.

Backlinks