Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov and Andrew Gordon Wilson (2018), Advances in Neural Information Processing Systems 31 (NeurIPS 2018) — ARXIV-1802.10026
Key takeaways
- Optima are connected by simple curves. Given two independently trained networks ŵ₁ and ŵ₂, the paper trains the bend θ of a one-bend polygonal chain or a quadratic Bezier curve φ_θ(t) between them by minimizing E_{t∼U(0,1)} L(φ_θ(t)) (Eq. 2), a tractable stand-in for the loss averaged uniformly along the curve (Eq. 1). Along the found curves train loss and test error stay close to the endpoints, while the straight segment does not: for VGG-16 on CIFAR-10 test error reaches 90% at its midpoint against at most 7.01% on the Bezier curve, and for ResNet-164 on CIFAR-100 98.83% against 26.1% (Table 2). It holds for fully connected, convolutional, residual and recurrent networks (Tables 2–3).
- The curves are not reparametrization artefacts. An ensemble of the endpoint with the point at t ≥ 0.4 is already as good as an ensemble of the two independent endpoints (Fig. 2, right), so the curve passes through functionally different networks. The supplement (A.8) builds a trivial path by scaling a ReLU network toward the origin, which keeps accuracy but raises training loss; the learned curves do not work that way. They are also not unique: two chains with the same endpoints had turning points 29.6 apart when the endpoints were 50 apart.
- More parameters, easier connection. Widening a small CNN (A.7) shrinks both the gap between the curve’s worst loss and the endpoints’ and the ratio of curve length to segment length.
- Fast Geometric Ensembling (FGE). Starting from about 80% of a normal training run, a cyclical learning rate with 2–4-epoch cycles collects one checkpoint per cycle. At equal budget FGE beats Snapshot Ensembles in every setting of Table 1, and beats independent ensembles on CIFAR-100 (ResNet-164 at budget 1B: 20.2% error against 21.5% for one network and 20.9% for Snapshot Ensembles). On ImageNet, 5 epochs of FGE from a pretrained ResNet-50 lower top-1 error from 23.87% to 23.31%. FGE members disagree with each other less than independent networks do (14.57% of test points against 19.97%).
Standing in the record
Filed at the owner’s request with the batch of mode-connectivity and
model-merging readings that ADR-027 opened the loss-landscapes topic for.
It is, with Draxler et al. (LIT-653), the starting point of the line:
“mode connectivity” is this paper’s name for the phenomenon, and every later
work in the batch measures against it.
The two papers appeared within days of each other and cite each other as independent and simultaneous. This paper’s §2 says Draxler et al. “simultaneously and independently discovered the existence of curves connecting local optima”, by a method from quantum chemistry. Draxler et al.’s §2 says that after their ICML submission this paper “independently reported” the same observation, and their §6 proposes this paper’s ensembling as a use for their paths. No relation is declared between them: neither builds on the other.
Benton et al. (LIT-656) generalize this paper’s curve-finding procedure from a single bend to simplexes and simplicial complexes of connecting points. Frankle et al. (LIT-654) take its definition of mode connectivity and note that its curves, unlike the paths they study, are not linear. Kuditipudi et al. (LIT-671) set out to explain its finding theoretically.
The supplement’s last observation (A.10), that points on the segments between
consecutive FGE checkpoints have lower train loss and test error than the
checkpoints themselves, is deferred to Izmailov et al.’s stochastic weight
averaging. The anthology holds that paper (ANTH-LIT-673) and reads it for
practice; it is the practical descendant of this paper’s ensembling half.
The anthology does not hold this paper, which carries an ensembling
procedure that an anthology topic could hold, hence the
anthology-candidate flag (ADR-005).