Gregory W. Benton, Wesley J. Maddox, Sanae Lotfi and Andrew Gordon Wilson (2021), Proceedings of the 38th International Conference on Machine Learning, PMLR 139 — ARXIV-2102.13042
Key takeaways
- Modes are joined by volumes, not only curves. SPRO trains each new connecting vertex θ_j to minimize the mean loss over H = 5 points sampled uniformly from the current simplicial complex, minus λ_j log V(K), with the volume from Cayley–Menger determinants (Eq. 1). Two modes and three connectors give a complex in which every point of the connectors’ simplex defines a low-loss path between the modes (Fig. 3). Complexes join 4 modes through 3 connectors (VGG-16, CIFAR-100) and 7 modes through 9 connectors in 12 simplexes (VGG-16, CIFAR-10; Fig. 4).
- The low-loss region is at least 10-dimensional. Adding connectors between two VGG-16 modes on CIFAR-10, the complex keeps volume (every one of 25 sampled models above 98% train accuracy) until the eleventh connector, when it collapses from about 10⁵ to about 10⁻⁴ (§4.3, Fig. 5).
- A curve is already a family. Treating Garipov et al.’s curve objective as a likelihood and fitting a SWAG posterior over the bend gives a distribution of connecting curves, all of low loss (§4.1, Fig. 2).
- Simplexes make cheap ensembles. From one trained network, ESPRO adds vertices one at a time (about 10 epochs each on CIFAR-10 and 20 on CIFAR-100, against 200 and 300 for a full run) and ensembles 25 samples from the resulting simplexes. A VGG-16 deep ensemble of 3 has about 6.2% error on CIFAR-10; the same 3 networks extended to 2-simplexes have about 5.7% (§5.4). It holds for ResNet-56 (with BatchNorm statistics recomputed per sample) and for vision transformers on CIFAR-100. With temperature scaling it beats deep ensembles, MultiSWA and MultiSWAG under CIFAR-10 corruptions (§6.2).
Standing in the record
Filed with the batch ADR-027 opened loss-landscapes for; “star domains and
connecting simplexes” is one of the topic’s named subjects. It extends
Garipov et al. (LIT-673) directly: SPRO is their curve-finding procedure
generalized from one bend to many connecting points, and from a curve to the
simplexes the points span. Its first experiment puts a posterior over their
curve. It credits Garipov et al. and Draxler et al. (LIT-653) jointly
with discovering low-loss curves.
Git Re-Basin (LIT-661) reads this paper as having shown that “SGD solutions form a connected volume of low loss”, and Entezari et al. (LIT-652) as conjecturing that the volume is convex modulo permutation. That pairing frames the question the star-domain readings in this batch (LIT-659, LIT-682) ask: whether the volume is convex after alignment, or star-shaped around some centre.
The ensembling half is an instruction for practice that the anthology could
hold, hence anthology-candidate. The anthology holds none of its baselines
(deep ensembles, MultiSWAG) by arXiv id.