Izmailov, Podoprikhin, Garipov, Vetrov and Wilson (2018) — ARXIV-1803.05407
Key takeaways
The method, which is almost nothing. Train as usual, then keep a running average of the weights visited over the last stretch of training while the learning rate is held high — cyclically or at a constant value rather than decayed to zero. Ship the average. The paper’s own description is that it “approximates the recent Fast Geometric Ensembling approach with a single model”, which is the point: an ensemble’s generalization for one model’s inference cost.
The numbers, from already-trained models. ImageNet top-1, running SWA for ten further epochs on pretrained torchvision checkpoints with one cyclical schedule shared across architectures, mean over three runs:
| SGD | SWA, 5 epochs | SWA, 10 epochs | |
|---|---|---|---|
| ResNet-50 | 76.15 | 76.83 ± 0.01 | 76.97 ± 0.05 |
| ResNet-152 | 78.31 | 78.82 ± 0.01 | 78.94 ± 0.07 |
So +0.82 and +0.63 for ten epochs of extra training and no change to the model. The paper also reports better than +1.3% on CIFAR-100 and better than +0.4% on CIFAR-10 across Preactivation ResNet-164, VGG-16 and Wide ResNet-28-10.
The gotcha, and it is in the paper rather than folklore. If the network uses batch normalization, the running mean and variance stored in the checkpoint belong to the weights that were visited, not to their average, so one additional pass over the data is needed to recompute them with the averaged weights. Skip it and the averaged model is broken rather than merely disappointing.
The account the paper offers. SWA’s solutions are flatter than SGD’s, and the paper argues SGD with a high learning rate circles the periphery of a wide flat region without entering it, while the average of those peripheral points lands inside. That is a claim about the geometry, and the record already has a cluster arguing about whether flatness earns its reputation — SOTA-012 is where the correlation is kept and hedged.
Standing in the anthology
Filed because LIT-478 named it. That note’s “Why it’s here” section says the
record has “nothing on weight-space ensembling. No stochastic weight averaging,
no model soups, no snapshot ensembles, no Polyak averaging”, and it names SWA as
the baseline the method it describes improves on. SOTA-288’s promote_when then
asks for a replication “with stochastic weight averaging as the baseline rather
than plain SGD” — a condition the record could not have checked, because the
baseline was not in it.
It is worth being precise about what was missing, because LIT-478 overstated it: the record held SOTA-217, THEORY-010, LIT-333 and LIT-251 — the permutation-alignment half of weight averaging — six days before that sentence was written. What was missing is the shared-trajectory half, which is this paper and LIT-674, and where no alignment is needed at all.
What this is not. The gains are fractions of a percent on ImageNet and around a point on CIFAR, on 2018 convolutional architectures, against an SGD baseline. Nothing here is measured on a transformer, which is exactly the gap SOTA-288’s condition names, and the extra epochs are real compute rather than a free lunch.