Source
Izmailov, Podoprikhin, Garipov, Vetrov and Wilson (2018), LIT-673 — ARXIV-1803.05407.
What to do
Three parts, and the third is the one people skip.
- Do not decay the learning rate to zero. Over the stretch of training you intend to average, hold it high — either constant or cyclical. The averaging is what replaces the decay.
- Keep a running average of the weights over that stretch, and ship the average rather than the last iterate. Inference cost is one model.
- Recompute the normalization statistics. A batch-normalized network stores running means and variances that belong to the weights that were visited, not to their average. One additional pass over the training data with the averaged weights fixes it. Omitting this does not cost a fraction of a point; it produces a model that does not work.
What it buys
ImageNet top-1, starting from pretrained torchvision checkpoints and running the averaging for further epochs under one cyclical schedule shared across architectures, mean over three runs:
| baseline | 5 epochs | 10 epochs | |
|---|---|---|---|
| ResNet-50 | 76.15 | 76.83 ± 0.01 | 76.97 ± 0.05 |
| ResNet-152 | 78.31 | 78.82 ± 0.01 | 78.94 ± 0.07 |
Better than +1.3% on CIFAR-100 and better than +0.4% on CIFAR-10 across Preactivation ResNet-164, VGG-16 and Wide ResNet-28-10. The paper’s framing is that this “approximates Fast Geometric Ensembling with a single model” — an ensemble’s generalization without an ensemble’s inference cost.
The method and these measurements are Izmailov et al.’s, LIT-673, which introduced stochastic weight averaging: ten further epochs of averaging buy +0.82 on ResNet-50 and +0.63 on ResNet-152, and the batch-normalization re-estimation in step 3 is the paper’s own instruction, not later folklore.
Conditions
The compute charge belongs to this method, not to averaging. SOTA-428 maintains its averages during the training run and stores two parameter vectors per snapshot, so it pays storage instead of epochs, and the length of the average is chosen after the fact rather than by the cyclical schedule. Different cost structure for a related end, in a different setting (ImageNet-512 diffusion against ResNet classification here), and the two have never been compared. The charge below still applies to what this practice asks for.
The extra epochs are real compute. +0.8 on ImageNet for ten epochs of averaging is a good trade at the end of a long run and a poor one if those epochs would otherwise have gone into training. The comparison the paper makes is against a converged SGD baseline, not against spending the same compute elsewhere.
Convolutional vision in 2018, against SGD. Every number here is a
convolutional architecture and the baseline is SGD with a decayed learning rate.
The promote_when asks for a transformer and a modern optimizer because that is
what the record does not have, and because a decayed-LR AdamW baseline is a
harder thing to beat than the one measured here.
Proposed while the consensus reads converged. This is the ADR-015
corner used deliberately: the technique ships in PyTorch and appears throughout
this record as a baseline, and the record has still never seen it measured in the
setting its readers train in. Adoption is not evidence — DP-005 — and here the
adoption is real and the evidence is old.
Related but distinct from interpolating two endpoints. SOTA-407 averages a zero-shot and a fine-tuned model; this averages points along one trajectory. Both avoid the permutation problem SOTA-217 exists for, for the same reason — a shared trajectory — and they are different operations with different evidence.
What it is evidence for, and what it is not. The paper argues the averaged solution is flatter and that this explains the gain. SOTA-012 is where this record keeps the flatness correlation and its hedges; nothing in this practice depends on that account being right.
Known implementations
- PyTorch, as
torch.optim.swa_utils, including anupdate_bnhelper for step 3. - SWA, the authors’ release.