Wortsman et al. (2022) — ARXIV-2203.05482
Key takeaways
The observation. The standard recipe fine-tunes k models over a
hyperparameter sweep, keeps the one with the best held-out accuracy, and
discards the rest. Because they were all fine-tuned from the same pretrained
weights, they “often appear to lie in a single low error basin” — so the
discarded models can be averaged into the kept one instead.
Two recipes, and only one of them is safe.
- Uniform soup: average all
k. One bad ingredient — a model fine-tuned with a learning rate that put it in a different basin — drags the average below the best individual model. The paper reports this failure directly. - Greedy soup: sort the models in decreasing held-out accuracy, then add each in turn and keep it only if held-out accuracy improves. This is the paper’s central method, and the reason is a guarantee rather than a measurement: it “can be no worse than the best individual model on the held-out validation set”.
A third learned soup optimizes the interpolation weights by gradient descent and needs every model resident in memory at once, which is what rules it out in practice.
Cost. Every soup is O(1) at inference against an ensemble’s O(k). The
soup is free given a sweep you were going to run anyway — the 32 models per
dataset in the NLP experiments are not free if you were not already sweeping.
What it buys, in the setting it was built for. ViT-G pretrained on JFT-3B and fine-tuned on ImageNet, top-1 and average over distribution shifts:
| ImageNet | shifts | |
|---|---|---|
| ViT-G, reevaluated | 90.47 | 82.06 |
| best model in the sweep | 90.78 | 84.68 |
| greedy soup | 90.94 | 85.02 |
Over the best sweep member: +0.16 on ImageNet and +0.34 under shift. On CLIP and ALIGN the greedy soup improves over the best sweep member by +0.7 and +0.5 pp.
The state-of-the-art claim is the least durable part, and the FLOPs are the most. 90.94 surpassed CoAtNet-7’s 90.88 — by 0.06 pp — “while requiring 25% fewer FLOPs at inference time”. Read the 25% and not the 0.06.
The NLP extension is weaker than the abstract makes it sound. The abstract says the approach “extends to multiple image classification and natural language processing tasks”. §3.3.3 calls the NLP work “preliminary experiments” and “a promising initial step”, and the BERT row on four GLUE tasks, from a 32-model random sweep each, reads:
| MRPC | RTE | CoLA | SST-2 | |
|---|---|---|---|---|
| best individual model | 88.3 | 61.0 | 59.1 | 92.5 |
| greedy soup | 88.3 (+0.0) | 61.7 (+0.7) | 59.1 (+0.0) | 93.0 (+0.5) |
Two of four gain nothing at all, and the authors say so: “the improvements are not as pronounced as in image classification”. The family now has a non-vision data point; it does not have a non-vision result.
And an account, offered analytically and checked. The paper relates how closely weight-averaging tracks logit-ensembling to the flatness of the loss and the confidence of the predictions, then validates the relation empirically. That is a claim about when the two operations coincide rather than about why either works, and the record’s flatness cluster — SOTA-012 — is where the surrounding argument is kept and hedged.
Standing in the anthology
Filed because SOTA-407’s consensus note named it as unheld, in a sentence written earlier today: the record could not name an adopter of weight-space ensembling “outside the authors’ own line of work — model soups, which this record does not yet hold”. That is the fourth time in a week that a unit’s own prose has set the next unit’s subject.
It does not do what the note wanted it to do. Mitchell Wortsman is first
author of both this and LIT-674. DP-005 says adoption is not evidence,
and a research group extending its own method is the clearest case of that:
this is a second paper, not a second opinion. SOTA-407 stays emerging
and its note now says why rather than leaving the dependency open.
What it does supply: SOTA-409, the recommendation to soup a sweep rather than discard it, and a third case for SOTA-217’s shared-trajectory scope — with the sharpest statement of why the scope matters, since the greedy recipe exists precisely to “avoid adding in models which may lie in a different basin of the error landscape”.
Where it stops. Image classification is where the numbers are; NLP is preliminary by the authors’ own label. Every ingredient must be fine-tuned from the same pretrained initialization — the single-basin assumption is what the whole method rests on, and the greedy recipe is an admission that the assumption fails often enough to need a guard.