Source
Wortsman et al. (2022), LIT-675 — ARXIV-2203.05482.
What to do
At the end of a fine-tuning sweep, you have k models and are about to keep one.
- Sort them by accuracy on a held-out set that is disjoint from training and test.
- Start the soup with the best one.
- Take each remaining model in order and add it to the average only if the soup’s held-out accuracy improves.
- Ship the soup.
Do not simply average all k. One ingredient fine-tuned with a learning
rate that put it in a different basin will drag a uniform average below the best
individual model, and the paper reports exactly that failure. The sorting and the
accept-only-if-it-helps test are what make this safe.
Why this is a recommendation rather than an option
The greedy recipe “can be no worse than the best individual model on the held-out validation set” — by construction, not by measurement. The thing you were going to ship is the soup’s first ingredient, and every subsequent step is gated on improving. The downside is bounded at zero on the selection metric and the cost is one averaging pass over weights you already have.
The recipe, the guarantee and the measurements below are Wortsman et al.’s in LIT-675, which proposed souping sweep members on the observation that models fine-tuned from the same pretrained weights “often appear to lie in a single low error basin”, and which reports the uniform soup’s failure alongside the greedy one’s gain. With ViT-G the greedy soup reaches 90.94 on ImageNet against 90.78 for the best sweep member.
What it gains, over the best sweep member:
| gain | |
|---|---|
| CLIP | +0.7 pp |
| ALIGN | +0.5 pp |
| ViT-G on ImageNet | +0.16 |
| ViT-G under distribution shift | +0.34 |
And it stays one model: O(1) at inference, against O(k) for the logit
ensemble that would otherwise be the way to use all k.
Conditions
Every ingredient must be fine-tuned from the same pretrained initialization. This is the single-basin assumption the method rests on, and it is the same precondition as SOTA-407 and SOTA-408: averaging works here because the weight vectors share a trajectory. For separately trained networks, SOTA-217 applies instead. The greedy recipe is best read as an admission that even a shared initialization is not sufficient — some sweep members land somewhere the average cannot use, which is why each one is tested rather than trusted.
The soup is free given the sweep, not free. Every number above assumes you
were already training k models. Running a sweep in order to soup it is a
different and unmeasured proposition; the paper’s NLP experiments used 32 models
per dataset.
A held-out set gets consumed. The greedy construction selects on it, so it is no longer clean for reporting. Budget a third split, or accept that the guarantee is about the set you selected on.
The vision numbers are the real ones. On four GLUE text-classification tasks, the BERT greedy soup gains +0.0, +0.7, +0.0 and +0.5 — two of four gain nothing, and the authors call the NLP work preliminary. Treat this as established for fine-tuning image models and as untested elsewhere.
Known implementations
- model-soups, the authors’ release.