Source
Wortsman et al. (2021), LIT-674 — ARXIV-2109.01903.
What to do
Keep the zero-shot weights. Fine-tune from them as usual. Then ship the average rather than the endpoint:
θ = (1 − α) · θ_zero-shot + α · θ_fine-tuned, α = 0.5
α = 0.5 is the default the paper recommends when nothing is known about the
deployment distribution, and it reports that value as close to optimal across
its experiments. Sweeping α costs no training, so treat it as a
free hyperparameter evaluated at deployment time — and the models it reaches are
reported to be as good as or better than those other hyperparameter settings
reach, which makes α a cheap substitute for part of a search.
Why it is not a trade-off dial
The reason to state this as a recommendation rather than as an option is that
intermediate α beats both endpoints on both axes. Against the
fine-tuned model at α = 0.5, accuracy under six distribution shifts rises by
3.5, 6.2, 1.7, 2.1, 9.0 and 23.2 pp, while accuracy on the reference
distribution falls by at most 0.3 pp and frequently improves. On ImageNet
plus five derived shifts the paper reports 4–6 pp better under shift than prior
work and 1.6 pp better on ImageNet itself.
These are Wortsman et al.’s measurements on CLIP-family models in LIT-674, which
proposed the interpolation (released as WiSE-FT) and is where the α = 0.5
default comes from: the paper recommends it when nothing is known about the
deployment distribution, and also reports 0.8 to 3.3 pp over standard
fine-tuning on seven transfer datasets.
A method that dominates one of its endpoints and costs nothing is not a knob to consider; it is what to do unless something rules it out.
The condition that rules it out
The two weight vectors must share an optimization trajectory. This is not a general operation on two networks that solve the same task. The paper is explicit that averaging all layers of unrelated networks “typically fails, achieving no better accuracy than a randomly initialized neural network”, and that the interpolation works here because the fine-tuned model came from the zero-shot one and the two are joined by a linear path of high accuracy.
If you need to average networks that were trained separately, this practice does not apply and SOTA-217 does: align the hidden-unit permutation first, for the reason THEORY-010 gives.
Other conditions
Fine-tuning only a linear head makes this identical to output-space ensembling, algebraically, because the head enters linearly. There is nothing wrong with using it there — it just is not a new method in that case, and a reader should not take a linear-probe result as evidence for the end-to-end claim.
Image classification only. The authors state the limitation. Every number here is a CLIP-family classifier on image benchmarks.
The wider family has since reached text, barely: LIT-675 reports greedy soups on four GLUE tasks with gains of +0.0, +0.7, +0.0 and +0.5 for BERT and calls the experiments preliminary. That is a data point for weight averaging outside vision and it is not a result for this practice, which nobody has run on text.
It preserves robustness rather than creating it. What the interpolation recovers is the zero-shot model’s own consistency across distributions. Starting from a model that was never robust leaves nothing to recover.
Known implementations
- WiSE-FT, the authors’ release.