Source
Radford et al. (2021), LIT-588 — ARXIV-2103.00020.
The claim
When you build a classifier by embedding class names with a contrastively trained text encoder, do not embed the bare class name. Put it in a sentence, build several sentences, and average their embeddings — not their probabilities — into one classifier.
The measurements, on ImageNet:
| step | gain |
|---|---|
"A photo of a {label}." over the bare name | +1.3% |
| ensembling 80 context prompts on top of that | +3.5% |
| together, averaged over 36 datasets | ~+5% |
The paper measures that ~5% as equivalent to using 4× more compute with the contextless method.
The recommendation is CLIP’s own: LIT-588 built its zero-shot classifiers this way, wrote the 80 ImageNet templates, measured each step above, and chose to average in embedding space rather than probability space precisely so that one cached set of class vectors would carry the whole ensemble.
Why it is nearly free, which is the part that makes it a practice
Averaging happens in the embedding space, not over softmax outputs. So the 80 prompts collapse into a single set of class vectors, computed once per label set and cached. Amortised over a dataset, an 80-prompt ensemble costs what one prompt costs. A probability-space ensemble would cost 80× and is the obvious wrong way to do it.
Why the bare class name loses
It is a distribution mismatch, not a semantic one. Almost nothing in a
web-scraped caption corpus is a single word; the text the encoder saw was
sentences. "A photo of a {label}." moves the query back into the
distribution the encoder was trained on. That also explains why the task
hints help — ", a type of pet" on Oxford-IIIT Pets, "a satellite photo of" on satellite imagery, quotes around the string for OCR: each names the
context the matching caption would have carried.
Conditions
- This is a property of the evaluation, not of the model. Two papers reporting zero-shot numbers for the same checkpoint can differ by ~5% on prompting alone, which is larger than most claimed improvements. A zero-shot number without its prompt set is not comparable to another one.
- The templates are dataset-specific and hand-written. The paper’s own list was tuned against the datasets it reports on, and the authors say their evaluation suite is “undeniably co-adapted” with CLIP. Carrying the ImageNet 80 to a new domain is a starting point, not a result.
- It is a prompting practice for contrastive image-text classifiers. The mechanism — put the query in the training distribution, ensemble cheaply where the geometry lets you — generalises further than the template list does.
Known implementations
mlfoundations/open_clip—OPENAI_IMAGENET_TEMPLATES(80) andSIMPLE_IMAGENET_TEMPLATES(7) insrc/open_clip/zero_shot_metadata.py.