Zhou et al. (2021) — ARXIV-2109.01134
Key takeaways
- Names prompt engineering as the deployment bottleneck for CLIP-style models — “a slight change in wording could have a huge impact on performance” — and replaces it with continuous context vectors learned end-to-end while every pretrained parameter stays frozen. Two variants: unified context, and class-specific context
- One or two shots beat hand-crafted prompts by a decent margin; at 16 shots the average gain across 11 datasets is about 15%, the highest over 45%
- Domain generalisation is strong against the zero-shot model with hand-crafted prompts — surprising, since a learned prompt is the thing one expects to overfit
- Failures are reported with a diagnosis: on Food101 both CoOp and linear probing underperform, attributed to data with “intense colors and sometimes wrong labels”, with the learning curves showing “seemingly an overfitting problem” and higher weight decay suggested as an unverified remedy
Standing in the anthology
Read — NOTE-049. Retagged from model-architecture — nothing architectural
happens — to adaptation-and-tuning. Nothing is sourced here.
The transferable statement the record lacks: replace hand search over a
discrete artefact with gradient descent over its continuous relaxation. Prompt
optimisation here, textual inversion in LIT-078, and soft prompting in
language models are three instances of one move across two modalities.
The takeaway “zero-shot capabilities” was misleading — CoOp is few-shot and needs labels. What it preserves is the zero-shot model’s generality, which is a different claim.
Audited by #121; unsourced. R2 — adapt in the input space before adapting weights — is the transferable claim, and the record’s adaptation practices all start at the weights (SOTA-184). Left unfiled because the record holds no second source for input-space adaptation and one 2022 vision-language result is thin for a claim that would sit above LoRA in the recommended order.