Yu et al. (2022) — ARXIV-2205.01917
Key takeaways
- One encoder-decoder trained with both a contrastive and a captioning loss, via a single change: the first half of the decoder layers omit cross-attention, so they produce a unimodal text representation for the contrastive loss, while the cascaded remainder cross-attends to the image encoder for captioning
- The two objectives share one computational graph and one forward pass, so the second costs almost nothing. Contrastive wants a text embedding that has not seen the image and generative wants one that has — and “has not seen it yet” is what early decoder layers are if you withhold cross-attention
- Pretrained end-to-end from scratch on web alt-text and annotated images by treating all labels simply as text, so heterogeneous supervision enters one objective
- ImageNet 86.3% zero-shot, 90.6% frozen-encoder with a learned head, 91.0% fine-tuned; state of the art across recognition, retrieval, multimodal understanding and captioning
Standing in the anthology
Read — NOTE-055. Nothing is sourced here.
The reading’s value is a second instance of a pattern. LIT-097 unifies several
distributions by making the timestep a per-modality value; CoCa unifies two
objectives by making fusion a function of depth. Both replace an
architectural branch with a continuous variable, and both get a capability or a
saving from it. Two independent instances is when a pattern is worth stating,
and the record states it nowhere.
“Zero-shot capabilities” is the fourth bullet in this batch to stand in for content, and none of the four mentioned the mechanism, which is one sentence long.
Audited by #121; unsourced. R3 is the one that leaves the domain: cost a second objective by whether it shares a forward pass. The reading marks it strong and underused, and the record’s multi-objective practices (SOTA-162) do not reason this way explicitly. It is a candidate; it is unfiled because it is an accounting rule rather than a measured result, and the record has no instance of the rule being wrong to bound it.