Ramesh et al. (2022) — ARXIV-2204.06125
Key takeaways
- Two explicit stages: a prior mapping a caption to a CLIP image embedding, and a diffusion decoder rendering from that embedding. The explicit intermediate is what buys zero-shot language-guided manipulation, since CLIP’s space is joint over text and images
- The split buys diversity and costs fidelity, stated plainly: humans slightly prefer GLIDE on photorealism — the gap is small — and strongly prefer unCLIP on diversity
- A diffusion prior beats an autoregressive one on both compute and sample quality — an early instance of diffusion displacing AR in a component role
- Guide the decoder, not the prior: “we found that guiding the prior hurt results”
- The embedding does not carry attribute binding. Figure 15’s reconstructions of difficult binding problems mix up which object has which colour and fail on relative size — a property of the representation, not the decoder
Standing in the anthology
Read — NOTE-064. Nothing is sourced here.
The reading generalises a condition the record holds in one place. SOTA-187
carries “the latent must be perceptually equivalent, and what the autoencoder
discards is a hard ceiling the diffusion model cannot see past”. This paper
shows the same failure in a different representation, with a concrete
diagnostic — reconstruct through the bottleneck and look at what breaks —
and what breaks is attribute binding, exactly what text-to-image models were
criticised for at the time. Better-supported than the record’s own source for
that condition, and it comes with a way to test it.
The one later comparison against this model, DALL-E 3 (LIT-648), lands on the same axis. Against DALL-E 2 it reports T2I-CompBench colour binding of 81.1% against 59.2%, GPT-V-judged Drawbench accuracy of 70.4% against 49.0%, and MSCOCO CLIP score of 32.0 against 31.4. Its own footnote says DALL-E 3 has improvements that could not be ablated, and that the numbers are not the effect of any one change. So it measures the binding gap without locating it in the CLIP bottleneck.
Of the takeaways replaced, “improved composition ability” is backwards: the paper’s own figure shows composition failing.
Audited by #121; unsourced. R1 is the methodological claim — when inserting an explicit intermediate representation, measure what it fails to carry by reconstructing through it — and it is a general instrument the record does not hold. R2 (expect a representation bottleneck to buy diversity and cost fidelity) is the finding, and it is one measurement. Unfiled: the instrument is worth having and the record has no second use of it to file it from.