nucleation

CLAIM-321: In text-to-image diffusion models each condition of a prompt can be met while their conjunction or relational binding fails: benchmarks find single objects rendered almost always and spatial relations and attribute binding failing most, and composing by adding scores does worst on relations

2 min read

Properties10
StatusActive
Version4
Rolethesis
Warrantempirical
Tagscompositionality, probabilistic-modeling
Linepragmatic-transport
Workswhat-survives-translation
GroundsLIT-782 — GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment, LIT-783 — T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation, LIT-798 — T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation, LIT-770 — Compositional Visual Generation with Composable Diffusion Models, THEORY-522 — Open text-to-image diffusion models of 2022–2023 render a single named object almost always but fail most on relative position and next on attribute binding, by human rating and by detector checks alike; scaling the image model or continuing pretraining does not fix position, and interactions are rated near ceiling only by rubrics that ask whether an action is depicted, THEORY-405 — Adding one classifier-free guidance term per concept to a diffusion model's unconditional noise prediction samples images that satisfy several concepts at once without retraining, and on synthetic scenes it composes object positions far better than compositional energy-based models but relations no better; the reading of the summed score as the score of a product of experts holds only for concepts conditionally independent given the image, unit weights, a conservative field and the noise-free distribution
CorroboratesCLAIM-123 — In conditional generation, adding scores composes two conditions exactly when the ratio p(y₁,y₂|x)/(p(y₁|x)p(y₂|x)) does not vary with the noisy state x, which conditional independence of the conditions at that state is sufficient for but does not require
Corroborated byTHEORY-678 — In Stable Diffusion, a subject named in a two-subject prompt is left out of the image when its token governs no region of the cross-attention map early in sampling, and raising that token's smoothed peak attention during the first half of sampling, by gradient steps on the latent with no retraining, brings the subject back; that this also fixes attribute binding is argued, not measured by the paper

Graph View

The claim

This is the observed half of CLAIM-123, stated on its own so that its warrant can be. CLAIM-123’s derivation says that adding scores, s₁₂ = s₁ + s₂ − s₀, composes conditions only under conditional independence at the noisy state and “does not guarantee correct joint bindings”. What the readings show is that joint bindings do in fact fail where single conditions are met.

  • GenEval (LIT-782, NOTE-593) scores every diffusion model it tests at 0.97–0.98 on single objects, and the best 2023 open model at most 0.35 on attribute binding and 0.15 on relative position; failures include colours swapped between the two objects. T2I-CompBench (LIT-783) builds its prompts around the same attribute-binding and relation failures.
  • Attend-and-Excite (LIT-806) documents conjunctions failing by omission (“catastrophic neglect”).
  • Composable Diffusion (LIT-770, NOTE-597), the method that adds scores, “loses to an energy-based baseline when composing relations”, and in T2I-CompBench++ (LIT-798) “Composable Diffusion is again the weakest method”.

What it does not say

  • What causes the failures. C7’s appendix, quoted in CLAIM-123: “When scores are approximate, compositional failures may reflect estimator error, model misspecification, or sampling limitations.” A failure is not evidence that conditional independence at the noisy state fails.
  • That a generative failure is evidence of sheaf contextuality, or that denoising time is historical or telephone-game time.
  • That binding always fails. The benchmarks report rates, for the models and prompt templates they test.
  • That every kind of relation fails. Interactions such as holding or wearing are rated near ceiling by human raters, under a rubric that asks only whether an action is depicted (THEORY-522, CLAIM-097).

Backlinks