nucleation

LIT-770: Compositional Visual Generation with Composable Diffusion Models

2 min read

Properties6
StatusActive — read 2026-10-09 (NOTE-597); worth reading as the paper that moved compositional energy-based generation onto diffusion models: reading a diffusion model's noise prediction as an energy gradient, concepts are composed at sampling time by adding one classifier-free-guidance term per concept (AND) or subtracting a concept's prediction (NOT), with no retraining. The product-of-experts reading is exact only for conditionally independent concepts, unit weights and a conservative score, and is derived at zero noise but applied at every noise level. It wins clearly on composing CLEVR object positions (31.36% against an EBM's 7.34% at three objects), not on relations (2.80% against 4.26%), and is shown on GLIDE and Stable Diffusion only qualitatively.
Tagscompositionality, probabilistic-modeling, anthology-candidate
Compared againstLIT-783 — T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation, LIT-798 — T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation, LIT-806 — Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models
SupportsCLAIM-123 — In conditional generation, adding scores composes two conditions exactly when the ratio p(y₁,y₂|x)/(p(y₁|x)p(y₂|x)) does not vary with the noisy state x, which conditional independence of the conditions at that state is sufficient for but does not require, CLAIM-321 — In text-to-image diffusion models each condition of a prompt can be met while their conjunction or relational binding fails: benchmarks find single objects rendered almost always and spatial relations and attribute binding failing most, and composing by adding scores does worst on relations
Cited as “paper” byNOTE-597 — Compositional Visual Generation with Composable Diffusion Models
Cited as “source” byTHEORY-405 — Adding one classifier-free guidance term per concept to a diffusion model's unconditional noise prediction samples images that satisfy several concepts at once without retraining, and on synthetic scenes it composes object positions far better than compositional energy-based models but relations no better; the reading of the summed score as the score of a product of experts holds only for concepts conditionally independent given the image, unit weights, a conservative field and the noise-free distribution

Graph View

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba and Joshua B. Tenenbaum (2022), ECCV 2022 — ARXIV-2206.01714

Key takeaways

  • Composition by adding scores. A diffusion model’s noise prediction is read as the gradient of an energy, so a product of densities is a sum of predictions. Conjunction is ε(x,t) + Σᵢ wᵢ(ε(x,t|cᵢ) − ε(x,t)), the score of p(x)Πᵢ p(x|cᵢ)/p(x) when the concepts are conditionally independent given the image; with one concept it is classifier-free guidance. Negation divides by the negated concept’s likelihood: ε(x,t) + w(ε(x,t|cᵢ) − ε(x,t|c̃ⱼ)). No retraining (Section 4, Appendix F).
  • Heuristic, not exact. The learned field need not be conservative (set aside by citation), the weights are tuned, all terms must come from the same network, and the product rule is derived for the data distribution but used at every noise level, which the paper does not discuss.
  • Results. CLEVR object positions, accuracy at one, two and three composed positions: 86.42, 59.20 and 31.36%, against the EBM baseline’s 70.54, 28.22 and 7.34%, with the best FID throughout (Table 1). Relations: 60.40, 21.84 and 2.80%, below the EBM’s 78.14, 24.16 and 4.26%, though with better FID (Table 2). FFHQ attributes: below LACE in accuracy at three attributes (68.86 against 80.88%), best FID (Table 3). Composed GLIDE and Stable Diffusion are shown in figures only.
  • Failure modes the authors list: concepts the base model does not know, attribute confusion that composition does not cure, and fusion of two objects into one hybrid when objects are centred.

The record’s reading of compositional generalization in recognition is LIT-667 and THEORY-117; the reading of this paper sets the two side by side (NOTE-597).

Standing in the record

Filed on 2026-10-09 at the owner’s request, as one of the works in the reference list of the owner’s working manuscript (October 2026) that the record did not yet hold. See the curation entry of that day. It is a machine-learning paper an anthology topic could hold (generative-modeling, analysis-and-evaluation), so it carries the anthology-candidate flag (ADR-005); it is here because the owner asked for the manuscript’s references to be filed in this record.

Backlinks