nucleation

THEORY-405: Adding one classifier-free guidance term per concept to a diffusion model's unconditional noise prediction samples images that satisfy several concepts at once without retraining, and on synthetic scenes it composes object positions far better than compositional energy-based models but relations no better; the reading of the summed score as the score of a product of experts holds only for concepts conditionally independent given the image, unit weights, a conservative field and the noise-free distribution

4 min read

Properties5
StatusActive
Version2
Tagscompositionality, probabilistic-modeling
SourceLIT-770 — Compositional Visual Generation with Composable Diffusion Models
SupportsCLAIM-123 — In conditional generation, adding scores composes two conditions exactly when the ratio p(y₁,y₂|x)/(p(y₁|x)p(y₂|x)) does not vary with the noisy state x, which conditional independence of the conditions at that state is sufficient for but does not require, CLAIM-321 — In text-to-image diffusion models each condition of a prompt can be met while their conjunction or relational binding fails: benchmarks find single objects rendered almost always and spatial relations and attribute binding failing most, and composing by adding scores does worst on relations

Graph View

Source

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba and Joshua B. Tenenbaum (2022), “Compositional Visual Generation with Composable Diffusion Models”, ECCV 2022, LIT-770, read in NOTE-597.

What was actually shown

The rule. A diffusion model’s noise prediction is, up to scale, a score. If concepts are conditionally independent given the image, p(x, c₁, …, cₙ) = p(x) Πᵢ p(cᵢ|x), and with p(cᵢ|x) ∝ p(x|cᵢ)/p(x) the target’s score is the unconditional score plus one likelihood-ratio gradient per concept (Eq. 9). Conjunction (Eq. 11): ε̂ = ε_θ(x_t, t) + Σᵢ wᵢ (ε_θ(x_t, t|cᵢ) − ε_θ(x_t, t)). With one concept and w > 1 this is classifier-free guidance. Negation (Eq. 15) divides by the negated concept’s likelihood. All terms come from one network trained with labels dropped 10% of the time.

Object positions, CLEVR (Table 1, 5,000 images per setting). Accuracy by a classifier trained on real images, 1/2/3 components: this method 86.42/59.20/31.36%, the compositional EBM 70.54/28.22/7.34%; StyleGAN2, LACE and GLIDE near 0 when composing. FID is best throughout.

Relations, CLEVR (Table 2). This method 60.40/21.84/2.80%, the EBM 78.14/24.16/4.26%. Three relations succeed under 3% of the time for every method. FID is better (26.11 against 58.66 at three).

Faces (Table 3). 99.26/92.68/68.86% against LACE’s 97.60/95.66/80.88%, with better FID at two and three components.

What could have come out otherwise. Composed sampling could have failed beyond one concept, as single-condition generators do. For positions it degrades but works; for relations it fails as badly as the energy-based baseline.

What this does not say

  • That the composed sample is the product distribution. The correspondence needs conditional independence, which is stated and not examined; unit weights, while the method tunes them; a conservative score field, which is set aside by citation; and the noise-free distribution. The summed scores are used at every noise level, where the score of the noised product is not in general the sum of the noised conditionals (NOTE-597’s observation; no paper the record holds states and tests it).
  • That it composes text prompts reliably. On GLIDE and Stable Diffusion there is no metric. The authors list concepts the base model does not know, attribute confusion that survives composition (“a bear in a red forest” AND “a car” gives a red bear), and fusion into one hybrid object (a dog-shaped cloud).
  • That it composes relations. Relations between objects are the obvious case of concepts that are not conditionally independent given the image, and they are where it fails.
  • That separately trained models compose. The authors report limited success composing models trained on different data.

Connections

  • Benchmarks of composition (THEORY-522, from LIT-782, LIT-783 and LIT-798). Re-implemented on Stable Diffusion v2, Composable Diffusion is the weakest method on most T2I-CompBench categories.
  • Attend-and-Excite (THEORY-678, from LIT-806) reports that its conjunction fuses two subjects into one object, qualitatively.
  • Compositional generalisation in recognition (THEORY-117, from LIT-667) reaches unseen pairs through a linearly factored representation. This method reaches unseen combinations by an additive structure imposed at sampling. Both need an independence or additivity assumption to hold; the parallel is NOTE-597’s, and this paper neither supports nor tests THEORY-117.
  • CLAIM-123 states the conditional-independence condition at the noisy state as the record’s own claim, and CLAIM-321 cites the relation failures.

Note of 2026-10-25: compositions that average

Two works read on 2026-10-25 compose by averaging where this paper sums. MultiDiffusion (LIT-1522, NOTE-1158) composes prompts by place: where regions overlap a pixel takes the weighted mean of the views’ proposals, and disjoint regions select (THEORY-958). LoRA Composite (LIT-1529, NOTE-1160) composes operators: every LoRA-modified model sees the whole image and their guided predictions are averaged, so total guidance stays at one scale instead of growing with k (THEORY-955). Under this account’s idealization an average of k scores is the score of the normalized geometric mean Πᵢ pᵢ^{1/k}, a product of experts each at temperature k, not of the product. That derivation is the record’s (NOTE-1160); neither paper gives a probabilistic reading. The account is unchanged.

Backlinks