nucleation

LIT-806: Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

3 min read

Properties6
StatusActive — read 2026-10-09 (NOTE-639); worth reading as the paper that names "catastrophic neglect" (a text-to-image model omits one of the subjects a prompt names) and links it to attribute binding: the subject token that no image patch attends to is the one that goes missing. Its fix is an inference-time correction of the latent: push up the maximum cross-attention of the most neglected subject token, smoothed over neighbouring patches, during the first half of sampling. Evidence is on 276 two-subject Stable Diffusion prompts, by CLIP similarities, BLIP captions and a 65-person preference study (77–91% preferred it). That it improves attribute binding is shown only in figures; no binding accuracy is measured, and relations are left out of scope.
Version2
Tagscompositionality, anthology-candidate
Compared againstLIT-770 — Compositional Visual Generation with Composable Diffusion Models, LIT-798 — T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation
Cited as “paper” byNOTE-639 — Attend-and-Excite
Cited as “source” byTHEORY-678 — In Stable Diffusion, a subject named in a two-subject prompt is left out of the image when its token governs no region of the cross-attention map early in sampling, and raising that token's smoothed peak attention during the first half of sampling, by gradient steps on the latent with no retraining, brings the subject back; that this also fixes attribute binding is argued, not measured by the paper

Graph View

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf and Daniel Cohen-Or (2023), ACM Transactions on Graphics 42(4) (SIGGRAPH 2023) — ARXIV-2301.13826

Key takeaways

  • Two failures, one cause (Section 1, Fig. 2). In Stable Diffusion, “catastrophic neglect”: one or more subjects of the prompt are not generated; and “incorrect attribute binding”: an attribute lands on the wrong subject or on none. The authors’ account: the CLIP text encoder already mixes “blue” into the token “cat”, so making the cat appear should also carry its colour.
  • The mechanism (Section 4, Algorithm 1). Cross-attention gives each 16×16 image patch a distribution over prompt tokens. Nothing makes every token dominant somewhere. At each of the first 25 of 50 steps, the loss L = max over subject tokens of (1 − max over patches of the Gaussian-smoothed attention map) is reduced by one gradient step on the latent; at steps 0, 10 and 20 the step is repeated until each subject reaches attention 0.05, 0.5 and 0.8. Without smoothing, one patch can satisfy the loss with a fragment (a crown-like patch on a rabbit’s head; Appendix B).
  • The evidence (Section 5). 276 prompts of three templates (two animals; an animal and a coloured object; two coloured objects), 64 seeds each, against Stable Diffusion, Composable Diffusion and StructureDiffusion. Minimum per-subject CLIP similarity beats Stable Diffusion and StructureDiffusion by at least 7%; BLIP-caption similarity to the prompt beats all three by at least 4.7% (Table 1); in a 65-respondent study it is preferred 90.7%, 77.6% and 77.2% of the time on the three subsets (Table 2). Composable Diffusion tends to fuse the two subjects into one object, which CLIP image–text similarity rewards and caption similarity does not.
  • What is not shown. Binding is not measured: the improvements in colour binding are seen in figures. Relations (“riding on”, “in front of”, “beneath”) are stated as out of scope.

Standing in the record

Filed on 2026-10-09 from the manuscript bibliography of 2026-10-09 (work what-survives-translation) at the owner’s request: the manuscript considered it and dropped it from the final reference list. It is read here on its own merits. See the curation entry of that day.

It is a machine-learning paper an anthology topic could hold (an inference-time method for text-to-image models), so it carries the anthology-candidate flag (ADR-005). It is filed under compositionality, the subject it shares with this record: whether every part of a prompt reaches the image, and whether the parts stay bound. The later benchmark T2I-CompBench (LIT-783) and its journal version (LIT-798) measure it re-implemented on Stable Diffusion v2, where it is the strongest of the 2022–2023 training-free methods on colour and texture binding and does not help relations.

Backlinks