nucleation

LIT-938: Evaluating Compositional Scene Understanding in Multimodal Generative Models

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-813) in full; worth reading as a human-rated measurement of relational binding in one text-to-image model: DALL-E 3 renders familiar single relations well, loses them when the roles of an asymmetric relation are reversed or two relations are combined, and fails on the relation or the roles rather than the objects. Read it knowing that it tests one generator, that its experiments differ in style, raters and prompt selection, and that its explanation by training-data likelihood rests on a correlation of 0.09. Filed for QUESTION-018; the finding is held as THEORY-250.
Version2
Tagscompositionality, anthology-candidate
Cited as “paper” byNOTE-813 — Fu et al. — Evaluating Compositional Scene Understanding in Multimodal Generative Models
Cited as “source” byTHEORY-250 — DALL-E 3 renders single familiar relations well but is not robustly compositional: human-judged agreement falls when subject and object of an asymmetric relation are reversed and when two relations are combined, and its failures on reversed prompts are mostly wrong or role-swapped relations, not missing objects

Graph View

Shuhao Fu, Andrew Jun Lee, Anna Wang, Ida Momennejad, Trevor Bihl, Hongjing Lu and Taylor Webb (2025), Transactions on Machine Learning Research.

Key takeaways

From the reading, NOTE-813.

  • Familiar single relations are largely solved. Agreement is above 0.8 for ‘on’, ‘near’ and ‘in’ (DALL-E 2: below 0.4), and higher than DALL-E 2 on every relation tested.
  • Reversal breaks asymmetric relations. “A rabbit chasing a tiger” fails where “a tiger chasing a rabbit” succeeds (t(29) = 3.67). The drop is large for ‘on’, ‘in’ and ‘chasing’, small for ‘near’ and ‘touching’.
  • Binding, not objects. Among the worst reversed images, the errors are mostly the wrong relation or the roles swapped back.
  • Two relations are mostly too many. Most combinations among three entities get low agreement.
  • The why is thin. Training-data likelihood is proposed on a correlation of r = 0.09 with a GPT-3 proxy. A multi-object binding problem is proposed by citation.
  • VLMs. Five models learn relational concepts few-shot below human level; four are at chance with six objects, and Claude 3.5 Sonnet is above it. As judges of image–prompt match they correlate with humans at r ≈ 0.43–0.56, against 0.91 between halves of the human raters.

Standing in the record

Filed 2026-10-12 for QUESTION-018 and read 2026-10-14 in NOTE-813. Its finding about DALL-E 3 is held as THEORY-250, Proposed. That theory measures QUESTION-018’s explanandum, conditions lost in conjunction and role bindings lost, and does not answer the question. It qualifies CLAIM-013’s picture of object-centred benchmarks, as CLAIM-097 does, and extends CLAIM-097’s doubt about automatic scoring to VLM judges. The benchmark sets, the judge result and the object-centric remedy are ML practice for the anthology.

Backlinks