nucleation

LIT-782: GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment

3 min read

Properties5
StatusActive — read 2026-10-09 (NOTE-593); worth reading as a measurement of where text-to-image models of 2023 fail at composition, and of how badly embedding similarity sees it: a COCO-trained detector plus masked-crop CLIP colour classification judges each image correct or not on presence, count, position and colour, agreeing with crowd annotators 83% of the time against 88% between annotators, and beating a per-task tuned CLIPScore on counting (by 22 points), position and attribute binding. The best open model (IF-XL) gets 0.61 overall but at most 0.15 on relative position and 0.35 on binding; scaling IF helps binding but not position, and continued training of Stable Diffusion v1 leaves scores flat.
Tagscompositionality, anthology-candidate
SupportsCLAIM-013 — Existing text-to-image benchmarks evaluate object presence, attributes and whether spatial or interactive relations are depicted, not the stance, social alignment or uptake an image conveys, and need extending to pragmatically consequential relations and social uptake, CLAIM-123 — In conditional generation, adding scores composes two conditions exactly when the ratio p(y₁,y₂|x)/(p(y₁|x)p(y₂|x)) does not vary with the noisy state x, which conditional independence of the conditions at that state is sufficient for but does not require, CLAIM-321 — In text-to-image diffusion models each condition of a prompt can be met while their conjunction or relational binding fails: benchmarks find single objects rendered almost always and spatial relations and attribute binding failing most, and composing by adding scores does worst on relations
Cited as “paper” byNOTE-593 — GenEval
Cited as “source” byTHEORY-522 — Open text-to-image diffusion models of 2022–2023 render a single named object almost always but fail most on relative position and next on attribute binding, by human rating and by detector checks alike; scaling the image model or continuing pretraining does not fix position, and interactions are rated near ceiling only by rubrics that ask whether an action is depicted

Graph View

Dhruba Ghosh, Hannaneh Hajishirzi and Ludwig Schmidt (2023), NeurIPS 2023 Datasets and Benchmarks — ARXIV-2310.11513

Key takeaways

  • Decompose the prompt, then check the parts. Each of 553 templated prompts over the 80 COCO classes (six tasks: single object, two object, counting, colours, position, attribute binding) is checked by a Mask2Former detector for presence and count, by box centroids with a minimum offset for position, and by zero-shot CLIP on a masked crop for colour. An image scores 1 only if every part is right, and a failed image comes with the reason.
  • Close to human agreement (Section 4). On 1,200 images with five annotations each, GenEval agrees with annotators 83% of the time, annotators with each other 88%, and a CLIPScore with its best backbone and a per-task tuned threshold 80%. GenEval wins on counting (by 22 points), position, two object and binding; CLIPScore is slightly ahead on the easy single-object and colour tasks.
  • Where 2023 models fail (Table 2). IF-XL 0.61 overall, SD-XL 0.55, SD v2.1 0.50. Relative position tops out at 0.15 and attribute binding at 0.35, while single objects are near 0.98. IF-XL places the first named object on the left more often than on the right; SD v2.1 swaps the two colours more often than IF-XL (Figure 6).
  • Scale against training (Figure 5). Larger IF models improve on binding and counting but not on position; five successive checkpoints of SD v1, trained on more LAION data, stay at 0.41–0.44, while SD v2, with a different text encoder, reaches 0.50–0.51.
  • Bounded by its detector: COCO classes, photographs, and failures on masks with holes and merged same-class objects (Figure 4, Section 6).

The record’s reading of compositional generalization in vision models is LIT-667 (THEORY-117). GenEval measures compositional failure in generators without controlling what their training data covered, so it does not test that account (NOTE-593). Its companion benchmark in the same NeurIPS track is T2I-CompBench (LIT-783).

Standing in the record

Filed on 2026-10-09 at the owner’s request, as one of the works in the reference list of the owner’s working manuscript (October 2026) that the record did not yet hold. See the curation entry of that day. It is a machine-learning paper an anthology topic could hold (generative-modeling, analysis-and-evaluation), so it carries the anthology-candidate flag (ADR-005); it is here because the owner asked for the manuscript’s references to be filed in this record.

After reading. The paper is an evaluation method for text-to-image models, and what it teaches (check detected objects rather than embedding similarity; crop and mask before classifying colour) is machine-learning practice, anthology material. The flag stays. It was first filed under representation-learning, which the reading did not support: the paper does not study how a model represents its data, only whether its outputs are right. It is filed under compositionality (ADR-031), the subject it shares with this record; the evaluation of generative models is an anthology subject.

Backlinks