nucleation

LIT-798: T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation

3 min read

Properties7
StatusActive — read 2026-10-09 (NOTE-610); worth reading as the 2025 journal extension of T2I-CompBench (LIT-783): 8,000 prompts in eight sub-categories, adding numeracy and 3D-spatial relations, eleven models up to SD3, DALL·E 3 and FLUX.1, and multimodal LLMs as judges. Two things change the conference picture. GPT-4V now agrees with human rankings better than CLIPScore on interactions (Kendall τ 0.48 against 0.25) and complex prompts (0.51 against 0.07), so interactions are no longer scored only by CLIPScore; but human scores on interactions are still near ceiling (0.95–0.99 for five of six rated models), and no new category asks about stance, purpose or social relation. Human ratings cover only the six 2023-era models; the 2024–25 models are ranked by automatic metrics alone.
Tagscompositionality, anthology-candidate
ExtendsLIT-783 — T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation
Compared againstLIT-770 — Compositional Visual Generation with Composable Diffusion Models, LIT-806 — Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models
SupportsCLAIM-321 — In text-to-image diffusion models each condition of a prompt can be met while their conjunction or relational binding fails: benchmarks find single objects rendered almost always and spatial relations and attribute binding failing most, and composing by adding scores does worst on relations
Cited as “paper” byNOTE-610 — T2I-CompBench++
Cited as “source” byTHEORY-522 — Open text-to-image diffusion models of 2022–2023 render a single named object almost always but fail most on relative position and next on attribute binding, by human rating and by detector checks alike; scaling the image model or continuing pretraining does not fix position, and interactions are rated near ceiling only by rubrics that ask whether an action is depicted, THEORY-678 — In Stable Diffusion, a subject named in a two-subject prompt is left out of the image when its token governs no region of the cross-attention map early in sampling, and raising that token's smoothed peak attention during the first half of sampling, by gradient steps on the latent with no retraining, brings the subject back; that this also fixes attribute binding is argued, not measured by the paper

Graph View

Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li and Xihui Liu (2025), IEEE Transactions on Pattern Analysis and Machine Intelligence 47(5):3563–3579 — DOI-10.1109/TPAMI.2025.3531907; also arXiv 2307.06350 v3

Key takeaways

  • What was added (Sections I, III; Table I). Two sub-categories: numeracy (1,000 prompts, one to eight objects of one or more kinds) and 3D-spatial relations (“in front of”, “behind”, “hidden by”), making eight sub-categories and 8,000 prompts, each split 700/300. A depth-plus- detector rule scores 3D relations, a detector count scores numeracy, and multimodal LLMs (MiniGPT-4, ShareGPT4V, GPT-4V) are tested as judges.
  • Judges against humans (Table XII). Disentangled BLIP-VQA stays best for colour and texture binding (τ 0.63, 0.52), and the UniDet rule for 2D spatial (0.48), 3D spatial (0.31) and numeracy (0.43); on shape, ShareGPT4V with chain-of-thought edges BLIP-VQA (0.29 against 0.27), and every judge is weak there. GPT-4V is best on interactions (0.48; CLIPScore 0.25) and on complex prompts (0.51; the 3-in-1 metric 0.28, CLIPScore 0.07). GPT-4V was run on a fifth of the images.
  • What is hard (Tables VIII–XI, human ratings of six models). Spatial relations hardest, then shape binding; non-spatial relations easiest, at 0.95–0.99 for five of six models (Composable Diffusion 0.81).
  • The models (Table XIII). DALL·E 3, SD3 and FLUX.1 lead on most sub-categories; SD3 has the best colour binding (B-VQA 0.81) and 2D spatial score (0.32, out of 1). GORS, the paper’s reward-weighted finetuning of SD v2, beats its own base everywhere; the 2024–25 models are not rated by humans.
  • Unchanged from the conference version: GORS’s binding score still falls on adjective–noun pairs absent from its finetuning prompts (B-VQA 0.72→0.54 colour, 0.55→0.34 shape, 0.76→0.36 texture, Table XV), and the text still calls it “slightly lower”.

Standing in the record

Filed on 2026-10-09 from the manuscript bibliography of 2026-10-09 (work what-survives-translation) at the owner’s request: the manuscript considered it and dropped it from the final reference list. It is read here on its own merits. See the curation entry of that day.

It is the journal version of LIT-783 and extends it. The record holds them as two works because they are two publications with different contents; the conference paper is the one the manuscript cites, and its reading (NOTE-592) is not superseded by this one. Like LIT-783 it is a machine-learning benchmark an anthology topic could hold, so it carries the anthology-candidate flag (ADR-005), and it is filed under compositionality, the subject it shares with this record.

After reading. Its interaction category is now judged by GPT-4V rather than CLIPScore, which matters to the manuscript’s argument about what the benchmark can and cannot tell apart; the reading (NOTE-610) sets out what that does and does not change.

Backlinks