Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li and Xihui Liu (2023), NeurIPS 2023 Datasets and Benchmarks — ARXIV-2307.06350
Key takeaways
Read in the conference version (arXiv v2), not the current T2I-CompBench++.
- Six kinds of composition, 1,000 prompts each (Section 3). Colour, shape and texture binding; spatial relations (seven phrases); non-spatial relations (interactions); complex prompts with more objects or mixed attributes. Each sub-category is split 700 for training and 300 for testing, and the binding test sets are split again by whether the adjective–noun pair occurs in training. Most prompts are templated or written by ChatGPT from given word lists.
- A judge per kind (Sections 4, 6.4; Table 5). Disentangled BLIP-VQA, one question per object–attribute pair, ranks images like human raters (Kendall τ 0.63 on colour, 0.52 on texture, 0.27 on shape) where CLIPScore does poorly (0.19, 0.29, 0.06). A UniDet box rule does the same for spatial relations (0.48 against 0.27). Nothing beat CLIPScore on interactions (0.25). MiniGPT-4 as a judge is near zero on spatial and complex prompts, better with chain-of-thought prompting, never best.
- What is hard (Tables 2–4, human ratings). Spatial relations are hardest (0.31–0.46 across models), then shape binding; interactions are near ceiling (0.81–0.99). SD v2 beats SD v1-4 everywhere; methods built for binding help binding and not relations.
- GORS (Section 5). Finetune SD v2 with LoRA on its own generated images that score above a threshold, weighting the loss by the score. It is rated best by humans in every sub-category, and a variant selecting with rewards different from the evaluation metrics does about as well. On attribute–noun pairs absent from its 700 finetuning prompts, its BLIP-VQA score falls from 0.72 to 0.54 (colour), 0.55 to 0.34 (shape) and 0.76 to 0.36 (texture) (Table 12), which the text calls “slightly lower”.
The record’s account of compositional generalization is THEORY-117 (LIT-667), about classifiers trained on controlled concept grids. Table 12’s gap points the same way but does not test it: the unseen pairs are also rarer, and no model is reported on the split before finetuning (NOTE-592). Its companion benchmark in the same track is GenEval (LIT-782). Among its baselines is Composable Diffusion (LIT-770), re-implemented on SD v2, which does worst on most categories.
Standing in the record
Filed on 2026-10-09 at the owner’s request, as one of the works in the
reference list of the owner’s working manuscript (October 2026) that the
record did not yet hold. See the curation entry of that day. It is a
machine-learning paper an anthology topic could hold (generative-modeling,
analysis-and-evaluation), so it carries the anthology-candidate flag
(ADR-005); it is here because the owner asked for
the manuscript’s references to be filed in this record.
After reading. It is a benchmark, a set of evaluation metrics and a
finetuning method for text-to-image models; all three are
machine-learning practice, anthology material. The flag stays. It was first filed
under representation-learning, which the reading did not support: the
paper does not study how a model represents its data, only whether its
outputs match the prompt. It is filed under compositionality (ADR-031),
the subject it shares with this record; the evaluation of generative
models is an anthology subject.