Kirstain, Polyak, Singer, Matiana, Penna and Levy, Tel Aviv University and Stability AI (2023) — ARXIV-2305.01569. NeurIPS 2023. Read at v2 (23 Nov 2023), the camera-ready, appendix included. v1 (2 May 2023) was checked for what changed: it reports PickScore’s accuracy as 70.2 from one run, where v2 gives 70.5 ± 0.142 over three seeds. v2 also adds a footnote that a second release, Pick-a-Pic v2, holds over a million examples.
Key takeaways
- The data comes from people generating for themselves (§2). Users write a prompt, see two images, and pick one or declare a tie. The loser is replaced and the round repeats. Two images with ties beat four images, and two without ties, on engagement and inter-rater agreement in a pilot. The images come from Stable Diffusion 2.1, Dreamlike Photoreal 2.0 (a fine-tune of SD 1.5) and SDXL variants, at varied guidance scales. Users log in with Gmail or Discord, are capped at 1,000 interactions, and are banned for NSFW prompts or judging too fast.
- The splits are by prompt and by user (§2). Validation and test take 500 prompts each, one example per prompt, each prompt from a different user. Training gets every example whose prompt is not in them. The experiments use an NSFW-filtered snapshot: 583,747 training examples, 37,523 prompts, 4,375 users. 968,965 rankings had been collected by then.
- PickScore is a CLIP model trained as a reward model (§3, Eqs. 1–3). The score is CLIP’s scaled inner product of text and image embeddings. The loss is the KL from the user’s choice, [1, 0], [0, 1] or [0.5, 0.5] for a tie, to the softmax over the two images’ scores. That is the InstructGPT reward-model objective with ties added. Each example is weighted by the inverse of its prompt’s frequency, so heavy prompts do not dominate. CLIP-H is fine-tuned for 4,000 steps at learning rate 3·10⁻⁶ and batch 128, in under an hour on 8 A100s, with no hyperparameter search. The checkpoint is chosen by validation accuracy. Using the rest of the batch as negatives, as CLIP’s own loss does, gives 65.2 instead (Appendix).
- It predicts users’ choices better than the alternatives (§4, Table 1b). The metric scores a match 1, a tie on one side only 0.5, and a miss 0. Each model’s tie threshold is tuned on validation. On the test set: random 56.8, aesthetics predictor 56.8, CLIP-H 60.8, ImageReward 61.1, HPS 66.7, expert humans 68.0, PickScore 70.5 ± 0.142.
- As an evaluation metric it tracks people better than FID or CLIP (§5). Real users’ 14,000 preferences on test prompts give Elo ratings for 45 configurations, four backbones at several guidance scales. Rated by each metric instead, Spearman correlation with the users’ ratings is 0.790 ± 0.054 for PickScore, 0.670 for HPS, 0.492 for ImageReward and 0.313 for CLIP-H (Fig. 7, 50 shuffles of the Elo order). On 100 MS-COCO captions and nine configurations (SD 1.5, SD 2.1 and Dreamlike, each at guidance 3, 6 and 9), expert rankings correlate 0.917 with PickScore and −0.900 with FID (Fig. 6).
- People prefer higher guidance, and FID penalizes it (§2 Fig. 3a, §5 Fig. 5). Users of SDXL-alpha usually prefer images at guidance 9 to those at 3. The authors read FID’s negative correlation as this: high guidance gives vivid images people like and that sit further from MS-COCO’s photographs. They recommend Pick-a-Pic prompts over MS-COCO captions, which describe everyday photographs rather than what users ask for.
- Best of 100 by PickScore is preferred (§6, Table 2). For 100 test prompts, Dreamlike Photoreal 2.0 generates 100 images each, five seeds by twenty prompt templates. Expert annotators prefer PickScore’s pick to the aesthetics predictor’s 85.1% of the time, to CLIP-H’s 71.3%, to a random image from the untemplated prompt 71.4%, and to a random image overall 82.0%. PickScore’s pick also has the higher aesthetic score 68.5% of the time against CLIP-H’s pick, and the higher CLIP-H score 90.5% of the time against the aesthetic predictor’s.
Where the hedges are
Per DP-010:
- “Superhuman” is against outsiders. The expert annotators are “friends and colleagues of the authors” (footnote 9). The paper’s own point is that they do not know what the user meant. So 70.5 against 68.0 says the model predicts this user population better than an outsider does. It does not say the model judges images better than people.
- The headroom is small. Chance under the tie-aware metric is 56.8, so PickScore is 13.7 points above it and humans 11.2. The aesthetics predictor scores exactly chance.
- Every agreement with “real users” is in-distribution. The test set, the 14,000 Elo preferences and the best-of-100 prompts all come from Pick-a-Pic’s own users and prompt distribution, which PickScore was trained on. HPS and ImageReward are scored on PickScore’s home ground, and the paper names data scale, distribution and implementation as confounds it did not separate (§7).
- The FID comparison is nine points. Three models at three guidance scales, 100 captions, ranked by colleague annotators. FID ranks a whole configuration and PickScore ranks per image. The correlations are between model win ratios, not per-image agreement.
- It is validated as a judge and a ranker, not as a target. The strongest use shown is picking one of 100 samples. The conclusion proposes it for RLHF, and that is how it has since been used. Nothing here tests what happens when a model is optimized against it at length. Best-of-100 moves only log n − (n − 1)/n, about 3.6 nats, from the base sampler. The reward-overoptimization study (LIT-795) finds that true reward under a learned scorer rises and then falls as optimization continues, and that RL spends far more KL than best-of-n for the same gain. A ranking result does not say where that turn comes for RL.
- Taste, not correctness. In disagreements with CLIP-H, PickScore “often chooses more aesthetically pleasing images”, “at times, at the cost of faithfulness to the prompt” (§4, Fig. 4). The dataset also contains NSFW prompts and images the filter missed, which the authors acknowledge (§2, §8).
- One run of everything but PickScore’s training. The ranking and evaluation studies are single runs. The three seeds are for the scoring function only.
Which comparisons are like for like
- PickScore against CLIP-H, the aesthetics predictor and humans (Table 1b) share the test set, the metric and per-model tie thresholds. Humans choose ties directly rather than through a tuned threshold.
- HPS and ImageReward are other groups’ models trained on other data, evaluated on this paper’s test set. Not like for like, and the paper says why.
- PickScore against FID (Fig. 6) compares a per-image, prompt-aware score with a set-level, prompt-blind distance. The setting is MS-COCO, which the authors chose as “convenient” for FID.
- Best-of-100 (Table 2) compares selectors on the same 100 images per prompt. The baselines include random selection, so part of the margin is just that any selector beats chance.
Standing in the anthology
It extends CLIP (LIT-588). PickScore keeps CLIP’s two-tower architecture and its scaled inner-product score (Eq. 1) and starts from the OpenCLIP CLIP-H checkpoint. What it changes is the training signal: one image of a pair preferred over the other, the reward-model objective of InstructGPT (LIT-377), instead of matching captions to images. Its zero-shot CLIP-H baseline is the starting checkpoint. Fine-tuning on user preferences takes it from 60.8 to 70.5, and CLIP-H ranks the 45 configurations at 0.313 against PickScore’s 0.790.
It measured itself against FID (LIT-611) as a way to rank text-to-image models, and found FID anti-correlated with expert preference across nine configurations. The reason it gives is classifier-free guidance (LIT-693): higher guidance is preferred and moves samples away from MS-COCO’s statistics. That is the trade SOTA-424 says to choose deliberately, measured here on human preference rather than on IS.
Two later papers in the record depend on it, in different ways. Diffusion-DPO (LIT-774) trains offline on the Pick-a-Pic v2 pairs, 851,293 of them. It picks checkpoints and sweeps β by median PickScore on 500 validation prompts, and confirms its headline with crowdworkers. Its own description of the data, images “generated by SDXL-beta and Dreamlike”, leaves out the SD 2.1 images this paper lists for the first release. Flow-GRPO (LIT-779) uses PickScore as the reward itself and reports it rising from 21.72 to 23.31. With the KL term removed, its outputs collapse to one style while PickScore and the other reward models stay high. That is the use this paper’s evidence covers least. SOTA-442’s evidence on that task is PickScore-judged as well.
Filed without a NOTE. The takeaways come from one full reading of v2,
appendix included, done for this filing. Figure values are quoted only
where the text or a figure label gives them.