nucleation

LIT-833: Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

2 min read

Properties5
StatusActive — read 2026-10-09 (NOTE-626); worth reading as the measurement that the order of the same few-shot examples, with content held fixed, can move a language model from near chance to near the fine-tuned state of the art, at every size from GPT-2 Small to GPT-3 175B. Good orders do not transfer between model sizes (rank correlations near zero), more examples do not remove the variance, and failing orders mostly collapse the predicted label distribution. The rest of the paper is a selection method (entropy over a self-generated probing set) whose interest here is secondary.
Tagsrepresentation-learning, anthology-candidate
SupportsCLAIM-248 — The founding stimuli (Villon, Henley's Villon, Hofstadter's Marot) and commentary on them are likely in language models' training data, and what a context does to one model need not be what it does to another, so a model's reading of their footing may be recall, and a directional match in one model may not hold in the next
Cited as “paper” byNOTE-626 — Fantastically Ordered Prompts
Cited as “source” byTHEORY-683 — Permuting a fixed set of few-shot examples moves a GPT-family model's classification accuracy between near chance and near the supervised state of the art, at every model size tested; the good orders are specific to the model, the spread survives more examples and calibration, and bad orders mostly fail by predicting one label

Graph View

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel and Pontus Stenetorp (2021), ACL 2022 — ARXIV-2104.08786, DOI-10.18653/v1/2022.acl-long.556

Key takeaways

  • Order alone, content fixed. For a fixed balanced set of four SST-2 examples, all 24 permutations as prompts: some reach accuracy comparable to supervised fine-tuning, others sit near the 50% baseline, in every GPT-2 and GPT-3 size tested (Figure 1). Larger models reduce the spread on SST-2 but not on Subj.
  • Not a small-sample artefact. Up to 32 examples (sampling at most 24 orders), accuracy rises but the variance stays high and can grow.
  • Not transferable. Pairwise Spearman correlations of permutation performance between model sizes are near zero (175B vs 2.7B: 0.05), and the same holds for the six label patterns. One permutation falls from 88.7% to 51.6% moving from GPT-2 XL to GPT-2 Large.
  • How bad orders fail. Mostly by highly unbalanced predicted label distributions; calibration (Zhao et al. 2021) raises accuracy but leaves the variance high.
  • Selection without labels. Generate a probing set by sampling from the model under each candidate order, then score orders by the entropy of predicted labels over it (GlobalE) or per item (LocalE). GlobalE gives an average 13% relative improvement over random orders across eleven tasks and sizes, and beats splitting the four training examples into train and dev.

Standing in the record

Filed on 2026-10-09 at the owner’s request, from the bibliography of the owner’s manuscript What Survives Translation? (work what-survives-translation), which considered it and dropped it from the final reference list. Read on its own terms (NOTE-626).

It carries anthology-candidate: its subject, prompt design for few-shot learning, is one an anthology topic holds, and its method is an instruction for practice. The primary tag is the nearest word the closed vocabulary offers, and the fit is weak. What it establishes that this record can use is the measurement of order sensitivity itself.

Backlinks