Contribution
Before it, prompt sensitivity had been studied for templates and for which examples to include. This paper isolates the order of a fixed set of examples, shows its effect is as large as a template’s, establishes that it is present across model sizes, tasks, templates and numbers of examples, and that the good orders are idiosyncratic to a model. It then gives a label-free procedure to pick orders.
Key insight
The same evidence, presented in a different sequence, is a different context for the model, and the difference is not small or systematic: a causal LM’s prediction depends on the order of its examples enough to decide whether it performs the task at all, and which orders work is a property of the particular model, not of the examples.
Assumptions
- Text classification (eleven datasets, from sentiment to entailment), answered by the most probable label word after a template.
- k = 4 examples (1 for DBPedia, 2 for AGNews) because of GPT-2’s 1024-token window; five random example sets per task, all 24 orders (two sets and 12 orders for GPT-3 175B); evaluation on 256 sampled validation items.
- The probing set is generated at temperature 2 with n-gram blocking, up to 128 tokens per generation; its generated labels are discarded.
- “True few-shot”: no labelled development set is used for selection.
Key results
- Figure 1. SST-2 and Subj, four-shot, 24 orders, eight model sizes: some orders above 85% (SST-2), others near 50%; spread shrinks with size on SST-2, not on Subj.
- Figure 3. Variance remains high from 1 to 32 examples (GPT-2).
- Figure 4. Spearman correlations of permutation accuracy between sizes lie between −0.35 and 0.27 off the diagonal; 175B vs 2.7B is 0.05.
- Figure 5. For the six label patterns (NNPP … PPNN), correlations range from −0.66 to 0.77: no pattern is consistently good.
- Figure 6. Failing prompts mostly predict one label; calibration raises 2-shot GPT-2 XL accuracy but the spread of orders stays.
- Table 2. GlobalE and LocalE select top-4 orders by entropy; GlobalE averages 13% relative improvement over all orders, LocalE 9.6%, with lower variance, and an oracle top-4 above both. Gains are large for high-variance models and marginal for GPT-3 175B, except CB (+4.9).
- Table 3. The method helps under four SST-2 templates.
- Table 5. Entropy probing beats selecting orders on half of the four training examples held out as dev, at every GPT-2 size.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Order of a fixed set of few-shot examples can move accuracy between near chance and near state of the art | strong | Figure 1, Table 2 baseline variances, many tasks and sizes |
| C2 | The sensitivity persists with more examples | moderate | Figure 3, GPT-2 only, at most 24 orders sampled |
| C3 | Good orders do not transfer across model sizes | moderate | Figures 4–5, one example set on SST-2 |
| C4 | Bad orders mostly fail by collapsing onto one label | moderate | Figure 6 error analysis |
| C5 | Entropy on a self-generated probing set identifies better orders without labels | moderate | Tables 2, 3, 5 |
| C6 | Order sensitivity is a fundamental property of in-context learning | weak | the authors’ conclusion from C1–C2, not tested beyond causal GPT models and classification |
Method
For a set of n training examples, form all n! orders (24 here). Under each order as context, sample a continuation and parse it into probe inputs (discarding the generated labels); pool these into a probing set. For each order, predict labels for all probe inputs and score the order by the entropy of the predicted label distribution (GlobalE) or the mean per-item prediction entropy (LocalE). Keep the top four orders.
Concepts
- order sensitivity: variation of accuracy across permutations of the same examples.
- probing set: an unlabelled artificial development set generated by the model from the candidate prompts.
- GlobalE / LocalE: entropy of the predicted-label histogram over the probing set / mean entropy of each probe’s predictive distribution.
Connections
Works in the frame of GPT-3’s few-shot prompting. Against Liu et al. (2021), who retrieve nearest-neighbour examples from a full training set and find order barely matters, it finds the opposite in the true few-shot setting. Zhao et al. (2021) is the calibration it tests. Xie et al. (LIT-797) reproduce a 10–40 point order effect in GINC, where it follows from the prompt’s mismatch with pretraining.
Bearing on the record
- It establishes, as a measurement, that a model’s response to a context is not invariant to the order of the context’s parts, even when the parts and the task are unchanged, and that which orders work is specific to a model. No THEORY in the record holds this; I file none, because it is a single robust measurement better cited from the LIT than restated.
- The paper’s method is an instruction for practice; it belongs to the
anthology’s question, not this record’s (
anthology-candidate).
Limitations
- Classification only, small k, causal GPT-family models.
- The transfer analysis (Figures 4–5) rests on one fixed set of four SST-2 examples.
- The text cites “Figure 5” for the comparison with split training data, which is Table 5.
- The method scales as n!; it is shown only for n ≤ 4 with all permutations.
- No account of why orders differ; the label-collapse observation describes the failure, not its cause.
Open questions
- What property of an order predicts its effect for a given model? The paper finds no common denominator.
- Does the sensitivity survive instruction tuning and models trained with in-context objectives?