Source
Shen, Mishra and Khashabi (2023), LIT-536 — ICML 2024. Read as NOTE-276.
The practice
When the claim is pretraining on a general objective produces a model that
does X, the model you measure must be one trained on that general
objective. If instead you train on sequences drawn from the task family that
X names, you have changed two things at once — the space of tasks and, with
it, the inductive bias of the resulting model — and a positive result is
compatible with the claim being false.
State which of the two you are testing:
- the expressivity claim — there exist weights for which the model does
X— which a construction or a purpose-trained model can establish, and which says nothing about what training on the general objective yields; - the emergence claim — weights arising from the general objective do
X— which needs a model that was never shown the task family.
Both are worth establishing. Only the second is usually what a reader takes away, and it is the one that almost never gets tested.
The practice originates in LIT-536, which drew this line for in-context learning as its Hypothesis 2 (weights exist that make ICL equivalent to gradient descent) against Hypothesis 1 (weights from pretraining do), and showed the supporting experiments had trained on the very task family they tested, so had only ever addressed the first. Tested instead on LLaMA-7B, a model trained on the general objective, in-context learning and gradient descent on the same demonstrations disagreed on accuracy, top-10 token overlap and overlap cosine similarity across AGNews, CB, SST-2 and RTE.
Why it is Proposed
One source, and the discipline is general while the evidence is one
dispute. The argument is clean and it is supported by measurements — a
sparsity requirement the constructions need and real models do not have, and
an accuracy-flat/parameters-moving checkpoint sweep showing the capability
belongs to a family of weights rather than a point. But it has been applied
once, to in-context learning, and a methodological practice earns Active
by working somewhere its author was not looking. The promote_when asks for
exactly that and not for a second paper agreeing with this one, which would
not add evidence (DP-005).
Scope
This is not an argument against training a model on a task family to study it. LIT-534’s setup is how the mechanism question became answerable at all, and the results in it are real results about transformers. The practice is about what a result in that setup licenses you to say about a model pretrained on text.
Nor is it specific to language models or to in-context learning. The shape — capability observed in the wild, reproduced in a purpose-built setting, then explained from the purpose-built setting — is common, and each step is reasonable while the composition quietly substitutes one model family for another.
How to tell whether it bit you
The test that made it visible here was not about the objective at all: it was Theorem 1’s observation that in-context learning is sensitive to the order of its demonstrations and batch gradient descent is not, so no argument about weights was needed to know the two cannot be equivalent. Look for a property the claimed mechanism must have, check whether the real phenomenon has it, and do that before building the apparatus that would confirm the mechanism. That half of the method is filed separately as SOTA-324.