Garg, Tsipras, Liang and Valiant (2022) — ARXIV-2208.01066, NeurIPS 2022. Read as NOTE-279.
Key takeaways
- A transformer can be trained from scratch to in-context learn a function
class. 12 layers, 8 heads, 256-dimensional embeddings, 9.5M parameters,
500k steps at batch 64, trained only on sampled
(x, f(x))sequences — no text, no pretrained initialization. - On linear functions in
d = 20it matches ordinary least squares, the optimal estimator for the problem, and keeps doing so under two kinds of distribution shift: between training prompts and inference prompts, and between the in-context examples and the query. - On harder classes it matches the algorithm built for each. 3-sparse
linear functions: 0.58 and 0.09 squared error at
k = 5andk = 10against Lasso’s 0.62 and 0.08 — and Lasso has no closed form, so the transformer is matching an iterative minimization in one forward pass. Also two-layer ReLU networks and depth-4 decision trees. - Curriculum learning is load-bearing for training cost, starting on functions restricted to a low-dimensional subspace.
Standing in the anthology
It is the substrate the mechanism papers are built on. Every later claim in this cluster — ARXIV-2212.07677, ARXIV-2211.15661, ARXIV-2310.17086 — trains in this setup, on this task, in most cases on this exact architecture, and measures against these baselines. The record held the claims’ descendants (SOTA-037, SOTA-038) and none of the substrate.
It also supplies the objection to itself. The model here is trained on the ICL objective — the same restricted task family it is then tested on — which is precisely the gap ARXIV-2310.08540 later argues makes these results say little about in-context learning in a model pretrained on text. That is an argument about what the setup licenses, not a defect in the setup, and the paper is careful about what it claims.