Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
In-context learning — getting behaviour out of a fixed model by what you put in the context — few-shot exemplars, chain of thought, prompting strategy, and what the context can and cannot buy.
13 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-037 | LM in-context learning emerges at scale | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-038 v3 | ICL permits few-shot task adaptability | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-127 v2 | Filter chain-of-thought traces out of the training data of tiny specialized models | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Reasoning traces interleaved with tool-calling data sent a 90M model into repetition loops; removing them fixed it at once. | Active |
| SOTA-279 v2 | Put worked reasoning steps in the few-shot exemplars when the task needs more than one step, and only once the model is large enough | Wei et al. (2022), LIT-467 — eight exemplars carrying worked steps more than doubled PaLM 540B on GSM8K, beating a fine-tuned model with a verifier. Three ablations rule out the equation, the extra tokens and knowledge activation. Below ~100B it does nothing or hurts. | Active |
| SOTA-280 | Sample several reasoning paths and take the majority answer rather than decoding one greedily | Wang et al. (2022), LIT-468 — replace greedy decoding with sample-and-marginalize: GSM8K +17.9, SVAMP +11.0, AQuA +12.2. Five or ten paths recover most of it, which is the whole cost question. | Active |
| SOTA-281 | Try the single step-by-step instruction before writing exemplars; it recovers most of the gain for none of the labour | Kojima et al. (2022), LIT-469 — one fixed sentence with no exemplars takes MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. It beats eight-shot standard prompting and loses to hand-written few-shot chains, which is the ordering that makes it a first move. | Active |
| SOTA-300 | To tell whether a model is doing non-trivial in-context computation, measure how well it predicts its own hidden states, not its next-token loss | Herrmann et al. (2025), LIT-489 — uniform noise maximizes next-token loss and requires no computation; reciting a memorized licence minimizes it and requires none either. Insert a variational bottleneck with a learned autoregressive prior and measure the posterior-prior KL: only the genuinely in-context task scores high, on a Transformer and an LSTM, and the signal survives controlling for next-token loss. | Proposed |
| SOTA-304 | Generate a long program unit by unit, freezing each unit that passes an execution-based check, instead of regenerating the whole artifact on failure | Cheng, Clark and Richardson (2025), LIT-493 — on 100 held-out proposals, the same model produced valid implementations 92% of the time generating unit by unit against an execution-based checker, and 6% of the time one-shot with retries; the one-shot outputs were also trivial, 49 lines against 181. Removing the checker alone costs 62 points, removing the decomposition alone costs 19 — so if you build one, build the checker. | Proposed |
| SOTA-308 | When a retrieval change raises recall, measure relevance and the end-task metric before calling it an improvement | Wu et al. (2026), LIT-502 — across three published GraphRAG pipelines on G-Medical, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops context relevance (38.5% against 62.9%). The wider context is noisier, and generation accuracy falls with it — a system that retrieves more of the right thing and much more of the wrong thing answers worse. | Active |
| SOTA-313 | Treat the inference-time decision procedure as part of what you are comparing, and report the ordering under each one | Gemini Team (2023), LIT-510 — on MMLU, greedy sampling puts GPT-4 ahead by 0.2, plain chain-of-thought at 32 samples puts it ahead by 2.3, and uncertainty-routed CoT at 32 samples puts Gemini Ultra ahead by 2.7. Same models, same benchmark, same harness. The procedure is worth 6.0 points to one model and 3.1 to the other, so running the same procedure on both is not enough — the ordering is a property of the procedure you chose to report. | Active |
| SOTA-323 | Test a claim about what pretraining produces on a model trained with the pretraining objective, not one trained on the task family you are testing | Shen, Mishra and Khashabi (2023), LIT-536 — “there exist weights such that” and “pretraining produces weights such that” are different claims, and training a model on the task family you are about to test collapses them. Their separation of these as Hypothesis 2 and Hypothesis 1 is what let them show that a literature everyone read as evidence about pretrained language models had tested neither the models nor the objective. | Proposed |
| SOTA-324 | Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output | Fu, Chen, Jia and Sharan (2023), LIT-535 — output similarity cannot tell gradient descent from Newton’s method, because both converge to the same answer. Matching each layer to the best-fitting step count of each candidate gives a linear trend for the right rate and an exponential one for the wrong one; a regime where the rivals must differ — here, condition number 100 — separates them again. Two measurements, both cheap, both decisive where a fit was not. | Proposed |
| SOTA-357 | Build a zero-shot classifier from prompt templates and ensemble them in embedding space | Active |