Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
In-context learning — getting behaviour out of a fixed model by what you put in the context — few-shot exemplars, chain of thought, prompting strategy, and what the context can and cannot buy.
25 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-035 v2 | Language Models are Few-Shot Learners | Brown et al. (2020), ARXIV-2005.14165. Introduces GPT-3. | Active |
| LIT-049 v3 | Learning to Prompt for Vision-Language Models | Zhou et al. (2021), ARXIV-2109.01134. Learnable prompt optimization. | Active |
| LIT-129 | Can Tiny Language Models Reason? | Jakimovski (2025), a write-up with code, no paper. SmolLM2-135M-Instruct taken through SFT on plain dialogue, SFT on reasoning traces in think tags, and one epoch of DPO on 50k preference pairs, on a single GPU; the DPO stage converges and the final model gains 3 to 9 points over its base on ARC-C, BBH, MMLU and IFEval. | Active |
| LIT-379 v2 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Akyürek et al. (2024), ARXIV-2411.07279. Temporarily update the model’s parameters at inference, using a loss built from the test instance’s own in-context examples. On ARC-AGI this is up to 6x a fine-tuned baseline — 53.0% with an 8B model, 61.9% ensembled with program synthesis, which the paper puts at average human performance. Read as evidence about in-context learning’s limits as much as about the technique. | Active |
| LIT-420 | Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them | Suzgun et al. (2022), ARXIV-2210.09261. The 23 BIG-bench tasks where models had not yet beaten the average human rater, split out as BBH — a suite selected by a model generation’s shortfall, and one whose headline finding is that the shortfall was partly a prompting artifact. | Active |
| LIT-464 | Theoretical limitations of multi-layer Transformer | Chen et al. (2024), ARXIV-2412.02975. The first unconditional lower bound for a decoder-only transformer with more than one layer: any constant-depth decoder needs polynomially many parameters to compose functions sequentially. Depth, an encoder, or chain of thought each buy what width cannot. | Active |
| LIT-467 | Chain-of-Thought Prompting Elicits Reasoning in Large Language Models | Wei et al. (2022), ARXIV-2201.11903. Put worked reasoning steps in the few-shot exemplars and a 540B model more than doubles on GSM8K. Three ablations rule out the obvious alternative explanations, and the effect is absent — sometimes negative — below about 100B parameters. | Active |
| LIT-468 | Self-Consistency Improves Chain of Thought Reasoning in Language Models | Wang et al. (2022), ARXIV-2203.11171. Replace greedy decoding with sample-and-marginalize: draw several reasoning paths, take the most common final answer. GSM8K +17.9, SVAMP +11.0, AQuA +12.2. The gain saturates by about five or ten samples. | Active |
| LIT-469 | Large Language Models are Zero-Shot Reasoners | Kojima et al. (2022), ARXIV-2205.11916. One fixed sentence — “Let’s think step by step” — with no exemplars at all takes MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. It beats eight-shot standard prompting and still loses to hand-written few-shot chains. | Active |
| LIT-470 | Emergent Abilities of Large Language Models | Wei et al. (2022), ARXIV-2206.07682. Names and defines emergent abilities — present in larger models, absent in smaller ones, and therefore not predictable by extrapolating a scaling law. A survey of existing curves rather than new experiments: eight few-shot examples across five model families, plus prompting strategies that only help past a threshold. Section 5.1 raises the metric explanation itself and argues against it on two specific grounds. | Active |
| LIT-489 | Measuring In-Context Computation Complexity via Hidden State Prediction | Herrmann, Csordás and Schmidhuber (2025), ARXIV-2503.13431. Next-token loss cannot tell hard from interesting: random noise scores high and requires no computation, memorized recitation scores low. Insert a variational bottleneck with a learned autoregressive prior mid-network and measure the KL between posterior and prior — the nats of hidden-state information the past did not predict. That quantity separates in-context learning from memorization and noise, and tracks formal-language description length and maths difficulty after controlling for next-token loss. | Active |
| LIT-493 | Language Modeling by Language Models | Cheng, Clark and Richardson (2025), ARXIV-2506.20249. An LLM agent system that proposed, implemented and pretrained 1,062 verified language-model block designs at 14M–350M parameters. The result that survives its own controls is the code generation: decomposing a long program into units and freezing each one that passes an execution-based checker takes valid-design rate from 6% to 92%. The architecture-discovery headline does not survive as well — the five best designs average 0.6 points below the five human seeds they were bred from. | Active |
| LIT-502 | MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation | Wu, Xiang, Tang, Chen, Zhang and Su (2026), ARXIV-2606.00610. A multi-agent GraphRAG system, filed for its pilot study rather than its system: across three published GraphRAG pipelines, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops relevance (38.5% against 62.9%), and end-task accuracy falls with it. Their own filtering probe finds 40% of extracted triples can be deleted without cost. | Active |
| LIT-510 | Gemini: A Family of Highly Capable Multimodal Models | Gemini Team, Google (2023), ARXIV-2312.11805 — held for its evaluation methodology rather than its models. Its appendices publish three things the record wanted: an MMLU sweep in which the ordering against GPT-4 flips depending on the inference procedure; a measurement of how cheaply a benchmark can be inflated (100 fine-tuning steps on HellaSwag-adjacent web extracts take Ultra to 96.0% at 1-shot); and a FLEURS ablation retraining without the benchmark’s own training set. Read as NOTE-255. | Active |
| LIT-532 | What learning algorithm is in-context learning? Investigations with linear models | Akyürek, Schuurmans, Andreas, Ma and Zhou (2022), ARXIV-2211.15661, ICLR 2023 — the independent construction, and the finding the title undersells. A transformer needs constant depth and O(d) hidden size for a gradient step and O(d²) for a Sherman–Morrison ridge update; but the trained model’s behaviour moves through three algorithmic phases as depth grows — gradient descent, then ridge, then ordinary least squares — and at every noise level the ridge parameter that best fits it is the one that minimizes Bayes risk. Read as NOTE-277. | Active |
| LIT-533 | Transformers learn in-context by gradient descent | von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov and Vladymyrov (2022), ARXIV-2212.07677, ICML 2023 — the paper that named the mesa-optimizer. Proposition 1 constructs key, query, value and projection matrices for which one linear self-attention layer is exactly one gradient-descent step on an implicit linear model, and a single trained layer is shown to find approximately those weights. Deeper trained models do not match plain gradient descent; they match GD++, gradient descent with an iterative curvature correction — a result of this paper’s own. Read as NOTE-280. | Active |
| LIT-534 | What Can Transformers Learn In-Context? A Case Study of Simple Function Classes | Garg, Tsipras, Liang and Valiant (2022), ARXIV-2208.01066, NeurIPS 2022 — the setup that turned “how does in-context learning work” into a question with a checkable answer. A 9.5M-parameter GPT-2 trained from scratch on sequences of (x, f(x)) pairs in-context-learns unseen linear functions at the accuracy of the optimal least-squares estimator, and sparse linear functions, two-layer ReLU networks and depth-4 decision trees at or above their task-specific algorithms. Read as NOTE-279. | Active |
| LIT-535 | Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression | Fu, Chen, Jia and Sharan (2023), ARXIV-2310.17086, NeurIPS 2024 — the challenge from inside the same setup. In Garg et al.’s exact training regime, successive transformer layers track Iterative Newton linearly (roughly 3 iterations per middle layer) and gradient descent only exponentially; on data with condition number 100 the transformer is unaffected while GD needs 2,000 steps, which twelve layers cannot hold. Theorem 5.1 builds k Newton iterations in k + 8 layers with O(d) width. Read as NOTE-278. | Active |
| LIT-536 | Do pretrained Transformers Learn In-Context by Gradient Descent? | Shen, Mishra and Khashabi (2023), ARXIV-2310.08540, ICML 2024 — the challenge from outside. It separates two hypotheses the field had been conflating: that ICL in a naturally pretrained model is gradient descent, and that transformer weights exist that simulate it. Theorem 1 says an algorithm equivalent to ICL must share its order sensitivity, and gradient descent is order-stable while ICL is not; on LLaMA-7B, ICL and fine-tuning disagree on accuracy, top-10 token overlap and overlap cosine similarity across four datasets. Read as NOTE-276. | Active |
| LIT-543 | Loss Landscape Degeneracy and Stagewise Development in Transformers | Hoogland, Wang, Farrugia-Roberts, Carroll, Wei and Murfet (2024), ARXIV-2402.02364, TMLR 2025 — the instrument applied. Tracking the LLC through training divides two transformers into five stages each, and the boundaries coincide with structural changes the loss curve does not show: bigrams, then n-grams, then previous-token heads, then the induction circuit. In the in-context regression transformer, in-context learning is acquired in LR2 and then deteriorates across LR3 and LR4 as the model specializes to its pre-training distribution. Read as NOTE-287. | Active |
| LIT-577 | Evaluating the Ripple Effects of Knowledge Editing in Language Models | Cohen, Biran, Yoran, Globerson, Geva (2023), ARXIV-2307.12976. An edit implies other facts: logical consequences, multi-hop compositions, aliases. RippleEdits tests 5K edits on six such criteria. ROME, MEMIT and MEND average 38–66 across GPT-2, GPT-J, GPT-NeoX and LLaMA. Logical generalization can be as low as 5.5–7.0. Stating the new fact in the prompt (in-context editing) scores best, beating ROME by more than 10 points on GPT-NeoX and by 29 on LLaMA. | Active |
| LIT-733 | Language Models are Unsupervised Multitask Learners | Radford et al. (2019) — GPT-2. Language models trained on WebText begin to perform tasks zero-shot when conditioned on a suitable context, and the performance rises log-linearly with capacity up to the 1.5B model, which still underfits. Its §2.3 carries two architecture changes the field kept: layer norm moved to the input of each sub-block, and residual-layer weights scaled at initialization by 1/sqrt(N), N the number of residual layers, to account for accumulation on the residual path with depth. | Active |
| LIT-740 | Role-Play with Large Language Models | Shanahan, McDonell and Reynolds (2023), ARXIV-2305.16367 — a conceptual paper, with no experiments. A base LLM behind a dialogue prompt is best described as role-playing the character the context implies. More precisely, it keeps a superposition of characters consistent with the conversation so far and commits to none of them, and every reply narrows the set. The framing lets folk-psychological words be used without ascribing beliefs or goals to the model. It also gives a behavioural test for kinds of falsehood: invented answers vary under regeneration, “good faith” errors do not, and “deliberate” deception shows when the same question is asked from different contexts. | Active |
| LIT-741 | Textoshop: Interactions Inspired by Drawing Software to Facilitate Text Editing | Masson, Kim and Chevalier (2024), ARXIV-2409.17088 (CHI 2025). A GPT-4o-backed text editor that replaces typed prompts with drawing-software operations, each of which sends an engineered prompt. Resizing a selection shortens or expands it, rotating it reorders the words, a colour picker sets tone, boolean operations merge passages, and layers hold versions. In a within-subject study with 12 participants, against a text editor plus a ChatGPT replica on the same model, participants rated themselves more successful (4.6 vs 3.9), gave a SUS of 90 against 69, and finished tasks 55 s faster. The quality of the resulting text was not measured. | Active |
| LIT-757 | Promptagator: Few-shot Dense Retrieval From 8 Examples | Dai et al. (2022), ARXIV-2209.11755 — Promptagator. Prompt a 137B FLAN with at most 8 examples from the target task to write queries for its corpus, keep a generated pair only if a retriever trained on the generated pairs ranks its source passage top-1, and train a 110M dual encoder on what is left. On 11 BEIR datasets it averages 47.8 nDCG@10 against 46.6 for SPLADE v2 and 46.2 for ColBERT v2, both trained on MS MARCO. The filter is worth 2.5 points on average and lowers three of the eleven datasets. | Active |