Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Capability thresholds — a capability that arrives abruptly rather than smoothly — emergence at scale, grokking after long training, phase transitions in learning — which axis it turns on, and whether the discontinuity is real or an artefact of how it was measured.
13 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-085 v3 | Progress measures for grokking via mechanistic interpretability | Nanda et al. (2023), ARXIV-2301.05217. Grokking measurement. | Active — measurement rather than actionable practice |
| LIT-470 | Emergent Abilities of Large Language Models | Wei et al. (2022), ARXIV-2206.07682. Names and defines emergent abilities — present in larger models, absent in smaller ones, and therefore not predictable by extrapolating a scaling law. A survey of existing curves rather than new experiments: eight few-shot examples across five model families, plus prompting strategies that only help past a threshold. Section 5.1 raises the metric explanation itself and argues against it on two specific grounds. | Active |
| LIT-471 | Are Emergent Abilities of Large Language Models a Mirage? | Schaeffer et al. (2023), ARXIV-2304.15004. A sharp capability curve can be produced from a smooth one by the metric alone. Rescoring fixed GPT-3 outputs under Token Edit Distance removes the emergence; >92% of hand-annotated BIG-Bench emergent abilities sit under two metrics, one nonlinear and one discontinuous; and emergence is induced on demand in vision models that had never shown it. | Active |
| LIT-537 | Grokking as the Transition from Lazy to Rich Training Dynamics | Kumar, Bordelon, Gershman and Pehlevan (2023), ARXIV-2310.06110, ICLR 2024 — the counterexample. A two-layer MLP groks on modular arithmetic with no weight decay at all, and its parameter norm rises through the transition, which the weight-norm accounts cannot produce. The proposed replacement: grokking is the network leaving the lazy (linearised) regime and starting to learn features, controlled by an output-scale parameter α and by the alignment between the initial neural tangent kernel and the target. Read as NOTE-284. | Active |
| LIT-538 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power, Burda, Edwards, Babuschkin and Misra (2022), ARXIV-2201.02177 — the paper that named grokking. A small transformer on division mod 97 at 50% data reaches near-perfect training accuracy in under 10³ steps and near-perfect validation accuracy around 10⁶, with almost no generalization before 10⁵. The regime dependence is measured here, not later: near 25–30% data on S₅, removing 1% of the training set raises median time to generalization by 40–50%, while time to fit stays at 10³–10⁴ throughout. Weight decay more than halves the data needed. Read as NOTE-281. | Active |
| LIT-539 | Explaining grokking through circuit efficiency | Varma, Shah, Kenton, Kramár and Kumar (2023), ARXIV-2309.02390, DeepMind — the account that earns its place by predicting things nobody had seen. Memorising and generalising circuits both fit the training set; weight decay prefers whichever produces more logit per unit parameter norm; memorisation gets less efficient as the dataset grows and generalisation does not, so there is a critical dataset size D_crit where the preference flips. From that the paper predicts ungrokking and semi-grokking, and then observes both. Read as NOTE-283. | Active |
| LIT-540 | Omnigrok: Grokking Beyond Algorithmic Data | Liu, Michaud and Tegmark (2022), ARXIV-2210.01117, ICLR 2023 — the paper that answers whether grokking is specific to algorithmic data. No: it is induced on MNIST, IMDb and QM9 by shrinking the training set and scaling the initialization, and eliminated on algorithmic data by constraining the weight norm. The proposed mechanism is the “LU” mismatch — reduced training loss is L-shaped in weight norm and test loss U-shaped, so above the critical norm the two disagree. Read as NOTE-282. | Active |
| LIT-543 | Loss Landscape Degeneracy and Stagewise Development in Transformers | Hoogland, Wang, Farrugia-Roberts, Carroll, Wei and Murfet (2024), ARXIV-2402.02364, TMLR 2025 — the instrument applied. Tracking the LLC through training divides two transformers into five stages each, and the boundaries coincide with structural changes the loss curve does not show: bigrams, then n-grams, then previous-token heads, then the induction circuit. In the in-context regression transformer, in-context learning is acquired in LR2 and then deteriorates across LR3 and LR4 as the model specializes to its pre-training distribution. Read as NOTE-287. | Active |
| LIT-544 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei and Murfet (2023), ARXIV-2310.06301 — the paper that separates two things everyone calls a phase transition. In the Toy Model of Superposition at two hidden dimensions, regular k-gons are critical points with derivable local learning coefficients; the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α] then predicts a Bayesian 5 → 6 transition at n_cr ≈ 600, observed at 600 ≤ n ≤ 700. The same critical points govern SGD plateaus — and the 5 → 6 transition is not one of the dynamical ones. Read as NOTE-288. | Active |
| LIT-617 | Scaling Language Models: Methods, Analysis & Insights from Training Gopher | Rae et al. (2021), ARXIV-2112.11446. Six models on one dataset at one token budget, 44M to 280B, evaluated on 152 tasks — a controlled study of parameter scale. The gains are nonuniform: 51.2% of tasks improve by over 25%, 10.5% not at all, and Gopher is worse than its own smaller models on Abstract Algebra, Temporal Sequences and High School Mathematics. Also the allocation LIT-068 corrects: 280B parameters on 300B tokens is 1.07 tokens per parameter against Chinchilla’s 20. | Active |
| LIT-656 | Intriguing Properties of Quantization at Scale | Ahmadian et al. (2023), ARXIV-2305.19268. A controlled study from 410M to 52B, same architecture, varying weight decay, dropout, gradient clipping and training dtype: outlier dimensions are not an inherent product of scale but of the optimization conditions during pre-training. Their 52B model gains 0.08% under plain INT8 (77.31 → 77.39 average) where OPT-66B loses about 42%. The record’s largest cluster had the emergence framing in an Active practice’s title. | Active |
| LIT-667 | Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization | Wang, Yue, Su and Sun (2024), ARXIV-2405.15071. Grokking outside algorithmic data, on reasoning over stored facts — and a named correction to the critical-data-size hypothesis the record holds as THEORY-071. With the inferred/atomic ratio φ held at 9.0, varying the training-set size changes nothing; varying φ at fixed size moves grokking speed monotonically and at φ = 18.0 all but removes it. Also traces the generalizing circuit for two tasks and explains from its layout why composition never generalizes out of distribution while comparison does. | Active |
| LIT-769 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li (2025), ARXIV-2506.20699. A theory paper with no experiments. It defines a problem’s width, the minimum number of locally feasible contexts needed to cover it, and states that width is incomparable with VC dimension and that a learner with fewer contexts than the width has an error floor no amount of within-context training removes. From the additive split of complexity into a routing term and a within-context term it argues that the parts of a model that keep context structure should not be trained by the gradients that fit predictions inside a context, and it reads hallucination, reward hacking and deceptive alignment as failures of that structure. The current version (v2, 2026) replaced a v1 about a “Context-Content Uncertainty Principle” for brains and machines. | Active |