Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Capability thresholds — a capability that arrives abruptly rather than smoothly — emergence at scale, grokking after long training, phase transitions in learning — which axis it turns on, and whether the discontinuity is real or an artefact of how it was measured.
9 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-040 | A metric that composes or thresholds per-token error turns a smooth capability curve into a sharp one, with nothing happening in the model | Schaeffer et al. (2023), LIT-471 — if per-token accuracy rises smoothly with scale, a metric demanding all L tokens goes as p^L and is flat-then-sharp by construction; a thresholded metric does the same by a step. Demonstrated by rescoring fixed outputs, and by manufacturing emergence in vision models that had never shown it. | Active |
| THEORY-069 v2 | Grokking is a regime rather than a property of algorithmic data, and at least three knobs move it | Delayed generalization is not a fact about modular arithmetic. It is produced and removed by moving the training regime, and three knobs have been shown to move it: training-set size (Power et al., LIT-538, and the ~60% threshold in LIT-085), initialization scale relative to the generalizing weight norm (Liu et al., LIT-540, which uses it to induce grokking on MNIST, IMDb and QM9 and to eliminate it on algorithmic data), and the alignment between the initial neural tangent kernel and the target (Kumar et al., LIT-537). The three mechanistic accounts in this record contradict each other; they agree on this. | Active |
| THEORY-070 v2 | Grokking is the transition from lazy to rich training dynamics | Kumar, Bordelon, Gershman and Pehlevan (LIT-537). The memorising phase is early lazy training: the network fits the training set in its initial feature basis, which the linearised approximation describes, so training loss falls with no test improvement. Grokking is the late breakdown of that approximation as feature learning begins. Two knobs control it — an output-scale laziness parameter and the alignment between the initial neural tangent kernel and the target — and neither is weight norm. Filed as a rival to both weight-norm accounts, because the paper’s §3 is an explicit counterexample to them and it cannot reproduce what they predicted. | Proposed |
| THEORY-071 v2 | A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive | Varma et al. (LIT-539), making precise the “simpler solution” genre LIT-085 proposed. Two circuit families fit the training set; once cross-entropy is near zero the only remaining pressure is weight decay, which prefers whichever produces a given logit at lower parameter norm. Memorisation gets less efficient as the dataset grows and generalisation does not, so they cross at a critical dataset size D_crit. From that the paper derived ungrokking and semi-grokking and then observed both. What crosses is not indexed on dataset size, though: LIT-667 holds the inferred/atomic ratio fixed, scales the data, and nothing happens. Proposed, on scope rather than on evidence: it needs weight decay, and LIT-537 groks without any. | Proposed |
| THEORY-072 | Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long | Liu, Michaud and Tegmark’s “LU mechanism” (LIT-540). Reduce training and test loss by minimizing over angular directions at each weight norm: training loss is L-shaped, falling and then flat near zero, while test loss is U-shaped with a minimum at a critical norm. A model initialized above that norm fits immediately and generalizes only as regularization walks the norm back down, which is slow — hence the delay. Proposed: it predicts the right dependence on weight decay, and LIT-537 exhibits grokking in which the norm goes the other way. | Proposed |
| THEORY-073 | Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them | Hoogland et al. (LIT-543) estimate the local learning coefficient through training and take critical points of that curve as stage boundaries. In a 2-layer language transformer the five stages land on bigrams, n-grams, previous-token heads and the induction circuit; in an in-context regression transformer they land on the acquisition of in-context learning and then its deterioration. The claimed advantage over a progress measure is that no mechanistic hypothesis is needed in advance. The evidence is coincidence across two models, and the authors call it suggestive. | Proposed |
| THEORY-074 | A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined | Chen, Lau, Mendel, Wei and Murfet (LIT-544). A Bayesian phase transition is a change in where the posterior concentrates as sample size n grows, follows from the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α], and is mathematically well defined. A dynamical transition is a change in the SGD trajectory as a function of step, and its “formal status has remained elusive”. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600, observed at 600–700 — has no dynamical counterpart at all. | Proposed |
| THEORY-100 | Activation outliers are a product of pre-training optimization choices, not an emergent property of scale | Ahmadian et al. (2023), LIT-656. The sharp post-training quantization failure above ~6B was described as emergent. Hold the architecture fixed and vary weight decay, dropout, gradient clipping and half-precision format, and the failure moves with those instead: 0.09% degradation against 1.36% at 6B, and a 52B model that loses nothing where OPT-66B loses about 42%. The discontinuity is in the recipe, not the parameter count. | Proposed |
| THEORY-103 v2 | A transformer stores a fact where it needed it, so two-hop composition generalizes only to facts it already saw as a second hop | Wang et al. (2024), LIT-667. Composition and comparison are both two-fact tasks that both arrive by grokking, and only one of them generalizes to facts held out of the training compositions. The difference is where the circuit keeps the facts: composition’s is sequential and ends up with a second copy of the atomic facts in the upper layers, containing only those that appeared as a second hop; comparison’s is parallel and keeps one copy downstairs. So the failure is not an inability to compose — the first hop works on held-out facts — but a missing copy, and tying the two halves of the stack together unlocks it. | Proposed |