Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Training optimization — optimizers, learning-rate schedules, batch size, training dynamics, scaling laws and scaling strategies.
34 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-012 | How far a batch-size scaling heuristic transfers is a property of the optimizer, not of the heuristic | Shallue et al. (2018), LIT-058, found no learning-rate scaling rule that held across 35 workloads and concluded that practitioners must retune at every batch size. You et al. (2019), LIT-265, then scaled BERT to 32K on a square- root rule with no retuning at all. The two results are usually read as a disagreement about heuristics. They are better read as one result about optimizers: a scaling rule is a guess about how the usable step size grows with the batch, that step is bounded by the worst-conditioned direction the optimizer does not normalize away, and an optimizer that normalizes more directions carries the same rule further. | Proposed |
| THEORY-013 | The SGD noise scale g = eps*N/B sets generalization, so the optimal batch size grows with the training set | Smith and Le (2017), LIT-305, read SGD as a stochastic differential equation with noise scale g = eps*N/B, argued that minibatch noise is what drives the optimizer toward minima that generalize, and derived that the optimal batch size is proportional to the learning rate and to the training set size. Filed Rejected and kept: LIT-058 swept 35 workloads thirteen months later, found no evidence that larger batches degrade out-of-sample performance once the metaparameters are retuned, and reviewed this paper by name as one that did not retune. The phenomenon the account explains is, on the larger measurement, mostly an artefact of holding the learning rate fixed. | Rejected |
| THEORY-024 v2 | Orthogonalising the update is dualising it, and muP and Shampoo are two partial approximations of the same duality map | Bernstein and Newhouse (2024), LIT-438 — gradients are dual vectors and weights are primal, so the subtraction in gradient descent needs a duality map. Build it recursively from per-layer operator norms and the Linear case is a Newton-Schulz iteration, which is what Muon does. The same construction shows muP and Shampoo are partial approximations of one map. | Proposed |
| THEORY-026 | Critical batch size is set by how much data has been seen, not by how large the model is | Zhang et al. (2024), LIT-445 — under maximal update parameterization there is a width past which more width does not raise the batch a step can usefully absorb, while for mini-batch SGD on least squares under power-law source and capacity conditions the useful batch grows as a power of the sample count. Two arguments, one for each half of the split the measurements show. | Proposed |
| THEORY-028 v2 | The plateau before factual recall is the formation of the attention circuit that recall needs | Zucchet et al. (2025), LIT-450 — during the plateau between generic statistics and entity-specific knowledge, the attention circuit that selects an attribute value is being built, and until it exists the error at the attribute token does not reach the name tokens. Shown by patching a reference model’s attention patterns in, which removes the plateau entirely. | Active |
| THEORY-029 | A model of bounded capacity allocates it across datasets like a knapsack, so the optimum jumps rather than sliding | Gu et al. (2025), LIT-451 — a model with bounded capacity training on a mixture must decide how much capacity each dataset gets, and the allocation minimising total loss is a discrete choice. A discrete optimum moves discontinuously when its inputs move continuously, which is why knowledge acquisition has thresholds in model size and mixing ratio rather than the smooth scaling the single-dataset case shows. | Proposed |
| THEORY-030 | Adaptive optimizers work by shaping the curvature they adapt to, and oscillation is how a first-order method sees curvature at all | Cohen et al. (2024), LIT-453 — at the edge of stability the effective step size is set by current sharpness, and the learning rate’s role is to modulate an implicit curvature penalty that steers the trajectory toward flatter regions where larger steps are stable. Ablating the shaping while keeping the adaptation makes the optimizer slower. Separately, oscillation is how a first-order method acquires second-order information at no extra cost. | Proposed |
| THEORY-032 v3 | A spectral update wins where the incoming activations are low stable rank and the gradient spectrum is spread out | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by exactly the ratio of the gradient’s nuclear rank to the incoming activations’ stable rank, and transformers sit where that ratio is large. A comparison of bounds, not of realized training. | Proposed |
| THEORY-033 | What a spectral optimizer buys is a step size that stays optimal, not adherence to a target geometry | Shumaylov et al. (2026), LIT-456 — an optimizer with random singular values matches Muon, so the target spectrum is not what pays. What the controls leave standing is that the spectral update’s optimal step size is constant where Euclidean descent’s oscillates. NanoGPT scale. | Proposed |
| THEORY-035 v2 | Gradient descent drives the sharpness up to the largest value its own step size tolerates, and then trains there | Cohen et al. (2021), LIT-461 — the top Hessian eigenvalue rises until it reaches 2/eta and then stops, so the step size sets the curvature rather than responding to it. Full-batch gradient descent only: under SGD the sharpness settles nowhere predictable. | Active |
| THEORY-037 | Residual-branch depth sets the depth scaling rule, because a branch with two transformations has a second-order update term that a branch with one does not | Zheng et al. (2026), LIT-462 — Depth-muP and CompleteP are k = 1 and k ≥ 2 of one condition. The cross term where both weights in a branch move in the same step is what tightens the residual multiplier from 1/√L to 1/L, and it does not exist when the branch holds one weight. | Active |
| THEORY-040 | A metric that composes or thresholds per-token error turns a smooth capability curve into a sharp one, with nothing happening in the model | Schaeffer et al. (2023), LIT-471 — if per-token accuracy rises smoothly with scale, a metric demanding all L tokens goes as p^L and is flat-then-sharp by construction; a thresholded metric does the same by a step. Demonstrated by rescoring fixed outputs, and by manufacturing emergence in vision models that had never shown it. | Active |
| THEORY-042 | An update smaller than the lattice spacing is not merely rounded away — it is cancelled exactly, and carrying the remainder makes the discrete path shadow the continuous one within half a grid cell | Xu et al. (2026), LIT-473 §5 — decompose rounding as identity plus error and expand the trajectory: when every step falls short of the grid, the accumulated quantization loss cancels the accumulated ideal update term for term and the weights never move. Stochastic rounding replaces stagnation with a random walk whose variance grows in T. Carrying the remainder bounds the deviation from the ideal path at Δ/2 for all T. | Active |
| THEORY-043 | Cross-entropy pays an unbounded price for mass a learner never had, and that price grows with sequence length — which is why it drifts from what post-training needs | Chen et al. (2025), LIT-474 — KL charges log(1/π̂) for every response a well-generalizing learner happens not to cover, and that charge is unbounded; the coverage profile charges at most the missing mass itself. Sequence-level KL therefore grows linearly in sequence length where coverage does not, and next-token prediction optimizes coverage faster than it optimizes the loss on the screen. | Proposed |
| THEORY-044 | Dropout early in training buys a biased gradient estimate with much less directional variance, and the trade is favourable until it is not | Liu et al. (2023), LIT-475 — with dropout, mini-batch gradients are a biased estimate of the whole-dataset gradient, because each batch runs through a different sub-network. Their directional variance falls far enough that the angle to the true gradient falls too — for about the first thousand iterations, after which it rises and dropout goes back to being a regularizer. | Proposed |
| THEORY-049 | Modalities compete for the same feed-forward parameters, the competition is not symmetric between them, and that is where separating by modality pays | Liang et al. (2024), LIT-483 — untying transformer components by modality at controlled FLOPs pays most in the feed-forward, less in the attention projections, and not at all in the layer norms. Merging any two modalities into one tower degrades both, and by different amounts in each direction. | Proposed |
| THEORY-060 | Calibration error and refinement error have separate minimizers during training, so the loss minimum is optimal for neither | Berta et al. (2025), LIT-514 — a proper loss is exactly calibration error plus refinement error, so minimizing it minimizes a sum whose two terms bottom out at different epochs. The proposed mechanism: as the training set becomes separable the model must grow confident to keep training calibration error small, and across a generalization gap that confidence does not transfer — while refinement is still improving. | Proposed |
| THEORY-068 | In-context learning is gradient descent on an implicit model | The identification the field read off LIT-533’s title: that what a transformer does with its context is gradient descent on a model held in its activations. Filed Rejected and kept, on three separate grounds. Beyond one layer the source paper’s own models match GD++ — gradient descent on data transformed by I − γXXᵀ — not gradient descent. LIT-535 matches layers to steps and finds a linear correspondence with Iterative Newton and an exponential one with GD, and a conditioning regime GD cannot reach. LIT-536 proves an equivalent algorithm must share ICL’s order sensitivity, which GD has none of. What survives is THEORY-067, the frame without the name. | Rejected |
| THEORY-069 v2 | Grokking is a regime rather than a property of algorithmic data, and at least three knobs move it | Delayed generalization is not a fact about modular arithmetic. It is produced and removed by moving the training regime, and three knobs have been shown to move it: training-set size (Power et al., LIT-538, and the ~60% threshold in LIT-085), initialization scale relative to the generalizing weight norm (Liu et al., LIT-540, which uses it to induce grokking on MNIST, IMDb and QM9 and to eliminate it on algorithmic data), and the alignment between the initial neural tangent kernel and the target (Kumar et al., LIT-537). The three mechanistic accounts in this record contradict each other; they agree on this. | Active |
| THEORY-070 v2 | Grokking is the transition from lazy to rich training dynamics | Kumar, Bordelon, Gershman and Pehlevan (LIT-537). The memorising phase is early lazy training: the network fits the training set in its initial feature basis, which the linearised approximation describes, so training loss falls with no test improvement. Grokking is the late breakdown of that approximation as feature learning begins. Two knobs control it — an output-scale laziness parameter and the alignment between the initial neural tangent kernel and the target — and neither is weight norm. Filed as a rival to both weight-norm accounts, because the paper’s §3 is an explicit counterexample to them and it cannot reproduce what they predicted. | Proposed |
| THEORY-071 v2 | A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive | Varma et al. (LIT-539), making precise the “simpler solution” genre LIT-085 proposed. Two circuit families fit the training set; once cross-entropy is near zero the only remaining pressure is weight decay, which prefers whichever produces a given logit at lower parameter norm. Memorisation gets less efficient as the dataset grows and generalisation does not, so they cross at a critical dataset size D_crit. From that the paper derived ungrokking and semi-grokking and then observed both. What crosses is not indexed on dataset size, though: LIT-667 holds the inferred/atomic ratio fixed, scales the data, and nothing happens. Proposed, on scope rather than on evidence: it needs weight decay, and LIT-537 groks without any. | Proposed |
| THEORY-072 | Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long | Liu, Michaud and Tegmark’s “LU mechanism” (LIT-540). Reduce training and test loss by minimizing over angular directions at each weight norm: training loss is L-shaped, falling and then flat near zero, while test loss is U-shaped with a minimum at a critical norm. A model initialized above that norm fits immediately and generalizes only as regularization walks the norm back down, which is slow — hence the delay. Proposed: it predicts the right dependence on weight decay, and LIT-537 exhibits grokking in which the norm goes the other way. | Proposed |
| THEORY-073 | Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them | Hoogland et al. (LIT-543) estimate the local learning coefficient through training and take critical points of that curve as stage boundaries. In a 2-layer language transformer the five stages land on bigrams, n-grams, previous-token heads and the induction circuit; in an in-context regression transformer they land on the acquisition of in-context learning and then its deterioration. The claimed advantage over a progress measure is that no mechanistic hypothesis is needed in advance. The evidence is coincidence across two models, and the authors call it suggestive. | Proposed |
| THEORY-074 | A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined | Chen, Lau, Mendel, Wei and Murfet (LIT-544). A Bayesian phase transition is a change in where the posterior concentrates as sample size n grows, follows from the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α], and is mathematically well defined. A dynamical transition is a change in the SGD trajectory as a function of step, and its “formal status has remained elusive”. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600, observed at 600–700 — has no dynamical counterpart at all. | Proposed |
| THEORY-075 | Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is | Watanabe’s singular learning theory, as this record receives it from LIT-541 and LIT-542. The set of parameters realising a given function is a variety with singularities, not a point, so the loss is not locally quadratic and d/2 is not the effective parameter count. What is, is the exponent λ in V(ε) ∝ ε^λ — the rate at which the volume of near-optimal parameters shrinks. It fixes the model-selection criterion (n L_n(w_0) + λ log n, not BIC), the Bayes generalisation error (λ/n), and the sense in which one trained network is simpler than another. Curvature is the prefactor, and the prefactor is not what governs any of them. | Active |
| THEORY-076 | Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally | Yang and Hu (2020), LIT-548 — Tensor Programs IV. Under SP a learning rate large enough to move the features blows up the logits, and one small enough to be stable (O(1/width)) leaves the wide limit a kernel machine. µP rescales the readout and first layer so a width-independent rate is both stable and maximal for every layer. A theorem for SGD on MLPs. Why the optimum then transfers, and under Adam, is TP-V’s claim, not this one. | Active |
| THEORY-099 | Muon’s update matches the outer-product structure of associative memories, which is why it learns tail classes that Adam under-trains | Wang et al. (2025), LIT-654. A linear associative memory is a sum of outer products; orthogonalising the update treats every outer-product direction alike. So Muon produces more isotropic weight spectra than Adam, and on heavy-tailed data that shows up as tail classes being learned where Adam under-trains them. The ablation localises the effect: Muon on value-output weights and the FFN nearly recovers full Muon, and Muon on query-key contributes little. | Proposed |
| THEORY-105 | Soft targets transfer the teacher’s similarity structure over wrong classes, which carries more information per example than a hard label | Hinton et al. (2015), LIT-680. The small probabilities a trained model assigns to wrong classes encode how it generalizes: a BMW is rarely mistaken for a garbage truck, but far more often than for a carrot. Raising the temperature exposes that structure, and a student trained on it gets more information and less gradient variance per example. This is the account usually called “dark knowledge”, a phrase that is not in the paper. Its evidence does not rule out the simpler reading, that soft targets are a regularizer. | Proposed |
| THEORY-106 v2 | Gaussian flow matching is a diffusion model; what separates the named recipes is loss weighting, network output and sampling schedule | Gao et al. (2024), LIT-678, on LIT-630. With a Gaussian source the flow-matching interpolant is a diffusion forward process with α_t = 1−t, σ_t = t. The conditional flow-matching loss is an ε-MSE under one particular weighting, and the flow-matching Euler sampler is DDIM. So a result labelled “flow matching against diffusion”, or “straight against curved path”, is a comparison of three things: the loss weighting, the network output and the sampling schedule. “Straight” describes the conditional path to one data point, not the marginal ODE a sampler integrates. | Active |
| THEORY-107 | A diffusion loss whose weighting falls monotonically with signal-to-noise is maximum likelihood on noise-augmented data; a non-monotone one is not a likelihood objective | Kingma and Gao (NeurIPS 2023), LIT-692. Every standard diffusion loss is ½∫ w(λ) E‖ε̂ − ε‖² dλ for an implied weighting. Integration by parts turns it into an expectation, over p_w = dw/dt, of the KL of the reverse process from time t. That is a distribution exactly when w decreases in log-SNR. The loss is then the expected ELBO of Gaussian-noise-augmented data, with the model told the noise level. v-MSE with a cosine schedule, flow matching on the OT path with uniform t, and InDI are monotone. ε-MSE with a cosine schedule, EDM, P2 and Min-SNR are not. | Active |
| THEORY-108 | Kaplan’s and Chinchilla’s compute-optimal exponents differ because of how small-scale runs were counted, warmed up and tuned, not because of the learning-rate decay | Porian et al. (NeurIPS 2024), LIT-690, reproduce Kaplan’s a ≈ 0.84–0.86 from 5M to 901M on two datasets, then remove it in steps. Counting the output head’s FLOPs takes 0.13 off. A warmup short enough for small models takes 0.10. Tuning batch, learning rate and β₂ per size, at a constant learning rate, takes another 0.10, landing on 0.497. Matching the cosine decay to each run, Chinchilla’s proposed cause, moves it 0.03. Pearce and Song (LIT-688) independently find counting matters and a constant rate suffices, at 0.8–4.6M. | Proposed |
| THEORY-111 | A hyperparameter can look important because it is entangled with an averaging length you froze, so sensitivity attributed to one knob may belong to the pair | Karras et al. (2023), LIT-720, measured on EDM2-S / ImageNet-512: with the weight-averaging length pinned at 13%, varying the learning-rate decay across t_ref ∈ [30k, 160k] moves FID by up to 72%; sweeping the averaging length post hoc puts the whole bracket within 10% of the optimum. The decay’s apparent importance was mostly a fact about the pair. Generalising: a sensitivity curve measured with a second knob frozen at a guess attributes the interaction to the knob being swept, and there is no way to tell from the curve. | Superseded — by SOTA-428; Retired as redundant rather than wrong. The claim — that a hyperparameter’s apparent importance can belong to its interaction with a frozen averaging length — is already stated in SOTA-428’s Conditions, in the same words the source uses: “Much of what looks like learning-rate sensitivity is EMA mismatch”, with the same 72%-against-10% bracket behind it. A theory document restating a practice’s own condition from the same single measurement adds a citation target and no claim. superseded_by: points at the practice because that is where the content lives, not because a practice supersedes a theory. |
| THEORY-116 | Gradient descent on a factorized model is biased toward low rank, not small norm: there are problems where it sends every norm to infinity to lower the rank | Razin (2024), LIT-767, Part II, “Implicit Regularization in Deep Learning May Not Be Explainable by Norms” (Razin and Cohen, NeurIPS 2020). Complete a 2×2 matrix from three entries, with every norm and quasi-norm minimized at a bounded value of the missing one. Gradient flow on a depth L ≥ 2 matrix factorization, from initialization arbitrarily close to zero, sends that entry to infinity with probability 0.5 — every norm diverges together while the effective rank falls to its minimum of 1. So no norm can be what the implicit regularization minimizes; rank is the reading left standing. Proved for linear factorizations and extended, under conditions, to tensor factorizations equivalent to polynomial-activation networks — not to ReLU networks. | Proposed |
| THEORY-126 | A more expressive architecture scales better by lowering the data coefficient rather than the exponent, because it can learn more of the discrete tasks in language-model data from the same tokens | Merrill et al. (2026), LIT-799, §4.2 and App. E. The quantization model of scaling laws treats language modelling as many discrete tasks with power-law frequencies. If a fraction ε of tasks is inexpressible, and those tasks need more tokens or parameters or give less loss reduction, lowering ε shifts the loss curve down without changing the exponent. The hybrid’s fitted data coefficient B is lower than the transformer’s (83.7 against 94.9, non-overlapping CIs), and the exponents overlap. But the instantiation that fits was chosen after the fits, and the same paper finds that removing negative eigenvalues barely changes the scaling. | Proposed |