Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Model stability — initialization, normalization, gradient handling, regularization, loss-landscape behaviour.
26 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-001 | Batch normalization works by reducing internal covariate shift | Ioffe and Szegedy (2015), LIT-002 — the explanation batch normalization was named after and introduced with: that training is slowed by each layer’s input distribution shifting as the layers below it update, and that holding those distributions steady is what buys the speed. Refuted in 2018; the technique was not. | Rejected — the experiment that would have confirmed it was eventually run, and the benefit survived the shift being put back |
| THEORY-003 | Batch normalization helps by smoothing the optimization landscape, not by reducing internal covariate shift | Santurkar et al. (2018), LIT-223 — the distributional stability batch normalization was named after turns out not to be what it does. Injecting covariate shift back in after the BN layer costs nothing; what BN changes is the smoothness of the loss surface, and other normalisations that do nothing for covariate shift change it comparably. | Active |
| THEORY-010 v2 | Most of the loss barrier between two independently trained networks is permutation, not disagreement | Two networks trained from different seeds land far apart in weight space and close together in function space, and the linear path between them crosses a loss barrier. The account is that the barrier is mostly an artefact of unit labelling: permute the hidden units of one to match the other and the barrier largely disappears — entirely, for wide enough networks, but not at standard width or on ImageNet. What looked like two different solutions was one solution written in two orders. | Proposed |
| THEORY-011 | Skip connections make a deep network trainable by smoothing the loss surface, not by making it more expressive | Li et al. (2017), LIT-014 — the same deep network plotted with and without skip connections gives a chaotic surface with visible barriers between nearby points, and a smooth near-convex one. The claim is about trainability rather than capacity: the residual network is not a larger function class, it is a reachable one. It is the reason depth stopped being the barrier it had been, and the reason every practice in this record about residual streams is about what to do with them rather than whether to have them. | Active |
| THEORY-015 | Dropout regularizes by imposing a data-dependent penalty, not an isotropic one | Wager et al. (2013), LIT-396 — for generalized linear models dropout is first-order equivalent to L2 applied after scaling the features by an estimate of the inverse diagonal Fisher information, so the penalty depends on the data and not only on the weights. Three derivations from three directions agree on that shape — inverse-curvature scaling here, variance-scaled weight decay in LIT-393, ridge with a standard-deviation-scaled Gamma in LIT-395 — and none of them covers a deep network. | Active |
| THEORY-016 | Test-time weight scaling computes a geometric-mean average over the dropout ensemble, exactly for logistic units and to second order elsewhere | Baldi and Sadowski (2013), LIT-393 — the single forward pass with scaled weights is not a heuristic stand-in for the ensemble. For a logistic unit it computes the normalized weighted geometric mean over all dropout configurations exactly, and the logistic and constant functions are the only ones with that property. In deep networks one approximation remains, it is exact for consistent units and good to second order, and LIT-395’s Monte-Carlo comparison is the empirical check. | Active |
| THEORY-041 | Left unconstrained, a transformer’s matrices drift into a badly conditioned, rank-deficient shape, and the norms are the part nothing was managing | Loshchilov et al. (2024), LIT-472 — trained GPT embeddings form a hyper-ellipsoid with a high condition number and attention matrices whose singular values suggest rank deficiency; renormalizing after training narrows the gap without closing it. Proposed, because the evidence is correlational: these models are worse conditioned and they also train slower, and nothing yet connects the two. | Proposed |
| THEORY-044 | Dropout early in training buys a biased gradient estimate with much less directional variance, and the trade is favourable until it is not | Liu et al. (2023), LIT-475 — with dropout, mini-batch gradients are a biased estimate of the whole-dataset gradient, because each batch runs through a different sub-network. Their directional variance falls far enough that the angle to the true gradient falls too — for about the first thousand iterations, after which it rises and dropout goes back to being a regularizer. | Proposed |
| THEORY-045 | How much of parameter space behaves like your trained network is a description length, and the better-generalizing network occupies more of it | Scherlis and Belrose (2025), LIT-476 — measure the region around a trained network whose behaviour matches it, under the initialization distribution rather than Lebesgue. Negative log of that measure is, by the bits-back argument, a description length. A ConvNeXt trained to generalize badly occupies a smaller region, and local volume falls through training as the model’s description grows. | Proposed |
| THEORY-061 | Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product | Zhai et al. (2023), LIT-523 — with σ = ‖W_K W_Q^T‖₂·‖XX^T‖₂, the minimum attainable attention entropy behaves like Ω(Tσe^{−σ}), and the bound is tight. So a saturated softmax is not bad luck: growing weights force it. Adaptive optimizers make it worse, because the idealized Adam update’s spectral norm grows like √w in the width. Proposed, because a later paper exhibits a stable network in precisely the collapsed state. | Proposed |
| THEORY-062 | What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such | Qi et al. (2025), LIT-521 — an attention map that is sparse but not low-rank has near-zero entropy and trains fine; one that is sparse and low-rank crashes. So entropy is the symptom. The cause offered is spectral energy concentration of W_q^T W_k, which in crashed runs collapses into fewer than 10 directions. Proposed: it is one group’s counterexample to another group’s account, at 300M and below. | Proposed |
| THEORY-066 | Pure self-attention collapses every token onto one, and skip connections prevent it by keeping short paths alive | Dong, Cordonnier and Loukas (2021), LIT-530 — decompose a self-attention network into paths, one head per layer, and the residual after removing the best rank-1 approximation shrinks at a cubic rate: doubly exponential in depth. Skip connections create short paths — including one that skips everything — and that is what stops it. MLPs slow it in proportion to their Lipschitz constant. Layer normalization cannot help at all, because it is a right multiplication. | Proposed |
| THEORY-069 v2 | Grokking is a regime rather than a property of algorithmic data, and at least three knobs move it | Delayed generalization is not a fact about modular arithmetic. It is produced and removed by moving the training regime, and three knobs have been shown to move it: training-set size (Power et al., LIT-538, and the ~60% threshold in LIT-085), initialization scale relative to the generalizing weight norm (Liu et al., LIT-540, which uses it to induce grokking on MNIST, IMDb and QM9 and to eliminate it on algorithmic data), and the alignment between the initial neural tangent kernel and the target (Kumar et al., LIT-537). The three mechanistic accounts in this record contradict each other; they agree on this. | Active |
| THEORY-070 v2 | Grokking is the transition from lazy to rich training dynamics | Kumar, Bordelon, Gershman and Pehlevan (LIT-537). The memorising phase is early lazy training: the network fits the training set in its initial feature basis, which the linearised approximation describes, so training loss falls with no test improvement. Grokking is the late breakdown of that approximation as feature learning begins. Two knobs control it — an output-scale laziness parameter and the alignment between the initial neural tangent kernel and the target — and neither is weight norm. Filed as a rival to both weight-norm accounts, because the paper’s §3 is an explicit counterexample to them and it cannot reproduce what they predicted. | Proposed |
| THEORY-071 v2 | A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive | Varma et al. (LIT-539), making precise the “simpler solution” genre LIT-085 proposed. Two circuit families fit the training set; once cross-entropy is near zero the only remaining pressure is weight decay, which prefers whichever produces a given logit at lower parameter norm. Memorisation gets less efficient as the dataset grows and generalisation does not, so they cross at a critical dataset size D_crit. From that the paper derived ungrokking and semi-grokking and then observed both. What crosses is not indexed on dataset size, though: LIT-667 holds the inferred/atomic ratio fixed, scales the data, and nothing happens. Proposed, on scope rather than on evidence: it needs weight decay, and LIT-537 groks without any. | Proposed |
| THEORY-072 | Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long | Liu, Michaud and Tegmark’s “LU mechanism” (LIT-540). Reduce training and test loss by minimizing over angular directions at each weight norm: training loss is L-shaped, falling and then flat near zero, while test loss is U-shaped with a minimum at a critical norm. A model initialized above that norm fits immediately and generalizes only as regularization walks the norm back down, which is slow — hence the delay. Proposed: it predicts the right dependence on weight decay, and LIT-537 exhibits grokking in which the norm goes the other way. | Proposed |
| THEORY-074 | A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined | Chen, Lau, Mendel, Wei and Murfet (LIT-544). A Bayesian phase transition is a change in where the posterior concentrates as sample size n grows, follows from the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α], and is mathematically well defined. A dynamical transition is a change in the SGD trajectory as a function of step, and its “formal status has remained elusive”. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600, observed at 600–700 — has no dynamical counterpart at all. | Proposed |
| THEORY-075 | Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is | Watanabe’s singular learning theory, as this record receives it from LIT-541 and LIT-542. The set of parameters realising a given function is a variety with singularities, not a point, so the loss is not locally quadratic and d/2 is not the effective parameter count. What is, is the exponent λ in V(ε) ∝ ε^λ — the rate at which the volume of near-optimal parameters shrinks. It fixes the model-selection criterion (n L_n(w_0) + λ log n, not BIC), the Bayes generalisation error (λ/n), and the sense in which one trained network is simpler than another. Curvature is the prefactor, and the prefactor is not what governs any of them. | Active |
| THEORY-076 | Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally | Yang and Hu (2020), LIT-548 — Tensor Programs IV. Under SP a learning rate large enough to move the features blows up the logits, and one small enough to be stable (O(1/width)) leaves the wide limit a kernel machine. µP rescales the readout and first layer so a width-independent rate is both stable and maximal for every layer. A theorem for SGD on MLPs. Why the optimum then transfers, and under Adam, is TP-V’s claim, not this one. | Active |
| THEORY-078 | Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked | Sitzmann et al. (2020), LIT-551 — with weights U(±√(6/n)), a sum of arcsine inputs is about N(0,1), and a unit normal through sin is arcsine again, so every layer sees the same distribution and few pre-activations exceed π. The mechanism is verified at initialization for 6 and 50 layers. The historical claim, that this is why periodic activations “failed to robustly outperform” before, is not tested. | Proposed |
| THEORY-087 v2 | Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other | Deferred | |
| THEORY-096 | A Pre-LN residual stream’s per-layer change decays as one over root k, so deep blocks cannot refine the representation | Xie et al. (2023), LIT-639 — in a Pre-LN transformer the residual stream accumulates unnormalised, so by block k it has variance ~k and a single block’s contribution is a 1/√k share of it. The change in the normalised input between consecutive blocks decays as O(1/√k), and adding a block to an N−1 block model moves the output O(1/√N). The architecture’s benefit and its cost are the same mechanism. | Proposed |
| THEORY-116 | Gradient descent on a factorized model is biased toward low rank, not small norm: there are problems where it sends every norm to infinity to lower the rank | Razin (2024), LIT-767, Part II, “Implicit Regularization in Deep Learning May Not Be Explainable by Norms” (Razin and Cohen, NeurIPS 2020). Complete a 2×2 matrix from three entries, with every norm and quasi-norm minimized at a bounded value of the missing one. Gradient flow on a depth L ≥ 2 matrix factorization, from initialization arbitrarily close to zero, sends that entry to infinity with probability 0.5 — every norm diverges together while the effective rank falls to its minimum of 1. So no norm can be what the implicit regularization minimizes; rank is the reading left standing. Proved for linear factorizations and extended, under conditions, to tensor factorizations equivalent to polynomial-activation networks — not to ReLU networks. | Proposed |
| THEORY-118 | Continuous-time consistency training is unstable through the time derivative in its tangent, and tangent normalization, which MeanFlow’s adaptive loss weight already is, controls it | Lu and Song (2024), LIT-790, §4.1–4.2. The continuous-time consistency gradient follows the tangent df/dt along the ODE. Of its parts, the authors find the time derivative of the network unstable, and derive why: EDM’s time transform contributes a factor 1/cos(t) that blows up at the noise end, and a large Fourier embedding scale oscillates. Normalizing the tangent caps the rest. Zheng et al. (LIT-783) find the same term fragile at 14B and in BF16, and state that MeanFlow’s adaptive weight with p = 1 is tangent normalization, which is checkable algebra. The localization is measured on CIFAR-10 gradient norms, and the fixes on one ImageNet-512 setting. | Proposed |
| THEORY-124 | Consistency training is a bootstrapped fixed-point iteration, not descent on a loss, so a target that lags the student swamps a data signal that enters only at the order of the step | Song and Dhariwal (2023), LIT-786, Prop. 1, and Boffi, Albergo and Vanden-Eijnden (2024), LIT-793, Prop. 3.12. Consistency training compares the model at one noise level with a target at the next, and the data enter only through that small step. With the target’s weights equal to the student’s under stop-gradient, the scaled update converges to a regression onto the data. With any other target, such as an EMA, the limiting objective does not depend on the data at all. Proved on a one-point distribution, where it refutes the argument that justified EMA targets. That it is why EMA targets hurt on real data rests on one CIFAR-10 curve. | Proposed |
| THEORY-127 | QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass | Bertsch et al. (2026), LIT-801, §5.4 and Fig. 7, with the dispersion bound of Veličković et al. (2024), LIT-653. In 26 controlled 7B runs, QK norm costs about 4 to 6 HELMET points at 32K. The QK-norm models have higher attention entropy, less attention-sink mass and less attention on the needle, and sink mass correlates positively with long-context score (R² 0.38). The account: QK norm bounds the logit spread, so a head can be neither sharp at long range nor able to park surplus mass on a sink. All of the evidence is correlational, and QK norm also changed norm order in the cleanest pair. | Proposed |