Explanation statuses — Whether the record believes the ACCOUNT, which is not what the same word means on a practice. An explanation can be Rejected while the thing it explained goes on working exactly as well as it ever did (ADR-031).
The current account — the best explanation the record holds for why this happens.
42 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-003 | Batch normalization helps by smoothing the optimization landscape, not by reducing internal covariate shift | Santurkar et al. (2018), LIT-223 — the distributional stability batch normalization was named after turns out not to be what it does. Injecting covariate shift back in after the BN layer costs nothing; what BN changes is the smoothness of the loss surface, and other normalisations that do nothing for covariate shift change it comparably. | Active |
| THEORY-004 | A lottery ticket wins by re-learning the solution its dense run already found | Evci et al. (2020), LIT-039 — sparse networks trained from scratch do worse because gradient flow at initialization is poor, and a rewound ticket does not escape that by having found a good sparse architecture: it lands back in the basin the dense pruning run reached. The hypothesis survives as a claim about initialization, not about architecture. | Active |
| THEORY-008 | An evolution-strategies update is mostly a loss-invariant random walk whose size grows with steps and shrinks with population | Hoy et al. (2026), LIT-235, confirmed by LIT-238 — an ES weight update splits into an on-manifold part that changes the loss and an off-manifold part that does not, and in a landscape with many flat directions the second dominates. Its squared norm grows as sigma^2 d T / N, so the drift that three papers in this record read as evidence of forgetting, of functional sparsity, and of a failing search is mostly a random walk that the loss cannot see. | Active |
| THEORY-009 | A wide two-layer network’s training dynamics are a gradient flow on the distribution of its neurons, and that flow is convex | Scale a two-layer network’s output by 1/N and the object that moves under SGD stops being the weights and becomes their empirical distribution, which follows a Wasserstein gradient flow of the population risk. The risk is convex as a functional of that distribution — so the non-convexity of the finite-width landscape, and the permutation symmetry that produces most of its apparent local minima, are both artefacts of the coordinates. Four groups reached this within about a year by four routes. | Active |
| THEORY-011 | Skip connections make a deep network trainable by smoothing the loss surface, not by making it more expressive | Li et al. (2017), LIT-014 — the same deep network plotted with and without skip connections gives a chaotic surface with visible barriers between nearby points, and a smooth near-convex one. The claim is about trainability rather than capacity: the residual network is not a larger function class, it is a reachable one. It is the reason depth stopped being the barrier it had been, and the reason every practice in this record about residual streams is about what to do with them rather than whether to have them. | Active |
| THEORY-015 | Dropout regularizes by imposing a data-dependent penalty, not an isotropic one | Wager et al. (2013), LIT-396 — for generalized linear models dropout is first-order equivalent to L2 applied after scaling the features by an estimate of the inverse diagonal Fisher information, so the penalty depends on the data and not only on the weights. Three derivations from three directions agree on that shape — inverse-curvature scaling here, variance-scaled weight decay in LIT-393, ridge with a standard-deviation-scaled Gamma in LIT-395 — and none of them covers a deep network. | Active |
| THEORY-016 | Test-time weight scaling computes a geometric-mean average over the dropout ensemble, exactly for logistic units and to second order elsewhere | Baldi and Sadowski (2013), LIT-393 — the single forward pass with scaled weights is not a heuristic stand-in for the ensemble. For a logistic unit it computes the normalized weighted geometric mean over all dropout configurations exactly, and the logistic and constant functions are the only ones with that property. In deep networks one approximation remains, it is exact for consistent units and good to second order, and LIT-395’s Monte-Carlo comparison is the empirical check. | Active |
| THEORY-018 | Influence functions on neural networks estimate the proximal Bregman response function, not leave-one-out retraining | Bae et al. (2022), LIT-402 — the influence/retraining discrepancy decomposes into five terms, and the two that are genuinely approximation error are an order of magnitude smaller than the three that are not. What the estimate tracks closely is the effect of removing a point while holding predictions near the trained model’s. So the method is not fragile; it was answering a different question. | Active |
| THEORY-019 | A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink | Xiao et al. (2023), LIT-191 — attention scores are normalized to sum to one, so a head with nothing it needs to attend to still has mass to shed, and it learns to dump it on the positions every query can see, which under causal masking are the first few tokens. The evidence that it is positional rather than semantic: replacing the first four tokens with linebreaks moves Llama-2-13B from 5.40 to 5.60 perplexity, while evicting them from the cache moves it to 5158. LIT-190 finds the same concentration in the residual stream, and SOTA-134’s gate removes sinks rather than relocating them, which is what the account predicts. | Active |
| THEORY-020 | Layer importance is not uniform with depth: middle layers tolerate deletion and reordering, the first and last do not | Lad et al. (2024), LIT-417 — deleting a layer outright, or swapping two adjacent ones, at inference and without fine-tuning, leaves 72-95% of top-1 predictions unchanged. The degradation is localized rather than uniform: the first and last layers are fragile, the middle is not, and swapping hurts less than dropping. Measured layer by layer across four GPT-2 and four Pythia models and confirmed on HellaSwag, ARC-Easy and LAMBADA. Robustness grows with depth, so it is not a small-model artifact. | Active |
| THEORY-021 | A model builds a latent vocabulary in its early layers whose units are not the tokenizer’s | Kaplan et al. (2024), LIT-412, with LIT-409 and LIT-417 — sub-word sequences are recombined into whole-unit representations at a unit’s last token, in early and middle layers, and the units include things the tokenizer has no entry for: arbitrary splits, typos, out-of-vocabulary words, named entities and non-compositional multi-word expressions. Three groups, three methods — a probe with a co-occurrence control, an erasure signature, and layer ablation — reaching the same place. The tokenizer’s vocabulary is the model’s input format, not its inventory of units. | Active |
| THEORY-022 v2 | A phrase placed in a word slot is read as a lemma, and the reading comes from the construction rather than the phrase’s frequency | Goldberg and Shirtz (2025), LIT-410 — English has a productive construction in which a unit with the internal syntax of a phrase, up to a whole sentence, occupies a slot reserved for a word, and comprehenders give it the kind of meaning a word has: it evokes a semantic frame presumed to be shared. Four preregistered surveys measure the reading, and the fourth removes the obvious deflation — the effect is the same size on high-frequency phrases, and estimated corpus frequency predicts nothing. So the lexical unit is delimited by construction, not by how often the string has been seen, which is the assumption a frequency-merge tokenizer makes. | Active |
| THEORY-027 v2 | In continuous time the diffusion bound depends on the noise schedule only through its endpoints | Kingma et al. (2021), LIT-446 — integrate the diffusion variational bound over signal-to-noise ratio instead of over time and the schedule leaves the integrand, remaining only in the limits. The bound and the generative distribution therefore depend on the schedule through two numbers, and variance-preserving and variance-exploding specifications are the same model up to a rescaling of the latents. | Active |
| THEORY-028 v2 | The plateau before factual recall is the formation of the attention circuit that recall needs | Zucchet et al. (2025), LIT-450 — during the plateau between generic statistics and entity-specific knowledge, the attention circuit that selects an attribute value is being built, and until it exists the error at the attribute token does not reach the name tokens. Shown by patching a reference model’s attention patterns in, which removes the plateau entirely. | Active |
| THEORY-031 | The aggregate loss curve is a lossy projection of training, in at least three measured ways | Three independent measurements of what the loss curve discards. It time-averages oscillation the optimizer is actually doing (LIT-453); it sums over transitions that are individually abrupt and differently timed, so smoothness is what many breakthroughs look like added up (LIT-455); and it reads flat while the weights keep travelling (LIT-454). | Active |
| THEORY-035 v2 | Gradient descent drives the sharpness up to the largest value its own step size tolerates, and then trains there | Cohen et al. (2021), LIT-461 — the top Hessian eigenvalue rises until it reaches 2/eta and then stops, so the step size sets the curvature rather than responding to it. Full-batch gradient descent only: under SGD the sharpness settles nowhere predictable. | Active |
| THEORY-037 | Residual-branch depth sets the depth scaling rule, because a branch with two transformations has a second-order update term that a branch with one does not | Zheng et al. (2026), LIT-462 — Depth-muP and CompleteP are k = 1 and k ≥ 2 of one condition. The cross term where both weights in a branch move in the same step is what tightens the residual multiplier from 1/√L to 1/L, and it does not exist when the branch holds one weight. | Active |
| THEORY-038 | A decoder cannot compose over a long context in few layers, because each position forgets what it forwarded | Chen et al. (2024), LIT-464 — causal masking makes a decoder a line of forgetful communicating players, one epoch per layer, and sequential composition needs more epochs than a constant-depth model has. An unconditional bound on what can be expressed, not on what can be learned. | Active |
| THEORY-039 | A measured capability is the capability minus whatever the evaluation itself demands, and the gap is widest for the weakest model | Hu and Frank (2024), LIT-465 — the same capacity measured two ways scores differently, and the difference shrinks with size and training. So a score is a joint function of model and design, and cross-scale comparisons under a demanding evaluation inflate the gap they report. | Active |
| THEORY-040 | A metric that composes or thresholds per-token error turns a smooth capability curve into a sharp one, with nothing happening in the model | Schaeffer et al. (2023), LIT-471 — if per-token accuracy rises smoothly with scale, a metric demanding all L tokens goes as p^L and is flat-then-sharp by construction; a thresholded metric does the same by a step. Demonstrated by rescoring fixed outputs, and by manufacturing emergence in vision models that had never shown it. | Active |
| THEORY-042 | An update smaller than the lattice spacing is not merely rounded away — it is cancelled exactly, and carrying the remainder makes the discrete path shadow the continuous one within half a grid cell | Xu et al. (2026), LIT-473 §5 — decompose rounding as identity plus error and expand the trajectory: when every step falls short of the grid, the accumulated quantization loss cancels the accumulated ideal update term for term and the weights never move. Stochastic rounding replaces stagnation with a random walk whose variance grows in T. Carrying the remainder bounds the deviation from the ideal path at Δ/2 for all T. | Active |
| THEORY-046 | Uniform-state discrete diffusion is the argmax of a Gaussian diffusion, which is why Gaussian technique transfers to it and not to masked diffusion | Sahoo et al. (2025), LIT-479 — taking the argmax of a Gaussian diffusion’s latents carries its marginals onto those of a uniform-state discrete diffusion, under a reparameterization of the noise schedule, and the discretized process satisfies the defining ODE of a discrete diffusion. So it is one. Masked diffusion has no such preimage. | Active |
| THEORY-048 | Randomizing individuals estimates an individual quantity, and when the units interact the collective effect is not the sum of it — so a null from such a trial is not evidence of no effect | Bak-Coleman et al. (2025), LIT-482 — four named reasons an individual-level randomized trial fails to estimate a collective effect: non-linearity across scale, hysteresis, feedback in time, and violation of the stable unit treatment value assumption through the network. The power grid shows the first and the last together: treat too few and the system absorbs it, treat enough and the controls fail too. | Active |
| THEORY-067 | A transformer can run a learning algorithm on a model held in its activations, and the architecture admits several, so a construction identifies none | Three groups have constructed transformer weights that run a named learning algorithm in the forward pass on a model carried in the activations: one gradient step per linear-self-attention layer (von Oswald et al., LIT-533), a Sherman–Morrison ridge update in constant depth and O(d²) width (Akyürek et al., LIT-532), and k Iterative Newton steps in k + 8 layers (Fu et al., LIT-535). The frame is what survives the dispute about which algorithm; the corollary is that no construction can settle that dispute, because the architecture admits all three. | Active |
| THEORY-069 v2 | Grokking is a regime rather than a property of algorithmic data, and at least three knobs move it | Delayed generalization is not a fact about modular arithmetic. It is produced and removed by moving the training regime, and three knobs have been shown to move it: training-set size (Power et al., LIT-538, and the ~60% threshold in LIT-085), initialization scale relative to the generalizing weight norm (Liu et al., LIT-540, which uses it to induce grokking on MNIST, IMDb and QM9 and to eliminate it on algorithmic data), and the alignment between the initial neural tangent kernel and the target (Kumar et al., LIT-537). The three mechanistic accounts in this record contradict each other; they agree on this. | Active |
| THEORY-075 | Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is | Watanabe’s singular learning theory, as this record receives it from LIT-541 and LIT-542. The set of parameters realising a given function is a variety with singularities, not a point, so the loss is not locally quadratic and d/2 is not the effective parameter count. What is, is the exponent λ in V(ε) ∝ ε^λ — the rate at which the volume of near-optimal parameters shrinks. It fixes the model-selection criterion (n L_n(w_0) + λ log n, not BIC), the Bayes generalisation error (λ/n), and the sense in which one trained network is simpler than another. Curvature is the prefactor, and the prefactor is not what governs any of them. | Active |
| THEORY-076 | Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally | Yang and Hu (2020), LIT-548 — Tensor Programs IV. Under SP a learning rate large enough to move the features blows up the logits, and one small enough to be stable (O(1/width)) leaves the wide limit a kernel machine. µP rescales the readout and first layer so a width-independent rate is both stable and maximal for every layer. A theorem for SGD on MLPs. Why the optimum then transfers, and under Adam, is TP-V’s claim, not this one. | Active |
| THEORY-077 | A coordinate MLP learns high frequencies slowly because its tangent kernel’s spectrum falls off fast, and a sinusoidal input mapping works by making that kernel stationary with a bandwidth you choose | Tancik et al. (2020), LIT-550 — in the NTK regime, error along each kernel eigenvector decays at a rate set by its eigenvalue, and a coordinate MLP’s eigenvalues fall off fast with frequency. Sinusoids of the input give a stationary composed kernel whose bandwidth the frequencies set. The linear model predicts trained networks’ loss curves. It is a kernel-regime account of an input encoding, not of feature learning. | Active |
| THEORY-083 | A message-passing GNN can distinguish no more graphs than the 1-WL test, and reaches that limit only if its neighbor aggregation is injective on multisets, as a sum followed by an MLP is and a mean or max is not | Xu et al. (2019), LIT-580 — each message-passing layer refines node labels as one 1-WL iteration can at best, so 1-WL bounds what the network separates. Injective aggregation reaches the bound. Sums of an MLP over a multiset are injective. Means lose counts and maxes lose multiplicities. It is a theorem, and it explains the paper’s training-fit ordering and the collapse of mean aggregators to chance on featureless graphs. | Active |
| THEORY-085 | InfoNCE maximizes a lower bound on mutual information that cannot exceed log N | Active | |
| THEORY-093 | Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily | Levy and Goldberg (2014), LIT-612 — setting the derivative of SGNS’s per-pair objective to zero gives w·c = PMI(w, c) − log k, with k the number of negatives. So word2vec’s skip-gram is a factorization of the matrix count-based methods use, shifted by a constant, with each pair’s error weighted by how often it occurs. NCE gives log P(w|c) − log k the same way. Exact at the optimum with unconstrained dimensions. At practical dimensions it describes the target, not the result. | Active |
| THEORY-095 | No unbiased estimator of FID exists, so its plug-in bias can reverse a comparison while the sample variance stays small | Bińkowski et al. (2018), LIT-615 — the FID is a non-linear functional of the two distributions, and by the Bickel–Lehmann argument no estimator of it is unbiased for all distributions. The bias is therefore a property of the quantity, not of the plug-in formula, and it does not appear in the standard error: at d=2048 and 50,000 samples the estimator reversed a true ordering in all 100 trials, with standard deviations of 0.2 and 0.5 on a gap of 2.5. The non-existence proof needs a mixture family; what rescues it for practice is that Inception codes are not normal. | Active |
| THEORY-098 | Softmax attention coefficients must disperse as the number of items grows, so no learned attention circuit stays sharp out of distribution | Veličković et al. (2024), LIT-653. With logits bounded and temperature non-zero, every softmax coefficient is at most (1/n)·exp(δ/θ) — and in a Transformer over a finite vocabulary the logits are always bounded, so the bound applies in every attention head. A head that looks sharp in-distribution is not a robust circuit; it is a circuit whose input size has not grown yet. | Active |
| THEORY-101 | A large trained ViT repurposes redundant patch tokens as scratch space for global information, which is what its feature-map artifacts are | Darcet et al. (2023), LIT-662. The high-norm tokens that spoil ViT feature maps are not damage: they sit on patches redundant with their neighbours, they have discarded their own local content, and they carry global information instead. The model found spare capacity and used it. The account’s own prediction — give the model dedicated slots and the artifacts should move there — was tested and holds. | Active |
| THEORY-102 | A causal mask implicitly encodes absolute position, because a token can count how many predecessors it may attend to | Haviv et al. (2022), LIT-665. A decoder-only transformer given no positional encoding recovers absolute position within four layers, and the reason is the mask: a token that can count its attendable predecessors knows its index. The account’s prediction is that removing the mask removes the mechanism, and a bidirectional MLM without an encoding does not merely degrade — it fails, at 147.18 perplexity against 4.00. | Active |
| THEORY-106 v2 | Gaussian flow matching is a diffusion model; what separates the named recipes is loss weighting, network output and sampling schedule | Gao et al. (2024), LIT-678, on LIT-630. With a Gaussian source the flow-matching interpolant is a diffusion forward process with α_t = 1−t, σ_t = t. The conditional flow-matching loss is an ε-MSE under one particular weighting, and the flow-matching Euler sampler is DDIM. So a result labelled “flow matching against diffusion”, or “straight against curved path”, is a comparison of three things: the loss weighting, the network output and the sampling schedule. “Straight” describes the conditional path to one data point, not the marginal ODE a sampler integrates. | Active |
| THEORY-107 | A diffusion loss whose weighting falls monotonically with signal-to-noise is maximum likelihood on noise-augmented data; a non-monotone one is not a likelihood objective | Kingma and Gao (NeurIPS 2023), LIT-692. Every standard diffusion loss is ½∫ w(λ) E‖ε̂ − ε‖² dλ for an implied weighting. Integration by parts turns it into an expectation, over p_w = dw/dt, of the KL of the reverse process from time t. That is a distribution exactly when w decreases in log-SNR. The loss is then the expected ELBO of Gaussian-noise-augmented data, with the model told the noise level. v-MSE with a cosine schedule, flow matching on the OT path with uniform t, and InDI are monotone. ε-MSE with a cosine schedule, EDM, P2 and Min-SNR are not. | Active |
| THEORY-114 v2 | A modified-backpropagation rule that keeps only non-negative relevance produces a matrix chain that converges to rank 1, so the later layers can only flip the map’s sign | Sixt, Granz and Landgraf (2019), LIT-729, Theorem 1. The z⁺-rule backpropagates relevance through a product of non-negative matrices; such a product converges to a rank-1 matrix C = c γᵀ, and then C v = λ c for every v. So the relevance vector set at the output — the thing that carries which class is being explained — survives only as a scalar, and can at most flip the map’s sign. Class-insensitivity and independence of the later layers’ weights are the same fact. Active because it is a theorem with a discriminating arm that was published (DeepLIFT does not converge) and a second that was built to order (the DeepLIFT ablation does). | Active |
| THEORY-115 | Guided backpropagation and DeconvNet do partial image recovery, not attribution: the backward ReLU plus a CNN’s local connections reconstruct the input regardless of the class or the weights | Nie, Zhang and Patel (2018), LIT-730, Theorems 1 and 2. In a random three-layer CNN, s_k^GBP(x) ≈ x — guided backpropagation recovers the input, with untrained weights and regardless of the class — while the saliency map and DeconvNet are N(0, I). The causes are the backward ReLU and the small filters that local connectivity implies. So these maps are “unrelated to the decision-making of neural networks”, which is why they survive weight randomization, why they look the same for every class, and why an untrained edge detector matches them. | Active |
| THEORY-119 | Consistency, shortcut, MeanFlow and progressive-distillation models all estimate the two-time flow map of the probability-flow ODE, and differ in which identity of that map they train on | Boffi, Albergo and Vanden-Eijnden (2024), LIT-793. The object a few-step model learns is X_{s,t}, which carries any point of the ODE from time s to time t. It satisfies three identities: a Lagrangian equation in the end time, an Eulerian equation in the start time, and composition. Progressive distillation and shortcut models train on composition, consistency models and MeanFlow on the Eulerian equation, and flow map matching’s Lagrangian loss on the first. Consistency models are the one-time case. Inductive Moment Matching is the exception: it matches marginals, and its minimizer need not be the ODE’s map. Which identity trains best is measured once, on CIFAR-10, where the Lagrangian form won by a wide margin. | Active |
| THEORY-122 | A diffusion or flow sampler can be trained by a likelihood-ratio policy gradient only when each step is stochastic with a tractable density, and a same-marginal SDE built from the trained velocity supplies that without changing the marginals | Black et al. (2023), LIT-776, and Liu et al. (2025), LIT-779. A policy gradient needs the log-density of each action. Treated as one action, a diffusion model’s sample has only a variational bound. Treated as T actions, each DDPM step is an isotropic Gaussian with an exact density. A flow’s ODE step is deterministic, so it has no per-step density and explores nothing past the initial noise. The stochastic- interpolant identity (LIT-645) gives, for any noise level, an SDE with the ODE’s marginals, built from the trained velocity alone. Each of its steps is Gaussian, so the ratio and the KL to the reference are closed form. All of this is derivation. What is not shown is that it survives the ten-step discretization used for rollouts. | Active |
| THEORY-123 | A flow’s sampling ODE curves because the noise-data pairs it was trained on cross and the learned velocity averages them, so a coupling whose straight paths do not cross gives a straight flow | Pooladian et al. (2023), LIT-789, and Tong et al. (2023), LIT-798, with Rectified Flow, LIT-636. Flow matching regresses onto the velocity of a straight line between a paired noise and data point, so the learned field at (x, t) is the average of the velocities of every pair whose line passes through x at t. Under independent pairing many lines cross there, and the average turns. A coupling with no crossings, the OT plan in the limit or the flow’s own map after reflow, leaves one line through each point and a straight flow. The mechanism is proved. That this is what limits few-step image quality is measured only at 64 px and below, and Lee et al. (LIT-791) argue, without proof, that one reflow is enough. | Active |