Explanation statuses — Whether the record believes the ACCOUNT, which is not what the same word means on a practice. An explanation can be Rejected while the thing it explained goes on working exactly as well as it ever did (ADR-031).
Offered — stated and plausible, on evidence that is suggestive rather than settling — the document says what would settle it.
79 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-002 | A dense network contains a sparse subnetwork that matches its accuracy when trained from the same initialization | Frankle and Carbin (2018), LIT-019 — the lottery ticket hypothesis. A randomly-initialized dense network contains a subnetwork that, trained alone from the same initial values, matches the full network in at most the same number of steps. The reset is the claim; the same structure re-initialized randomly does not do it. | Proposed |
| THEORY-005 | Dense feed-forward layers are already mixtures of experts, and pre-training settles the partition before the neurons | Zhang et al. (2021, 2023), LIT-226 and LIT-228 — a trained dense FFN uses a tiny fraction of its neurons per input, the co-activating neurons partition into functional experts that can be recovered post hoc with the same parameters, and through pre-training the partition stabilizes earlier than the neurons in it. A mixture of experts makes explicit a structure dense training arrives at anyway. | Proposed |
| THEORY-006 | Task-improving weight perturbations are dense around pretrained weights, and denser the larger the model | Gan and Isola (2026), LIT-233, with a prediction of it independently confirmed by LIT-230 — the fraction of random Gaussian weight perturbations that improve a downstream task rises monotonically with model scale, from 0% at 0.5B to 64% at 32B on GSM8K, and the perturbations that help are task specialists rather than uniform improvements. It is the record’s account of why gradient-free post-training works at all, measured on one model family by one group. | Proposed |
| THEORY-007 | Fine-tuning landscapes are low-dimensional in curvature, and improving directions are degenerate rather than unique | Liang et al. (2026), LIT-236 — a small set of stiff directions carries the improvement in a fine-tuning landscape and their number does not grow with the model. Because improvement depends only on a perturbation’s projection onto that subspace, many ambient perturbations share a useful component, so a fixed population of about thirty keeps working as dimension grows. The same heterogeneity gives rise-then-decay: stiff modes saturate while variance keeps accumulating in the flat bulk. | Proposed |
| THEORY-010 v2 | Most of the loss barrier between two independently trained networks is permutation, not disagreement | Two networks trained from different seeds land far apart in weight space and close together in function space, and the linear path between them crosses a loss barrier. The account is that the barrier is mostly an artefact of unit labelling: permute the hidden units of one to match the other and the barrier largely disappears — entirely, for wide enough networks, but not at standard width or on ImageNet. What looked like two different solutions was one solution written in two orders. | Proposed |
| THEORY-012 | How far a batch-size scaling heuristic transfers is a property of the optimizer, not of the heuristic | Shallue et al. (2018), LIT-058, found no learning-rate scaling rule that held across 35 workloads and concluded that practitioners must retune at every batch size. You et al. (2019), LIT-265, then scaled BERT to 32K on a square- root rule with no retuning at all. The two results are usually read as a disagreement about heuristics. They are better read as one result about optimizers: a scaling rule is a guess about how the usable step size grows with the batch, that step is bounded by the worst-conditioned direction the optimizer does not normalize away, and an optimizer that normalizes more directions carries the same rule further. | Proposed |
| THEORY-014 | The outer optimizer is what buys the inner step count | Three papers in this record prescribe how many steps a worker takes between synchronisations, and they differ by a factor of thirty: LIT-362 says 16 to 32, LIT-373 says 12 to 48, LIT-212 says 500. They are not disagreeing about the same quantity. The two with small intervals average parameters and stop there; the one with a large interval runs Nesterov momentum over the accumulated deltas, and its own ablation reports that plain averaging at that interval performs poorly. The interval is not a constant to transfer. It is bought, and the outer optimizer is what pays for it. | Proposed |
| THEORY-023 | Depth-dependent computation in a transformer falls into four stages, of which detokenization is the first | Lad et al. (2024), LIT-417 — the reading the authors put on their own localized-sensitivity result, and they put a question mark in the title. Four stages: detokenization lifts raw token embeddings into contextual ones by integrating local context; feature engineering refines task and entity features; prediction ensembling aggregates hidden states toward next-token predictions; residual sharpening suppresses irrelevant features at the end. The fourth carries a real experiment. The boundaries are approximate by the authors’ own account, and stages may co-occur in one layer. | Proposed |
| THEORY-024 v2 | Orthogonalising the update is dualising it, and muP and Shampoo are two partial approximations of the same duality map | Bernstein and Newhouse (2024), LIT-438 — gradients are dual vectors and weights are primal, so the subtraction in gradient descent needs a duality map. Build it recursively from per-layer operator norms and the Linear case is a Newton-Schulz iteration, which is what Muon does. The same construction shows muP and Shampoo are partial approximations of one map. | Proposed |
| THEORY-025 v2 | A transformer holds about 3.6 bits per parameter, and generalization begins where the data outgrows that budget | Morris et al. (2025), LIT-440 — measured on uniform random bitstrings where generalization is impossible, GPT-style transformers store 3.5-4 bits per parameter, linear in parameter count and almost indifferent to weight precision. On real text, double descent begins exactly where the data’s information content crosses that capacity. The proposed mechanism — that a model past capacity must share representation, and sharing is generalization — is an interpretation, not a measurement. | Proposed |
| THEORY-026 | Critical batch size is set by how much data has been seen, not by how large the model is | Zhang et al. (2024), LIT-445 — under maximal update parameterization there is a width past which more width does not raise the batch a step can usefully absorb, while for mini-batch SGD on least squares under power-law source and capacity conditions the useful batch grows as a power of the sample count. Two arguments, one for each half of the split the measurements show. | Proposed |
| THEORY-029 | A model of bounded capacity allocates it across datasets like a knapsack, so the optimum jumps rather than sliding | Gu et al. (2025), LIT-451 — a model with bounded capacity training on a mixture must decide how much capacity each dataset gets, and the allocation minimising total loss is a discrete choice. A discrete optimum moves discontinuously when its inputs move continuously, which is why knowledge acquisition has thresholds in model size and mixing ratio rather than the smooth scaling the single-dataset case shows. | Proposed |
| THEORY-030 | Adaptive optimizers work by shaping the curvature they adapt to, and oscillation is how a first-order method sees curvature at all | Cohen et al. (2024), LIT-453 — at the edge of stability the effective step size is set by current sharpness, and the learning rate’s role is to modulate an implicit curvature penalty that steers the trajectory toward flatter regions where larger steps are stable. Ablating the shaping while keeping the adaptation makes the optimizer slower. Separately, oscillation is how a first-order method acquires second-order information at no extra cost. | Proposed |
| THEORY-032 v3 | A spectral update wins where the incoming activations are low stable rank and the gradient spectrum is spread out | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by exactly the ratio of the gradient’s nuclear rank to the incoming activations’ stable rank, and transformers sit where that ratio is large. A comparison of bounds, not of realized training. | Proposed |
| THEORY-033 | What a spectral optimizer buys is a step size that stays optimal, not adherence to a target geometry | Shumaylov et al. (2026), LIT-456 — an optimizer with random singular values matches Muon, so the target spectrum is not what pays. What the controls leave standing is that the spectral update’s optimal step size is constant where Euclidean descent’s oscillates. NanoGPT scale. | Proposed |
| THEORY-034 v2 | Concepts sit in embedding space as intersections of half-spaces, so inclusion, intersection and union are geometric operations | Xiong (2026), LIT-460 — a linear attribute direction plus a separating threshold is a half-space, a concept is the intersection of its attributes’ half-spaces, and the resulting regions form a complete lattice. Conceptual refinement is geometric meet, generalisation is join. Evidence is WordNet, which is a hand-built hierarchy and so the friendliest test there is. | Proposed |
| THEORY-036 | Representations converge across architectures, objectives and modalities, and the endpoint is a model of what generated the data | Huh et al. (2024), LIT-458 — vision and language models measure distance between datapoints increasingly alike as they scale, across architectures and objectives. The conjecture is that they are converging on a representation of the joint distribution that generated the observations. The convergence is measured; the endpoint is proved only for a world of bijective observations. | Proposed |
| THEORY-041 | Left unconstrained, a transformer’s matrices drift into a badly conditioned, rank-deficient shape, and the norms are the part nothing was managing | Loshchilov et al. (2024), LIT-472 — trained GPT embeddings form a hyper-ellipsoid with a high condition number and attention matrices whose singular values suggest rank deficiency; renormalizing after training narrows the gap without closing it. Proposed, because the evidence is correlational: these models are worse conditioned and they also train slower, and nothing yet connects the two. | Proposed |
| THEORY-043 | Cross-entropy pays an unbounded price for mass a learner never had, and that price grows with sequence length — which is why it drifts from what post-training needs | Chen et al. (2025), LIT-474 — KL charges log(1/π̂) for every response a well-generalizing learner happens not to cover, and that charge is unbounded; the coverage profile charges at most the missing mass itself. Sequence-level KL therefore grows linearly in sequence length where coverage does not, and next-token prediction optimizes coverage faster than it optimizes the loss on the screen. | Proposed |
| THEORY-044 | Dropout early in training buys a biased gradient estimate with much less directional variance, and the trade is favourable until it is not | Liu et al. (2023), LIT-475 — with dropout, mini-batch gradients are a biased estimate of the whole-dataset gradient, because each batch runs through a different sub-network. Their directional variance falls far enough that the angle to the true gradient falls too — for about the first thousand iterations, after which it rises and dropout goes back to being a regularizer. | Proposed |
| THEORY-045 | How much of parameter space behaves like your trained network is a description length, and the better-generalizing network occupies more of it | Scherlis and Belrose (2025), LIT-476 — measure the region around a trained network whose behaviour matches it, under the initialization distribution rather than Lebesgue. Negative log of that measure is, by the bits-back argument, a description length. A ConvNeXt trained to generalize badly occupies a smaller region, and local volume falls through training as the model’s description grows. | Proposed |
| THEORY-047 | Personalizing a feed concentrates what it shows rather than broadening it, and a handful of moderate signals is enough to start | Ye, Luceri and Ferrara (2024), LIT-480 — across four arms of an audit, the accounts following nobody receive the most diverse recommendations and the partisan arms the least. Ten moderate media follows and four political accounts are enough to amplify aligned voices more than 50% above a balanced baseline. | Proposed |
| THEORY-049 | Modalities compete for the same feed-forward parameters, the competition is not symmetric between them, and that is where separating by modality pays | Liang et al. (2024), LIT-483 — untying transformer components by modality at controlled FLOPs pays most in the feed-forward, less in the attention projections, and not at all in the layer norms. Merging any two modalities into one tower degrades both, and by different amounts in each direction. | Proposed |
| THEORY-050 | Parameters carry memorization and parallel computation carries reasoning | Chen et al. (2025), LIT-486 — the exchange rate between parallel streams and parameters is fitted higher on code (0.39) than on general text (0.33), and downstream a 1.6B model at P = 8 matches a 4.4B model on coding but only a 2.8B one on general tasks. Two independent measurements pointing the same way, and an account the authors themselves call a conjecture. | Proposed |
| THEORY-051 v2 | A text feature predicts success partly by being uncommon, so a tool that gives it to everyone destroys its predictive value | Corpus et al. (2025), LIT-487 — length, readability and lexical variety predicted petition success, an AI drafting tool moved all three in the successful direction and raised rated quality, and success did not follow. Inter-petition similarity rose 23% in the same window, and the features’ predictive strength weakened or reversed. | Proposed |
| THEORY-052 | Synthesis requires equivariant representations and discriminative pretraining rewards invariant ones, which is why a generative prior groups objects it was never shown | Khangaonkar and Pirsiavash (2025), LIT-488 — self-distillation and contrastive objectives explicitly reward representations that do not change under augmentation; instance segmentation needs representations that do. The paper’s DINO control activates on objects and cannot separate their instances, which is the shape this account predicts. | Proposed |
| THEORY-053 | The Stein posterior-mean denoiser acts as an approximate projection onto the learned data geometry | Zhang et al. (2026), LIT-490 — D(x) = E[x₀ | x] maps a noisy point toward the data support, and where the clean data lie on a linear subspace, inside a compact convex set, or on a compact smooth submanifold, the paper bounds its distance from the corresponding projection. The reverse process then reads as a time-varying inexact projected-gradient method rather than as sampling with a nudge. | Proposed |
| THEORY-054 | The regularizer that is intractable in data space is tractable in noise space, and bounds the data-space divergence | Eyring et al. (2025), LIT-491 — the KL to the base model needs Jacobian determinants through the generator and is intractable in data space. Posed over the input noise it reduces, under a Lipschitz condition, to an L2 penalty on the modification, and the data processing inequality makes that an upper bound on the data-space divergence. | Proposed |
| THEORY-055 | A producer’s own rate of return is the price of their time, so making them faster raises the bar on every marginal hour of improvement | Duede, Gross, Crockett and Bergstrom (2026), LIT-492 — in a marginal-value-theorem model of effort allocation, the long-run rate of return is simultaneously the objective and the opportunity cost of the marginal hour. Any acceleration raises it, so the marginal hour of improvement has to clear a higher bar. Whether quality then falls is a race between that bar and the value of the hour, which is why the sign depends on which phase was accelerated rather than on how much time was saved. | Proposed |
| THEORY-056 v2 | Distinct state machines agree at chance on long random inputs, so a learner that reads only statistics loses the signal as the sequences get longer | Giapitzakis, Fountoulakis, Nichani and Lee (2025), LIT-495 — reading a uniformly random word drives a random walk on S_N × S_N, and the probability two distinct semiautomata land in the same state is 1/N + error with |error| ≤ (1 − 1/2N)^T. The quantity that identifies which machine you are looking at decays exponentially in sequence length, so more data of this kind is less evidence, not more. | Proposed |
| THEORY-057 v2 | A corpus a model generates is narrower than the distribution it imitates, and the narrowing shows up as n-gram over-concentration | Zhu et al. (2024), LIT-205, and Finke et al. (2025), LIT-485 — two groups, different corpora, no citation between them, both locating the damage in the same statistic. Generated text concentrates on the generator’s high-probability continuations, so the corpus is narrower than what it imitates; 59.38% of TinyStories contains “once upon a time”, and the proportion of synthetic pretraining data correlates negatively with downstream performance. | Proposed |
| THEORY-058 | Apparent compression in the information plane is saturating activations collapsing into extreme bins, not information being discarded | Saxe et al. (2018), LIT-509 — a tanh unit must grow its weights to compute anything nonlinear, and as it does its activity piles into the saturation regions. Under a fixed binning that is a distribution collapsing into two bins — about 1 bit — which the information plane draws as a compression phase. Proposed, because LIT-507 shows compression in some non-saturating networks once the bins are placed adaptively. | Proposed |
| THEORY-059 v2 | A low-rank branch corrects quantization because weight spectra are steep and quantization-error spectra are flat | Li et al. (2024), LIT-512 — two propositions bound the output error by the magnitude of weights and activations, not only by their rounding errors. So a rank-r branch helps exactly when the thing it subtracts has a few dominant singular values. A weight matrix does; a quantization error does not, which is why the same trick applied to the error rather than the weights underperforms. | Proposed |
| THEORY-060 | Calibration error and refinement error have separate minimizers during training, so the loss minimum is optimal for neither | Berta et al. (2025), LIT-514 — a proper loss is exactly calibration error plus refinement error, so minimizing it minimizes a sum whose two terms bottom out at different epochs. The proposed mechanism: as the training set becomes separable the model must grow confident to keep training calibration error small, and across a generalization gap that confidence does not transfer — while refinement is still improving. | Proposed |
| THEORY-061 | Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product | Zhai et al. (2023), LIT-523 — with σ = ‖W_K W_Q^T‖₂·‖XX^T‖₂, the minimum attainable attention entropy behaves like Ω(Tσe^{−σ}), and the bound is tight. So a saturated softmax is not bad luck: growing weights force it. Adaptive optimizers make it worse, because the idealized Adam update’s spectral norm grows like √w in the width. Proposed, because a later paper exhibits a stable network in precisely the collapsed state. | Proposed |
| THEORY-062 | What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such | Qi et al. (2025), LIT-521 — an attention map that is sparse but not low-rank has near-zero entropy and trains fine; one that is sparse and low-rank crashes. So entropy is the symptom. The cause offered is spectral energy concentration of W_q^T W_k, which in crashed runs collapses into fewer than 10 directions. Proposed: it is one group’s counterexample to another group’s account, at 300M and below. | Proposed |
| THEORY-063 | Contextual concept directions sit in the low-variance tail of the unembedding spectrum while vocabulary contrasts sit at the top | Acharya, Rimal and Dhakal (2026), LIT-526 — concept directions read out of the residual stream anti-concentrate in the low-eigenvalue directions of the unembedding second moment, in 17 of 17 models and by three independent extraction methods. Static unembedding-row contrasts do the opposite. So the same model carries semantic content at both ends of one spectrum, depending on whether the representation is contextual. | Proposed |
| THEORY-064 | Transformers are biased toward low-sensitivity functions, and that bias is what their robustness is made of | Vasudeva et al. (2024), LIT-525 — the probability that a model’s output changes under a random perturbation of one input token is lower for transformers than for MLPs, CNNs, ConvMixers or LSTMs, on vision and language alike. Low sensitivity is a simplicity bias in established senses, it provably implies robustness in the Boolean case, and it tracks flatness. Proposed: the theory is Boolean and NTK, the experiments are not, and the metric is defined relative to a perturbation distribution nobody has swept. | Proposed |
| THEORY-065 | Self-attention drives tokens into a few clusters, and the spectrum of the value matrix decides the geometry they land in | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), LIT-528 — with weights held fixed, self-attention is an interacting particle system and its tokens cluster. Which clusters is not arbitrary: V = I sends them to the vertices of a convex polytope, a simple positive leading eigenvalue to at most three parallel hyperplanes, V = −I to a single point at the origin. In one dimension the attention matrix provably becomes low-rank and Boolean — the structure Linformer and LoRA assume and impose. | Proposed |
| THEORY-066 | Pure self-attention collapses every token onto one, and skip connections prevent it by keeping short paths alive | Dong, Cordonnier and Loukas (2021), LIT-530 — decompose a self-attention network into paths, one head per layer, and the residual after removing the best rank-1 approximation shrinks at a cubic rate: doubly exponential in depth. Skip connections create short paths — including one that skips everything — and that is what stops it. MLPs slow it in proportion to their Lipschitz constant. Layer normalization cannot help at all, because it is a right multiplication. | Proposed |
| THEORY-070 v2 | Grokking is the transition from lazy to rich training dynamics | Kumar, Bordelon, Gershman and Pehlevan (LIT-537). The memorising phase is early lazy training: the network fits the training set in its initial feature basis, which the linearised approximation describes, so training loss falls with no test improvement. Grokking is the late breakdown of that approximation as feature learning begins. Two knobs control it — an output-scale laziness parameter and the alignment between the initial neural tangent kernel and the target — and neither is weight norm. Filed as a rival to both weight-norm accounts, because the paper’s §3 is an explicit counterexample to them and it cannot reproduce what they predicted. | Proposed |
| THEORY-071 v2 | A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive | Varma et al. (LIT-539), making precise the “simpler solution” genre LIT-085 proposed. Two circuit families fit the training set; once cross-entropy is near zero the only remaining pressure is weight decay, which prefers whichever produces a given logit at lower parameter norm. Memorisation gets less efficient as the dataset grows and generalisation does not, so they cross at a critical dataset size D_crit. From that the paper derived ungrokking and semi-grokking and then observed both. What crosses is not indexed on dataset size, though: LIT-667 holds the inferred/atomic ratio fixed, scales the data, and nothing happens. Proposed, on scope rather than on evidence: it needs weight decay, and LIT-537 groks without any. | Proposed |
| THEORY-072 | Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long | Liu, Michaud and Tegmark’s “LU mechanism” (LIT-540). Reduce training and test loss by minimizing over angular directions at each weight norm: training loss is L-shaped, falling and then flat near zero, while test loss is U-shaped with a minimum at a critical norm. A model initialized above that norm fits immediately and generalizes only as regularization walks the norm back down, which is slow — hence the delay. Proposed: it predicts the right dependence on weight decay, and LIT-537 exhibits grokking in which the norm goes the other way. | Proposed |
| THEORY-073 | Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them | Hoogland et al. (LIT-543) estimate the local learning coefficient through training and take critical points of that curve as stage boundaries. In a 2-layer language transformer the five stages land on bigrams, n-grams, previous-token heads and the induction circuit; in an in-context regression transformer they land on the acquisition of in-context learning and then its deterioration. The claimed advantage over a progress measure is that no mechanistic hypothesis is needed in advance. The evidence is coincidence across two models, and the authors call it suggestive. | Proposed |
| THEORY-074 | A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined | Chen, Lau, Mendel, Wei and Murfet (LIT-544). A Bayesian phase transition is a change in where the posterior concentrates as sample size n grows, follows from the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α], and is mathematically well defined. A dynamical transition is a change in the SGD trajectory as a function of step, and its “formal status has remained elusive”. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600, observed at 600–700 — has no dynamical counterpart at all. | Proposed |
| THEORY-078 | Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked | Sitzmann et al. (2020), LIT-551 — with weights U(±√(6/n)), a sum of arcsine inputs is about N(0,1), and a unit normal through sin is arcsine again, so every layer sees the same distribution and few pre-activations exceed π. The mechanism is verified at initialization for 6 and 50 layers. The historical claim, that this is why periodic activations “failed to robustly outperform” before, is not tested. | Proposed |
| THEORY-080 v2 | A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes | Hsueh et al. (2025), LIT-555 — blur removes high frequencies deterministically, so many images share a blurry one and the deblurring target is their mean. Stepping toward it leaves the manifold unless noise has already made those images indistinguishable. Natural spectra fall as 1/f² against flat noise, so blur is safe in a band only once noise dominates it. That puts the useful ratio near 0.5 on CIFAR-10. The FID sweep fits the account, and the manifold departure itself is not measured. | Proposed |
| THEORY-081 | A transformer’s feed-forward layers work as key-value memories: each hidden unit’s input weights match input patterns, and its output weights promote the tokens that follow them | Geva et al. (2021), LIT-575 — read FF(x) = f(xK)V as a memory. Keys fire on recognizable prefix patterns, shallow in lower layers and semantic in upper ones. Values, projected to the vocabulary, favor the tokens that follow those patterns. The evidence is 160 keys of one 16-layer model, with value-key agreement of at most 3.5%, and outputs are mostly compositions. This reading is behind ROME, MEMIT and tools that list features by what they “hear” and “predict”. | Proposed |
| THEORY-084 | AlphaFold 2 needs the sequence alignment to find a protein’s coarse fold, not to refine it, so accuracy collapses below a minimum alignment depth and barely improves above it | Jumper et al. (2021), LIT-583 — accuracy falls steeply when the median alignment depth is below about 30 sequences and gains little above about 100. Dropping both metagenomic databases costs 6.1 GDT, almost all from a few targets losing 20 or more. The authors’ hypothesis is that covariation in the alignment is needed early, to place the coarse fold, and that refinement to atomic accuracy does not depend on it. The threshold is measured. The two-stage account is not tested. | Proposed |
| THEORY-086 v2 | A text document is better modelled as a mixture of several topics, each a distribution over words, than as a draw from a single topic | Blei, Ng and Jordan (2003), LIT-592 — exchangeability of a document’s words implies, by de Finetti, a mixture over a latent parameter. LDA takes that parameter to be a per-document distribution over topics. It has lower held-out perplexity than a mixture of unigrams, which forces one topic per document, on AP newswire and C. elegans abstracts, at every number of topics. The multi-topic structure and LDA’s Dirichlet smoothing are not separated, and perplexity is the only measure. | Proposed |
| THEORY-088 v3 | Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal’s information density | Proposed | |
| THEORY-089 v3 | Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition | Pennington et al. (2014), LIT-602, and Mikolov et al. (2013), LIT-603 — what distinguishes words is the ratio of how often they co-occur with probe words (ice and steam with solid versus gas). A model whose dot products fit log co-occurrence turns ratios into vector differences, so relations become offsets. The same log-linearity makes a sum of vectors act like a product of context distributions, an AND of their contexts. Derived from the objectives and illustrated. The mechanism is not measured. | Proposed |
| THEORY-090 | A concept’s shared word-pair direction, its linear probe and its steering vector are one object, related by an inner product under which causally separable concepts are orthogonal | Park, Choe and Veitch (2024), LIT-606 — the output-space direction that counterfactual word pairs share is provably a logit-linear probe, and the input-space direction is provably a steering vector that leaves separable concepts alone. Training fixes neither space’s inner product, so cosine similarity is arbitrary. The inner product that makes separable concepts orthogonal maps one direction onto the other, and one member of that family is the inverse unembedding covariance. Proved under stated assumptions and shown on LLaMA-2 7B. | Proposed |
| THEORY-091 | Command-to-outcome mutual information scores an interface because it measures how reliably the operator’s input determines what happens at the chosen horizon, which rewards a learnable channel rather than an intuitive one | Reddy, Levine and Dragan (2022), LIT-503. The paper’s reason is that an intuitive interface makes commands less noisy. Its own results fit a narrower account: the score measures how much the operator’s input determines the state Δ steps later. Users adapt, so any consistent mapping scores well, and learned interfaces converge on both no perturbation and exact inversion. The account also explains the one failure: an assistant that overrides commands to prevent crashes lowers one-step influence and raises long-horizon influence. | Proposed |
| THEORY-092 v2 | Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores | Levy, Goldberg and Dagan (2015), LIT-607 — PMI divides by the context’s probability, so a rare context seen once with a word gets a high score and crowds out the informative ones. Raising context counts to α = 0.75 before normalizing raises rare contexts’ probability and lowers their PMI. That is why word2vec’s smoothed negative distribution helps, and why the same smoothing helps PPMI most (up to 9.2 points) and SGNS least (at most 1.4). | Proposed |
| THEORY-094 | Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors | Arora et al. (2016), LIT-613 — if each word is emitted with probability ∝ exp(⟨c, v_w⟩) as a discourse vector c drifts, and word vectors are isotropic, the partition function is nearly constant and PMI(w,w’) ≈ ⟨v_w, v_w’⟩/d holds in low dimensions. That carries THEORY-093 from its unconstrained optimum to practical sizes. Isotropy also makes solving for a relation’s offset a well-conditioned regression that averages away per-word error, which is the step THEORY-089 skipped. The assumptions are checked only loosely. | Proposed |
| THEORY-096 | A Pre-LN residual stream’s per-layer change decays as one over root k, so deep blocks cannot refine the representation | Xie et al. (2023), LIT-639 — in a Pre-LN transformer the residual stream accumulates unnormalised, so by block k it has variance ~k and a single block’s contribution is a 1/√k share of it. The change in the normalised input between consecutive blocks decays as O(1/√k), and adding a block to an N−1 block model moves the output O(1/√N). The architecture’s benefit and its cost are the same mechanism. | Proposed |
| THEORY-097 | Attention is an approximation of Kanerva’s sparse distributed memory, which is why it needs normalized vectors and a fitted temperature | Bricken and Pehlevan (2021), LIT-641 — attention’s update rule is SDM’s read operation under two conditions, L² normalized vectors and a fitted softmax temperature. Its one non-trivial consequence is a retrodiction: those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], inside the range SDM’s optimality criteria span. | Proposed |
| THEORY-099 | Muon’s update matches the outer-product structure of associative memories, which is why it learns tail classes that Adam under-trains | Wang et al. (2025), LIT-654. A linear associative memory is a sum of outer products; orthogonalising the update treats every outer-product direction alike. So Muon produces more isotropic weight spectra than Adam, and on heavy-tailed data that shows up as tail classes being learned where Adam under-trains them. The ablation localises the effect: Muon on value-output weights and the FFN nearly recovers full Muon, and Muon on query-key contributes little. | Proposed |
| THEORY-100 | Activation outliers are a product of pre-training optimization choices, not an emergent property of scale | Ahmadian et al. (2023), LIT-656. The sharp post-training quantization failure above ~6B was described as emergent. Hold the architecture fixed and vary weight decay, dropout, gradient clipping and half-precision format, and the failure moves with those instead: 0.09% degradation against 1.36% at 6B, and a 52B model that loses nothing where OPT-66B loses about 42%. The discontinuity is in the recipe, not the parameter count. | Proposed |
| THEORY-103 v2 | A transformer stores a fact where it needed it, so two-hop composition generalizes only to facts it already saw as a second hop | Wang et al. (2024), LIT-667. Composition and comparison are both two-fact tasks that both arrive by grokking, and only one of them generalizes to facts held out of the training compositions. The difference is where the circuit keeps the facts: composition’s is sequential and ends up with a second copy of the atomic facts in the upper layers, containing only those that appeared as a second hop; comparison’s is parallel and keeps one copy downstairs. So the failure is not an inability to compose — the first hop works on held-out facts — but a missing copy, and tying the two halves of the stack together unlocks it. | Proposed |
| THEORY-104 | A large guidance scale amplifies the model’s derivatives, which narrows a high-order solver’s convergence radius | Lu et al. (2022), LIT-676. Why the fast diffusion samplers invert under guidance. Classifier-free guidance scales the difference between conditional and unconditional predictions, which amplifies the model’s derivatives as well as its output; a k-th order solver is built from k-th order derivatives, so its convergence radius shrinks fastest for the largest k. At a fixed evaluation budget the higher-order method is then worse than the lower, which is what the measured FIDs show — 13.04, 114.62 and 164.74 at 10 evaluations for orders 1, 2 and 3. | Proposed |
| THEORY-105 | Soft targets transfer the teacher’s similarity structure over wrong classes, which carries more information per example than a hard label | Hinton et al. (2015), LIT-680. The small probabilities a trained model assigns to wrong classes encode how it generalizes: a BMW is rarely mistaken for a garbage truck, but far more often than for a carrot. Raising the temperature exposes that structure, and a student trained on it gets more information and less gradient variance per example. This is the account usually called “dark knowledge”, a phrase that is not in the paper. Its evidence does not rule out the simpler reading, that soft targets are a regularizer. | Proposed |
| THEORY-108 | Kaplan’s and Chinchilla’s compute-optimal exponents differ because of how small-scale runs were counted, warmed up and tuned, not because of the learning-rate decay | Porian et al. (NeurIPS 2024), LIT-690, reproduce Kaplan’s a ≈ 0.84–0.86 from 5M to 901M on two datasets, then remove it in steps. Counting the output head’s FLOPs takes 0.13 off. A warmup short enough for small models takes 0.10. Tuning batch, learning rate and β₂ per size, at a constant learning rate, takes another 0.10, landing on 0.497. Matching the cosine decay to each run, Chinchilla’s proposed cause, moves it 0.03. Pearce and Song (LIT-688) independently find counting matters and a constant rate suffices, at 0.8–4.6M. | Proposed |
| THEORY-109 v3 | Classifier guidance may flatter classifier-based metrics because it steps along a classifier gradient, and guidance without a classifier is the control | Ho and Salimans (2022), LIT-693, raising the question about the method they replace. Classifier guidance mixes the score with an image classifier’s input gradient, which “can be interpreted as attempting to confuse an image classifier with a gradient-based adversarial attack” — and FID and Inception Score are both computed with an Inception classifier. So the metric gains may be partly adversarial rather than perceptual. Classifier-free guidance obtains the same trade with no classifier anywhere, which is the control, and it wins — but on the same classifier-based metrics. The antecedent (LIT-699) had already asserted the negative, in one clause of its introduction, with no test anywhere in the paper. SOTA-337 names the instrument that would settle it and nobody has run it on guided sampling. | Proposed |
| THEORY-110 | FID is not neutral between fidelity and coverage: it weights coverage more, so optimizing it walks toward variety at the cost of per-sample quality | Kynkäänniemi et al. (2019), LIT-703, measured on StyleGAN: “FID favors configurations with high recall over the ones with high precision”, and the best-recall configuration set a new state-of-the-art FID. The mechanism they give is that FID is a Wasserstein-2 distance in feature space, so low intrinsic variation yields low FID even when much of that variation is missed. LIT-699 is an independent instance in another family: ADM-G beats BigGAN-deep on FID at three resolutions while losing precision at all three and winning recall. So FID does not merely mix fidelity and coverage — it mixes them with a direction, and a model tuned on FID is tuned toward coverage. | Proposed |
| THEORY-112 | Classifier-free guidance improves image quality because its unconditional reference model is worse, not because it emphasizes the class, so guidance is adaptive truncation toward the better-fit density | Karras et al. (2024), LIT-721. Guidance adds a force toward higher log[p₁/p₀]. The reference p₀ is more spread out than p₁ — a harder task on a smaller training slice — so it falls off slower away from the data and the ratio’s gradient points inward, concentrating samples on the manifold. That, not class emphasis, is where CFG’s image-quality gain comes from; the class emphasis is the separate part that costs diversity. Tested with a negative control: degrade the two models incompatibly and guidance stops helping at any weight. | Proposed |
| THEORY-113 v4 | A gradient attribution survives weight randomization because of its input multiplier, not because the attribution is insensitive to the weights | Kokhlikyan et al. (2021), LIT-724. Integrated Gradients and its relatives multiply a path integral by (x − x₀), which does not depend on the model. That factor carries the input’s structure into the map, so the map keeps looking the same when the weights are destroyed. LIT-713 suspected this and said it did not measure it; this does, with the arm that discriminates — drop the multiplier and the map becomes parameter-sensitive. A second, harder arm comes free: on text, where summing over embedding dimensions destroys the multiplier’s structure, the global variant becomes parameter-sensitive too. | Proposed |
| THEORY-116 | Gradient descent on a factorized model is biased toward low rank, not small norm: there are problems where it sends every norm to infinity to lower the rank | Razin (2024), LIT-767, Part II, “Implicit Regularization in Deep Learning May Not Be Explainable by Norms” (Razin and Cohen, NeurIPS 2020). Complete a 2×2 matrix from three entries, with every norm and quasi-norm minimized at a bounded value of the missing one. Gradient flow on a depth L ≥ 2 matrix factorization, from initialization arbitrarily close to zero, sends that entry to infinity with probability 0.5 — every norm diverges together while the effective rank falls to its minimum of 1. So no norm can be what the implicit regularization minimizes; rank is the reading left standing. Proved for linear factorizations and extended, under conditions, to tensor factorizations equivalent to polynomial-activation networks — not to ReLU networks. | Proposed |
| THEORY-117 | A separator token condenses the content of the segment it closes, rather than only receiving attention a head has nowhere else to put | Chen et al. (2024), LIT-748 — SepLLM. Attention in Llama-3-8B-Instruct concentrates on punctuation and whitespace, and the paper reads that as each segment’s content being condensed into its closing separator. Its evidence is what masking costs: keeping only separators, a window and three initial tokens holds GSM8K-CoT at 77.18 against 77.79 with 47% of the KV, where StreamingLLM at the same budget gets 70.89 and keeping every fourth token instead gets 72.71. The record already holds the opposite reading — LIT-414 found heads parking no-op attention on [SEP], periods and commas, whose values are small, and THEORY-019 is the sink account it belongs to. No experiment in either paper separates the two. | Proposed |
| THEORY-118 | Continuous-time consistency training is unstable through the time derivative in its tangent, and tangent normalization, which MeanFlow’s adaptive loss weight already is, controls it | Lu and Song (2024), LIT-790, §4.1–4.2. The continuous-time consistency gradient follows the tangent df/dt along the ODE. Of its parts, the authors find the time derivative of the network unstable, and derive why: EDM’s time transform contributes a factor 1/cos(t) that blows up at the noise end, and a large Fourier embedding scale oscillates. Normalizing the tangent caps the rest. Zheng et al. (LIT-783) find the same term fragile at 14B and in BF16, and state that MeanFlow’s adaptive weight with p = 1 is tangent normalization, which is checkable algebra. The localization is measured on CIFAR-10 gradient norms, and the fixes on one ImageNet-512 setting. | Proposed |
| THEORY-120 | A likelihood-trained normalizing flow samples well only when its training data carry Gaussian noise well above the quantization width, because its inverse must be well conditioned over the whole Gaussian latent | Zhai et al. (2024), LIT-794, §2.4 and §3.3. Trained on the uniform dequantization noise that likelihood evaluation uses, TarFlow could not sample: “constant numerical issues” and nothing sensible. Gaussian noise of standard deviation about 0.05, against the dequantization noise’s 0.002, gave its best samples. The authors offer two reasons and a hypothesis, and test none of them apart. STARFlow (LIT-785) keeps the noise, at 0.3 in latent space, on TarFlow’s word. One ablation from one group, and an account that is the authors’ own guess. | Proposed |
| THEORY-121 | Distribution-matching distillation loses coverage because it descends a reverse divergence on the student’s own samples, and a consistency term on teacher trajectories restores it | Lu and Song (2024), LIT-790, Fig. 7, and Zheng et al. (2025), LIT-783. DMD (LIT-643) descends KL(p_student ‖ p_teacher), evaluated on the student’s own samples. That direction penalizes mass where the teacher has none and is indifferent to a mode the student no longer produces. A consistency loss supervises points along the teacher’s trajectories that the student did not choose. On one backbone, sCM found distribution matching raising precision and losing recall as guidance rises, while consistency stayed near the teacher; read from the curves, the recall gap is already there at guidance 1. The account is rCM’s framing. That figure is its only measurement, from one group and one backbone, and DMD’s own authors give a different mechanism. | Proposed |
| THEORY-124 | Consistency training is a bootstrapped fixed-point iteration, not descent on a loss, so a target that lags the student swamps a data signal that enters only at the order of the step | Song and Dhariwal (2023), LIT-786, Prop. 1, and Boffi, Albergo and Vanden-Eijnden (2024), LIT-793, Prop. 3.12. Consistency training compares the model at one noise level with a target at the next, and the data enter only through that small step. With the target’s weights equal to the student’s under stop-gradient, the scaled update converges to a regression onto the data. With any other target, such as an EMA, the limiting objective does not depend on the data at all. Proved on a one-point distribution, where it refutes the argument that justified EMA targets. That it is why EMA targets hurt on real data rests on one CIFAR-10 curve. | Proposed |
| THEORY-125 | One affine autoregressive flow block makes every conditional a single Gaussian and cannot approximate an arbitrary density, while two blocks in opposite orders make every conditional but the last a Gaussian mixture and three make all of them one | Gu et al. (2025), LIT-785, §3.1, Prop. 1 and App. A.1. In one affine autoregressive block each coordinate is a mean plus a scale times fresh Gaussian noise, both functions of the coordinates before it, so every conditional is one Gaussian and no multimodal conditional can be represented. A second block in the opposite order feeds later latents into those parameters and turns each conditional into an infinite Gaussian mixture, except the last coordinate’s; a third restores that one. The one-block half is proved and is what TarFlow (LIT-794) measured, FID 267 at matched depth. The universality half is a sketch. | Proposed |
| THEORY-126 | A more expressive architecture scales better by lowering the data coefficient rather than the exponent, because it can learn more of the discrete tasks in language-model data from the same tokens | Merrill et al. (2026), LIT-799, §4.2 and App. E. The quantization model of scaling laws treats language modelling as many discrete tasks with power-law frequencies. If a fraction ε of tasks is inexpressible, and those tasks need more tokens or parameters or give less loss reduction, lowering ε shifts the loss curve down without changing the exponent. The hybrid’s fitted data coefficient B is lower than the transformer’s (83.7 against 94.9, non-overlapping CIs), and the exponents overlap. But the instantiation that fits was chosen after the fits, and the same paper finds that removing negative eigenvalues barely changes the scaling. | Proposed |
| THEORY-127 | QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass | Bertsch et al. (2026), LIT-801, §5.4 and Fig. 7, with the dispersion bound of Veličković et al. (2024), LIT-653. In 26 controlled 7B runs, QK norm costs about 4 to 6 HELMET points at 32K. The QK-norm models have higher attention entropy, less attention-sink mass and less attention on the needle, and sink mass correlates positively with long-context score (R² 0.38). The account: QK norm bounds the logit spread, so a head can be neither sharp at long range nor able to park surplus mass on a sink. All of the evidence is correlational, and QK norm also changed norm order in the cleanest pair. | Proposed |
| THEORY-128 | A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies | Kozyrev and Maiboroda (2026), LIT-800, §5.3, Table 6, Fig. 1. In Qwen3.8-27B’s Gated DeltaNet layers, run in FP32 lockstep over 32K tokens, 4-bit quantization noise holds state error at a plateau of about 12.6%. A 1% state impulse falls to 1/e in 80–1,382 steps, where the decay gates imply horizons of 1,895–61,659 tokens. The authors attribute the extra forgetting to the delta rule’s overwrite along each new key. No arm removes the delta rule, so the attribution is inferred, not measured. | Proposed |
| THEORY-129 | Interleaving attention with a negative-eigenvalue delta-rule recurrence expresses problems neither layer type can express alone, because state tracking and recall come from different layers and one alternation composes them | Merrill et al. (2026), LIT-799, §3.3–3.4 and App. B. Fixed-depth transformers lie in TC⁰ and cannot track state (if TC⁰ ≠ NC¹). Bounded-state recurrences cannot recall from a long context. A Gated DeltaNet with negative eigenvalues tracks state, and attention recalls. One alternation of the two solves state-based recall, which neither can alone, and with padding a hybrid recognizes all of NC¹. The proofs are conditional on a complexity conjecture. The synthetic evidence does not separate expressivity from learnability, and the account does not say the expressive gap is what makes hybrids better language models. | Proposed |