Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Analysis and evaluation — how to find out whether something worked — what to measure, what a measurement cannot tell you, and which comparisons are unsound; theory, interpretability and debugging belong here too.
68 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-002 | A dense network contains a sparse subnetwork that matches its accuracy when trained from the same initialization | Frankle and Carbin (2018), LIT-019 — the lottery ticket hypothesis. A randomly-initialized dense network contains a subnetwork that, trained alone from the same initial values, matches the full network in at most the same number of steps. The reset is the claim; the same structure re-initialized randomly does not do it. | Proposed |
| THEORY-004 | A lottery ticket wins by re-learning the solution its dense run already found | Evci et al. (2020), LIT-039 — sparse networks trained from scratch do worse because gradient flow at initialization is poor, and a rewound ticket does not escape that by having found a good sparse architecture: it lands back in the basin the dense pruning run reached. The hypothesis survives as a claim about initialization, not about architecture. | Active |
| THEORY-006 | Task-improving weight perturbations are dense around pretrained weights, and denser the larger the model | Gan and Isola (2026), LIT-233, with a prediction of it independently confirmed by LIT-230 — the fraction of random Gaussian weight perturbations that improve a downstream task rises monotonically with model scale, from 0% at 0.5B to 64% at 32B on GSM8K, and the perturbations that help are task specialists rather than uniform improvements. It is the record’s account of why gradient-free post-training works at all, measured on one model family by one group. | Proposed |
| THEORY-007 | Fine-tuning landscapes are low-dimensional in curvature, and improving directions are degenerate rather than unique | Liang et al. (2026), LIT-236 — a small set of stiff directions carries the improvement in a fine-tuning landscape and their number does not grow with the model. Because improvement depends only on a perturbation’s projection onto that subspace, many ambient perturbations share a useful component, so a fixed population of about thirty keeps working as dimension grows. The same heterogeneity gives rise-then-decay: stiff modes saturate while variance keeps accumulating in the flat bulk. | Proposed |
| THEORY-008 | An evolution-strategies update is mostly a loss-invariant random walk whose size grows with steps and shrinks with population | Hoy et al. (2026), LIT-235, confirmed by LIT-238 — an ES weight update splits into an on-manifold part that changes the loss and an off-manifold part that does not, and in a landscape with many flat directions the second dominates. Its squared norm grows as sigma^2 d T / N, so the drift that three papers in this record read as evidence of forgetting, of functional sparsity, and of a failing search is mostly a random walk that the loss cannot see. | Active |
| THEORY-009 | A wide two-layer network’s training dynamics are a gradient flow on the distribution of its neurons, and that flow is convex | Scale a two-layer network’s output by 1/N and the object that moves under SGD stops being the weights and becomes their empirical distribution, which follows a Wasserstein gradient flow of the population risk. The risk is convex as a functional of that distribution — so the non-convexity of the finite-width landscape, and the permutation symmetry that produces most of its apparent local minima, are both artefacts of the coordinates. Four groups reached this within about a year by four routes. | Active |
| THEORY-017 | An influence-function estimate predicts the effect of removing a training point and retraining | Koh and Liang (2017), LIT-403 — the account the method was derived from and named for: the estimate approximates leave-one-out retraining, so a large influence means the model would have been meaningfully different without that point. True for convex models, where it was checked. Corrected for neural networks in 2022; the method was not. | Rejected — correct where it was validated, which was the convex case; for neural networks the three largest terms of the discrepancy turn out not to be approximation error at all |
| THEORY-018 | Influence functions on neural networks estimate the proximal Bregman response function, not leave-one-out retraining | Bae et al. (2022), LIT-402 — the influence/retraining discrepancy decomposes into five terms, and the two that are genuinely approximation error are an order of magnitude smaller than the three that are not. What the estimate tracks closely is the effect of removing a point while holding predictions near the trained model’s. So the method is not fragile; it was answering a different question. | Active |
| THEORY-023 | Depth-dependent computation in a transformer falls into four stages, of which detokenization is the first | Lad et al. (2024), LIT-417 — the reading the authors put on their own localized-sensitivity result, and they put a question mark in the title. Four stages: detokenization lifts raw token embeddings into contextual ones by integrating local context; feature engineering refines task and entity features; prediction ensembling aggregates hidden states toward next-token predictions; residual sharpening suppresses irrelevant features at the end. The fourth carries a real experiment. The boundaries are approximate by the authors’ own account, and stages may co-occur in one layer. | Proposed |
| THEORY-025 v2 | A transformer holds about 3.6 bits per parameter, and generalization begins where the data outgrows that budget | Morris et al. (2025), LIT-440 — measured on uniform random bitstrings where generalization is impossible, GPT-style transformers store 3.5-4 bits per parameter, linear in parameter count and almost indifferent to weight precision. On real text, double descent begins exactly where the data’s information content crosses that capacity. The proposed mechanism — that a model past capacity must share representation, and sharing is generalization — is an interpretation, not a measurement. | Proposed |
| THEORY-031 | The aggregate loss curve is a lossy projection of training, in at least three measured ways | Three independent measurements of what the loss curve discards. It time-averages oscillation the optimizer is actually doing (LIT-453); it sums over transitions that are individually abrupt and differently timed, so smoothness is what many breakthroughs look like added up (LIT-455); and it reads flat while the weights keep travelling (LIT-454). | Active |
| THEORY-038 | A decoder cannot compose over a long context in few layers, because each position forgets what it forwarded | Chen et al. (2024), LIT-464 — causal masking makes a decoder a line of forgetful communicating players, one epoch per layer, and sequential composition needs more epochs than a constant-depth model has. An unconditional bound on what can be expressed, not on what can be learned. | Active |
| THEORY-039 | A measured capability is the capability minus whatever the evaluation itself demands, and the gap is widest for the weakest model | Hu and Frank (2024), LIT-465 — the same capacity measured two ways scores differently, and the difference shrinks with size and training. So a score is a joint function of model and design, and cross-scale comparisons under a demanding evaluation inflate the gap they report. | Active |
| THEORY-040 | A metric that composes or thresholds per-token error turns a smooth capability curve into a sharp one, with nothing happening in the model | Schaeffer et al. (2023), LIT-471 — if per-token accuracy rises smoothly with scale, a metric demanding all L tokens goes as p^L and is flat-then-sharp by construction; a thresholded metric does the same by a step. Demonstrated by rescoring fixed outputs, and by manufacturing emergence in vision models that had never shown it. | Active |
| THEORY-043 | Cross-entropy pays an unbounded price for mass a learner never had, and that price grows with sequence length — which is why it drifts from what post-training needs | Chen et al. (2025), LIT-474 — KL charges log(1/π̂) for every response a well-generalizing learner happens not to cover, and that charge is unbounded; the coverage profile charges at most the missing mass itself. Sequence-level KL therefore grows linearly in sequence length where coverage does not, and next-token prediction optimizes coverage faster than it optimizes the loss on the screen. | Proposed |
| THEORY-045 | How much of parameter space behaves like your trained network is a description length, and the better-generalizing network occupies more of it | Scherlis and Belrose (2025), LIT-476 — measure the region around a trained network whose behaviour matches it, under the initialization distribution rather than Lebesgue. Negative log of that measure is, by the bits-back argument, a description length. A ConvNeXt trained to generalize badly occupies a smaller region, and local volume falls through training as the model’s description grows. | Proposed |
| THEORY-047 | Personalizing a feed concentrates what it shows rather than broadening it, and a handful of moderate signals is enough to start | Ye, Luceri and Ferrara (2024), LIT-480 — across four arms of an audit, the accounts following nobody receive the most diverse recommendations and the partisan arms the least. Ten moderate media follows and four political accounts are enough to amplify aligned voices more than 50% above a balanced baseline. | Proposed |
| THEORY-048 | Randomizing individuals estimates an individual quantity, and when the units interact the collective effect is not the sum of it — so a null from such a trial is not evidence of no effect | Bak-Coleman et al. (2025), LIT-482 — four named reasons an individual-level randomized trial fails to estimate a collective effect: non-linearity across scale, hysteresis, feedback in time, and violation of the stable unit treatment value assumption through the network. The power grid shows the first and the last together: treat too few and the system absorbs it, treat enough and the controls fail too. | Active |
| THEORY-050 | Parameters carry memorization and parallel computation carries reasoning | Chen et al. (2025), LIT-486 — the exchange rate between parallel streams and parameters is fitted higher on code (0.39) than on general text (0.33), and downstream a 1.6B model at P = 8 matches a 4.4B model on coding but only a 2.8B one on general tasks. Two independent measurements pointing the same way, and an account the authors themselves call a conjecture. | Proposed |
| THEORY-051 v2 | A text feature predicts success partly by being uncommon, so a tool that gives it to everyone destroys its predictive value | Corpus et al. (2025), LIT-487 — length, readability and lexical variety predicted petition success, an AI drafting tool moved all three in the successful direction and raised rated quality, and success did not follow. Inter-petition similarity rose 23% in the same window, and the features’ predictive strength weakened or reversed. | Proposed |
| THEORY-053 | The Stein posterior-mean denoiser acts as an approximate projection onto the learned data geometry | Zhang et al. (2026), LIT-490 — D(x) = E[x₀ | x] maps a noisy point toward the data support, and where the clean data lie on a linear subspace, inside a compact convex set, or on a compact smooth submanifold, the paper bounds its distance from the corresponding projection. The reverse process then reads as a time-varying inexact projected-gradient method rather than as sampling with a nudge. | Proposed |
| THEORY-054 | The regularizer that is intractable in data space is tractable in noise space, and bounds the data-space divergence | Eyring et al. (2025), LIT-491 — the KL to the base model needs Jacobian determinants through the generator and is intractable in data space. Posed over the input noise it reduces, under a Lipschitz condition, to an L2 penalty on the modification, and the data processing inequality makes that an upper bound on the data-space divergence. | Proposed |
| THEORY-055 | A producer’s own rate of return is the price of their time, so making them faster raises the bar on every marginal hour of improvement | Duede, Gross, Crockett and Bergstrom (2026), LIT-492 — in a marginal-value-theorem model of effort allocation, the long-run rate of return is simultaneously the objective and the opportunity cost of the marginal hour. Any acceleration raises it, so the marginal hour of improvement has to clear a higher bar. Whether quality then falls is a race between that bar and the value of the hour, which is why the sign depends on which phase was accelerated rather than on how much time was saved. | Proposed |
| THEORY-056 v2 | Distinct state machines agree at chance on long random inputs, so a learner that reads only statistics loses the signal as the sequences get longer | Giapitzakis, Fountoulakis, Nichani and Lee (2025), LIT-495 — reading a uniformly random word drives a random walk on S_N × S_N, and the probability two distinct semiautomata land in the same state is 1/N + error with |error| ≤ (1 − 1/2N)^T. The quantity that identifies which machine you are looking at decays exponentially in sequence length, so more data of this kind is less evidence, not more. | Proposed |
| THEORY-057 v2 | A corpus a model generates is narrower than the distribution it imitates, and the narrowing shows up as n-gram over-concentration | Zhu et al. (2024), LIT-205, and Finke et al. (2025), LIT-485 — two groups, different corpora, no citation between them, both locating the damage in the same statistic. Generated text concentrates on the generator’s high-probability continuations, so the corpus is narrower than what it imitates; 59.38% of TinyStories contains “once upon a time”, and the proportion of synthetic pretraining data correlates negatively with downstream performance. | Proposed |
| THEORY-058 | Apparent compression in the information plane is saturating activations collapsing into extreme bins, not information being discarded | Saxe et al. (2018), LIT-509 — a tanh unit must grow its weights to compute anything nonlinear, and as it does its activity piles into the saturation regions. Under a fixed binning that is a distribution collapsing into two bins — about 1 bit — which the information plane draws as a compression phase. Proposed, because LIT-507 shows compression in some non-saturating networks once the bins are placed adaptively. | Proposed |
| THEORY-060 | Calibration error and refinement error have separate minimizers during training, so the loss minimum is optimal for neither | Berta et al. (2025), LIT-514 — a proper loss is exactly calibration error plus refinement error, so minimizing it minimizes a sum whose two terms bottom out at different epochs. The proposed mechanism: as the training set becomes separable the model must grow confident to keep training calibration error small, and across a generalization gap that confidence does not transfer — while refinement is still improving. | Proposed |
| THEORY-063 | Contextual concept directions sit in the low-variance tail of the unembedding spectrum while vocabulary contrasts sit at the top | Acharya, Rimal and Dhakal (2026), LIT-526 — concept directions read out of the residual stream anti-concentrate in the low-eigenvalue directions of the unembedding second moment, in 17 of 17 models and by three independent extraction methods. Static unembedding-row contrasts do the opposite. So the same model carries semantic content at both ends of one spectrum, depending on whether the representation is contextual. | Proposed |
| THEORY-064 | Transformers are biased toward low-sensitivity functions, and that bias is what their robustness is made of | Vasudeva et al. (2024), LIT-525 — the probability that a model’s output changes under a random perturbation of one input token is lower for transformers than for MLPs, CNNs, ConvMixers or LSTMs, on vision and language alike. Low sensitivity is a simplicity bias in established senses, it provably implies robustness in the Boolean case, and it tracks flatness. Proposed: the theory is Boolean and NTK, the experiments are not, and the metric is defined relative to a perturbation distribution nobody has swept. | Proposed |
| THEORY-065 | Self-attention drives tokens into a few clusters, and the spectrum of the value matrix decides the geometry they land in | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), LIT-528 — with weights held fixed, self-attention is an interacting particle system and its tokens cluster. Which clusters is not arbitrary: V = I sends them to the vertices of a convex polytope, a simple positive leading eigenvalue to at most three parallel hyperplanes, V = −I to a single point at the origin. In one dimension the attention matrix provably becomes low-rank and Boolean — the structure Linformer and LoRA assume and impose. | Proposed |
| THEORY-067 | A transformer can run a learning algorithm on a model held in its activations, and the architecture admits several, so a construction identifies none | Three groups have constructed transformer weights that run a named learning algorithm in the forward pass on a model carried in the activations: one gradient step per linear-self-attention layer (von Oswald et al., LIT-533), a Sherman–Morrison ridge update in constant depth and O(d²) width (Akyürek et al., LIT-532), and k Iterative Newton steps in k + 8 layers (Fu et al., LIT-535). The frame is what survives the dispute about which algorithm; the corollary is that no construction can settle that dispute, because the architecture admits all three. | Active |
| THEORY-068 | In-context learning is gradient descent on an implicit model | The identification the field read off LIT-533’s title: that what a transformer does with its context is gradient descent on a model held in its activations. Filed Rejected and kept, on three separate grounds. Beyond one layer the source paper’s own models match GD++ — gradient descent on data transformed by I − γXXᵀ — not gradient descent. LIT-535 matches layers to steps and finds a linear correspondence with Iterative Newton and an exponential one with GD, and a conditioning regime GD cannot reach. LIT-536 proves an equivalent algorithm must share ICL’s order sensitivity, which GD has none of. What survives is THEORY-067, the frame without the name. | Rejected |
| THEORY-069 v2 | Grokking is a regime rather than a property of algorithmic data, and at least three knobs move it | Delayed generalization is not a fact about modular arithmetic. It is produced and removed by moving the training regime, and three knobs have been shown to move it: training-set size (Power et al., LIT-538, and the ~60% threshold in LIT-085), initialization scale relative to the generalizing weight norm (Liu et al., LIT-540, which uses it to induce grokking on MNIST, IMDb and QM9 and to eliminate it on algorithmic data), and the alignment between the initial neural tangent kernel and the target (Kumar et al., LIT-537). The three mechanistic accounts in this record contradict each other; they agree on this. | Active |
| THEORY-070 v2 | Grokking is the transition from lazy to rich training dynamics | Kumar, Bordelon, Gershman and Pehlevan (LIT-537). The memorising phase is early lazy training: the network fits the training set in its initial feature basis, which the linearised approximation describes, so training loss falls with no test improvement. Grokking is the late breakdown of that approximation as feature learning begins. Two knobs control it — an output-scale laziness parameter and the alignment between the initial neural tangent kernel and the target — and neither is weight norm. Filed as a rival to both weight-norm accounts, because the paper’s §3 is an explicit counterexample to them and it cannot reproduce what they predicted. | Proposed |
| THEORY-071 v2 | A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive | Varma et al. (LIT-539), making precise the “simpler solution” genre LIT-085 proposed. Two circuit families fit the training set; once cross-entropy is near zero the only remaining pressure is weight decay, which prefers whichever produces a given logit at lower parameter norm. Memorisation gets less efficient as the dataset grows and generalisation does not, so they cross at a critical dataset size D_crit. From that the paper derived ungrokking and semi-grokking and then observed both. What crosses is not indexed on dataset size, though: LIT-667 holds the inferred/atomic ratio fixed, scales the data, and nothing happens. Proposed, on scope rather than on evidence: it needs weight decay, and LIT-537 groks without any. | Proposed |
| THEORY-072 | Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long | Liu, Michaud and Tegmark’s “LU mechanism” (LIT-540). Reduce training and test loss by minimizing over angular directions at each weight norm: training loss is L-shaped, falling and then flat near zero, while test loss is U-shaped with a minimum at a critical norm. A model initialized above that norm fits immediately and generalizes only as regularization walks the norm back down, which is slow — hence the delay. Proposed: it predicts the right dependence on weight decay, and LIT-537 exhibits grokking in which the norm goes the other way. | Proposed |
| THEORY-073 | Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them | Hoogland et al. (LIT-543) estimate the local learning coefficient through training and take critical points of that curve as stage boundaries. In a 2-layer language transformer the five stages land on bigrams, n-grams, previous-token heads and the induction circuit; in an in-context regression transformer they land on the acquisition of in-context learning and then its deterioration. The claimed advantage over a progress measure is that no mechanistic hypothesis is needed in advance. The evidence is coincidence across two models, and the authors call it suggestive. | Proposed |
| THEORY-074 | A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined | Chen, Lau, Mendel, Wei and Murfet (LIT-544). A Bayesian phase transition is a change in where the posterior concentrates as sample size n grows, follows from the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α], and is mathematically well defined. A dynamical transition is a change in the SGD trajectory as a function of step, and its “formal status has remained elusive”. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600, observed at 600–700 — has no dynamical counterpart at all. | Proposed |
| THEORY-075 | Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is | Watanabe’s singular learning theory, as this record receives it from LIT-541 and LIT-542. The set of parameters realising a given function is a variety with singularities, not a point, so the loss is not locally quadratic and d/2 is not the effective parameter count. What is, is the exponent λ in V(ε) ∝ ε^λ — the rate at which the volume of near-optimal parameters shrinks. It fixes the model-selection criterion (n L_n(w_0) + λ log n, not BIC), the Bayes generalisation error (λ/n), and the sense in which one trained network is simpler than another. Curvature is the prefactor, and the prefactor is not what governs any of them. | Active |
| THEORY-076 | Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally | Yang and Hu (2020), LIT-548 — Tensor Programs IV. Under SP a learning rate large enough to move the features blows up the logits, and one small enough to be stable (O(1/width)) leaves the wide limit a kernel machine. µP rescales the readout and first layer so a width-independent rate is both stable and maximal for every layer. A theorem for SGD on MLPs. Why the optimum then transfers, and under Adam, is TP-V’s claim, not this one. | Active |
| THEORY-077 | A coordinate MLP learns high frequencies slowly because its tangent kernel’s spectrum falls off fast, and a sinusoidal input mapping works by making that kernel stationary with a bandwidth you choose | Tancik et al. (2020), LIT-550 — in the NTK regime, error along each kernel eigenvector decays at a rate set by its eigenvalue, and a coordinate MLP’s eigenvalues fall off fast with frequency. Sinusoids of the input give a stationary composed kernel whose bandwidth the frequencies set. The linear model predicts trained networks’ loss curves. It is a kernel-regime account of an input encoding, not of feature learning. | Active |
| THEORY-079 | What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise | Bansal et al. (2022), LIT-553 — the account cold diffusion was named for: the train-to-restore, sample-by-restore-and-redegrade procedure generates whatever the degradation is, so noise is incidental. Its own Table 5 has noiseless blur generation at FID 97.00 on CelebA against 23.11 with noise, and Warm Diffusion’s sweep degrades from 1.85 to 11.97 as noise is taken out. The procedure survives. The claim that noise is not what matters does not. | Rejected — its own paper’s unconditional numbers contradict it, and a blur-to-noise sweep shows generation degrading sharply as noise is removed |
| THEORY-080 v2 | A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes | Hsueh et al. (2025), LIT-555 — blur removes high frequencies deterministically, so many images share a blurry one and the deblurring target is their mean. Stepping toward it leaves the manifold unless noise has already made those images indistinguishable. Natural spectra fall as 1/f² against flat noise, so blur is safe in a band only once noise dominates it. That puts the useful ratio near 0.5 on CIFAR-10. The FID sweep fits the account, and the manifold departure itself is not measured. | Proposed |
| THEORY-081 | A transformer’s feed-forward layers work as key-value memories: each hidden unit’s input weights match input patterns, and its output weights promote the tokens that follow them | Geva et al. (2021), LIT-575 — read FF(x) = f(xK)V as a memory. Keys fire on recognizable prefix patterns, shallow in lower layers and semantic in upper ones. Values, projected to the vocabulary, favor the tokens that follow those patterns. The evidence is 160 keys of one 16-layer model, with value-key agreement of at most 3.5%, and outputs are mostly compositions. This reading is behind ROME, MEMIT and tools that list features by what they “hear” and “predict”. | Proposed |
| THEORY-082 | Causal tracing shows where a language model stores a fact, and that is where the fact should be edited | Meng et al. (2022), LIT-578 — tracing puts factual recall in mid-layer MLPs at the last subject token, and ROME and MEMIT edit there on that basis. Hase et al. (2023), LIT-574, test the link per fact on GPT-J. Tracing peaks are spread across layers, while edits succeed at layer 6 for almost everything. The edit layer explains 94.7% of the variance in success and tracing 0.1% more. The editors work. The reason given for where they edit does not hold. | Rejected — per-fact causal-tracing localization explains 0.1% of the variance in edit success once the edit layer is accounted for, and edits succeed at layers tracing does not point to |
| THEORY-083 | A message-passing GNN can distinguish no more graphs than the 1-WL test, and reaches that limit only if its neighbor aggregation is injective on multisets, as a sum followed by an MLP is and a mean or max is not | Xu et al. (2019), LIT-580 — each message-passing layer refines node labels as one 1-WL iteration can at best, so 1-WL bounds what the network separates. Injective aggregation reaches the bound. Sums of an MLP over a multiset are injective. Means lose counts and maxes lose multiplicities. It is a theorem, and it explains the paper’s training-fit ordering and the collapse of mean aggregators to chance on featureless graphs. | Active |
| THEORY-084 | AlphaFold 2 needs the sequence alignment to find a protein’s coarse fold, not to refine it, so accuracy collapses below a minimum alignment depth and barely improves above it | Jumper et al. (2021), LIT-583 — accuracy falls steeply when the median alignment depth is below about 30 sequences and gains little above about 100. Dropping both metagenomic databases costs 6.1 GDT, almost all from a few targets losing 20 or more. The authors’ hypothesis is that covariation in the alignment is needed early, to place the coarse fold, and that refinement to atomic accuracy does not depend on it. The threshold is measured. The two-stage account is not tested. | Proposed |
| THEORY-085 | InfoNCE maximizes a lower bound on mutual information that cannot exceed log N | Active | |
| THEORY-087 v2 | Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other | Deferred | |
| THEORY-088 v3 | Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal’s information density | Proposed | |
| THEORY-090 | A concept’s shared word-pair direction, its linear probe and its steering vector are one object, related by an inner product under which causally separable concepts are orthogonal | Park, Choe and Veitch (2024), LIT-606 — the output-space direction that counterfactual word pairs share is provably a logit-linear probe, and the input-space direction is provably a steering vector that leaves separable concepts alone. Training fixes neither space’s inner product, so cosine similarity is arbitrary. The inner product that makes separable concepts orthogonal maps one direction onto the other, and one member of that family is the inverse unembedding covariance. Proved under stated assumptions and shown on LLaMA-2 7B. | Proposed |
| THEORY-091 | Command-to-outcome mutual information scores an interface because it measures how reliably the operator’s input determines what happens at the chosen horizon, which rewards a learnable channel rather than an intuitive one | Reddy, Levine and Dragan (2022), LIT-503. The paper’s reason is that an intuitive interface makes commands less noisy. Its own results fit a narrower account: the score measures how much the operator’s input determines the state Δ steps later. Users adapt, so any consistent mapping scores well, and learned interfaces converge on both no perturbation and exact inversion. The account also explains the one failure: an assistant that overrides commands to prevent crashes lowers one-step influence and raises long-horizon influence. | Proposed |
| THEORY-095 | No unbiased estimator of FID exists, so its plug-in bias can reverse a comparison while the sample variance stays small | Bińkowski et al. (2018), LIT-615 — the FID is a non-linear functional of the two distributions, and by the Bickel–Lehmann argument no estimator of it is unbiased for all distributions. The bias is therefore a property of the quantity, not of the plug-in formula, and it does not appear in the standard error: at d=2048 and 50,000 samples the estimator reversed a true ordering in all 100 trials, with standard deviations of 0.2 and 0.5 on a gap of 2.5. The non-existence proof needs a mixture family; what rescues it for practice is that Inception codes are not normal. | Active |
| THEORY-103 v2 | A transformer stores a fact where it needed it, so two-hop composition generalizes only to facts it already saw as a second hop | Wang et al. (2024), LIT-667. Composition and comparison are both two-fact tasks that both arrive by grokking, and only one of them generalizes to facts held out of the training compositions. The difference is where the circuit keeps the facts: composition’s is sequential and ends up with a second copy of the atomic facts in the upper layers, containing only those that appeared as a second hop; comparison’s is parallel and keeps one copy downstairs. So the failure is not an inability to compose — the first hop works on held-out facts — but a missing copy, and tying the two halves of the stack together unlocks it. | Proposed |
| THEORY-104 | A large guidance scale amplifies the model’s derivatives, which narrows a high-order solver’s convergence radius | Lu et al. (2022), LIT-676. Why the fast diffusion samplers invert under guidance. Classifier-free guidance scales the difference between conditional and unconditional predictions, which amplifies the model’s derivatives as well as its output; a k-th order solver is built from k-th order derivatives, so its convergence radius shrinks fastest for the largest k. At a fixed evaluation budget the higher-order method is then worse than the lower, which is what the measured FIDs show — 13.04, 114.62 and 164.74 at 10 evaluations for orders 1, 2 and 3. | Proposed |
| THEORY-108 | Kaplan’s and Chinchilla’s compute-optimal exponents differ because of how small-scale runs were counted, warmed up and tuned, not because of the learning-rate decay | Porian et al. (NeurIPS 2024), LIT-690, reproduce Kaplan’s a ≈ 0.84–0.86 from 5M to 901M on two datasets, then remove it in steps. Counting the output head’s FLOPs takes 0.13 off. A warmup short enough for small models takes 0.10. Tuning batch, learning rate and β₂ per size, at a constant learning rate, takes another 0.10, landing on 0.497. Matching the cosine decay to each run, Chinchilla’s proposed cause, moves it 0.03. Pearce and Song (LIT-688) independently find counting matters and a constant rate suffices, at 0.8–4.6M. | Proposed |
| THEORY-109 v3 | Classifier guidance may flatter classifier-based metrics because it steps along a classifier gradient, and guidance without a classifier is the control | Ho and Salimans (2022), LIT-693, raising the question about the method they replace. Classifier guidance mixes the score with an image classifier’s input gradient, which “can be interpreted as attempting to confuse an image classifier with a gradient-based adversarial attack” — and FID and Inception Score are both computed with an Inception classifier. So the metric gains may be partly adversarial rather than perceptual. Classifier-free guidance obtains the same trade with no classifier anywhere, which is the control, and it wins — but on the same classifier-based metrics. The antecedent (LIT-699) had already asserted the negative, in one clause of its introduction, with no test anywhere in the paper. SOTA-337 names the instrument that would settle it and nobody has run it on guided sampling. | Proposed |
| THEORY-110 | FID is not neutral between fidelity and coverage: it weights coverage more, so optimizing it walks toward variety at the cost of per-sample quality | Kynkäänniemi et al. (2019), LIT-703, measured on StyleGAN: “FID favors configurations with high recall over the ones with high precision”, and the best-recall configuration set a new state-of-the-art FID. The mechanism they give is that FID is a Wasserstein-2 distance in feature space, so low intrinsic variation yields low FID even when much of that variation is missed. LIT-699 is an independent instance in another family: ADM-G beats BigGAN-deep on FID at three resolutions while losing precision at all three and winning recall. So FID does not merely mix fidelity and coverage — it mixes them with a direction, and a model tuned on FID is tuned toward coverage. | Proposed |
| THEORY-111 | A hyperparameter can look important because it is entangled with an averaging length you froze, so sensitivity attributed to one knob may belong to the pair | Karras et al. (2023), LIT-720, measured on EDM2-S / ImageNet-512: with the weight-averaging length pinned at 13%, varying the learning-rate decay across t_ref ∈ [30k, 160k] moves FID by up to 72%; sweeping the averaging length post hoc puts the whole bracket within 10% of the optimum. The decay’s apparent importance was mostly a fact about the pair. Generalising: a sensitivity curve measured with a second knob frozen at a guess attributes the interaction to the knob being swept, and there is no way to tell from the curve. | Superseded — by SOTA-428; Retired as redundant rather than wrong. The claim — that a hyperparameter’s apparent importance can belong to its interaction with a frozen averaging length — is already stated in SOTA-428’s Conditions, in the same words the source uses: “Much of what looks like learning-rate sensitivity is EMA mismatch”, with the same 72%-against-10% bracket behind it. A theory document restating a practice’s own condition from the same single measurement adds a citation target and no claim. superseded_by: points at the practice because that is where the content lives, not because a practice supersedes a theory. |
| THEORY-112 | Classifier-free guidance improves image quality because its unconditional reference model is worse, not because it emphasizes the class, so guidance is adaptive truncation toward the better-fit density | Karras et al. (2024), LIT-721. Guidance adds a force toward higher log[p₁/p₀]. The reference p₀ is more spread out than p₁ — a harder task on a smaller training slice — so it falls off slower away from the data and the ratio’s gradient points inward, concentrating samples on the manifold. That, not class emphasis, is where CFG’s image-quality gain comes from; the class emphasis is the separate part that costs diversity. Tested with a negative control: degrade the two models incompatibly and guidance stops helping at any weight. | Proposed |
| THEORY-113 v4 | A gradient attribution survives weight randomization because of its input multiplier, not because the attribution is insensitive to the weights | Kokhlikyan et al. (2021), LIT-724. Integrated Gradients and its relatives multiply a path integral by (x − x₀), which does not depend on the model. That factor carries the input’s structure into the map, so the map keeps looking the same when the weights are destroyed. LIT-713 suspected this and said it did not measure it; this does, with the arm that discriminates — drop the multiplier and the map becomes parameter-sensitive. A second, harder arm comes free: on text, where summing over embedding dimensions destroys the multiplier’s structure, the global variant becomes parameter-sensitive too. | Proposed |
| THEORY-114 v2 | A modified-backpropagation rule that keeps only non-negative relevance produces a matrix chain that converges to rank 1, so the later layers can only flip the map’s sign | Sixt, Granz and Landgraf (2019), LIT-729, Theorem 1. The z⁺-rule backpropagates relevance through a product of non-negative matrices; such a product converges to a rank-1 matrix C = c γᵀ, and then C v = λ c for every v. So the relevance vector set at the output — the thing that carries which class is being explained — survives only as a scalar, and can at most flip the map’s sign. Class-insensitivity and independence of the later layers’ weights are the same fact. Active because it is a theorem with a discriminating arm that was published (DeepLIFT does not converge) and a second that was built to order (the DeepLIFT ablation does). | Active |
| THEORY-115 | Guided backpropagation and DeconvNet do partial image recovery, not attribution: the backward ReLU plus a CNN’s local connections reconstruct the input regardless of the class or the weights | Nie, Zhang and Patel (2018), LIT-730, Theorems 1 and 2. In a random three-layer CNN, s_k^GBP(x) ≈ x — guided backpropagation recovers the input, with untrained weights and regardless of the class — while the saliency map and DeconvNet are N(0, I). The causes are the backward ReLU and the small filters that local connectivity implies. So these maps are “unrelated to the decision-making of neural networks”, which is why they survive weight randomization, why they look the same for every class, and why an untrained edge detector matches them. | Active |
| THEORY-116 | Gradient descent on a factorized model is biased toward low rank, not small norm: there are problems where it sends every norm to infinity to lower the rank | Razin (2024), LIT-767, Part II, “Implicit Regularization in Deep Learning May Not Be Explainable by Norms” (Razin and Cohen, NeurIPS 2020). Complete a 2×2 matrix from three entries, with every norm and quasi-norm minimized at a bounded value of the missing one. Gradient flow on a depth L ≥ 2 matrix factorization, from initialization arbitrarily close to zero, sends that entry to infinity with probability 0.5 — every norm diverges together while the effective rank falls to its minimum of 1. So no norm can be what the implicit regularization minimizes; rank is the reading left standing. Proved for linear factorizations and extended, under conditions, to tensor factorizations equivalent to polynomial-activation networks — not to ReLU networks. | Proposed |
| THEORY-126 | A more expressive architecture scales better by lowering the data coefficient rather than the exponent, because it can learn more of the discrete tasks in language-model data from the same tokens | Merrill et al. (2026), LIT-799, §4.2 and App. E. The quantization model of scaling laws treats language modelling as many discrete tasks with power-law frequencies. If a fraction ε of tasks is inexpressible, and those tasks need more tokens or parameters or give less loss reduction, lowering ε shifts the loss curve down without changing the exponent. The hybrid’s fitted data coefficient B is lower than the transformer’s (83.7 against 94.9, non-overlapping CIs), and the exponents overlap. But the instantiation that fits was chosen after the fits, and the same paper finds that removing negative eigenvalues barely changes the scaling. | Proposed |
| THEORY-127 | QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass | Bertsch et al. (2026), LIT-801, §5.4 and Fig. 7, with the dispersion bound of Veličković et al. (2024), LIT-653. In 26 controlled 7B runs, QK norm costs about 4 to 6 HELMET points at 32K. The QK-norm models have higher attention entropy, less attention-sink mass and less attention on the needle, and sink mass correlates positively with long-context score (R² 0.38). The account: QK norm bounds the logit spread, so a head can be neither sharp at long range nor able to park surplus mass on a sink. All of the evidence is correlational, and QK norm also changed norm order in the cleanest pair. | Proposed |
| THEORY-128 | A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies | Kozyrev and Maiboroda (2026), LIT-800, §5.3, Table 6, Fig. 1. In Qwen3.8-27B’s Gated DeltaNet layers, run in FP32 lockstep over 32K tokens, 4-bit quantization noise holds state error at a plateau of about 12.6%. A 1% state impulse falls to 1/e in 80–1,382 steps, where the decay gates imply horizons of 1,895–61,659 tokens. The authors attribute the extra forgetting to the delta rule’s overwrite along each new key. No arm removes the delta rule, so the attribution is inferred, not measured. | Proposed |
| THEORY-129 | Interleaving attention with a negative-eigenvalue delta-rule recurrence expresses problems neither layer type can express alone, because state tracking and recall come from different layers and one alternation composes them | Merrill et al. (2026), LIT-799, §3.3–3.4 and App. B. Fixed-depth transformers lie in TC⁰ and cannot track state (if TC⁰ ≠ NC¹). Bounded-state recurrences cannot recall from a long context. A Gated DeltaNet with negative eigenvalues tracks state, and attention recalls. One alternation of the two solves state-based recall, which neither can alone, and with padding a hybrid recognizes all of NC¹. The proofs are conditional on a complexity conjecture. The synthetic evidence does not separate expressivity from learnability, and the account does not say the expressive gap is what makes hybrids better language models. | Proposed |