Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Model architecture — architecture patterns, component design, structural choices, model families — how the network is shaped, not how many signals it takes in.
23 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-005 | Dense feed-forward layers are already mixtures of experts, and pre-training settles the partition before the neurons | Zhang et al. (2021, 2023), LIT-226 and LIT-228 — a trained dense FFN uses a tiny fraction of its neurons per input, the co-activating neurons partition into functional experts that can be recovered post hoc with the same parameters, and through pre-training the partition stabilizes earlier than the neurons in it. A mixture of experts makes explicit a structure dense training arrives at anyway. | Proposed |
| THEORY-020 | Layer importance is not uniform with depth: middle layers tolerate deletion and reordering, the first and last do not | Lad et al. (2024), LIT-417 — deleting a layer outright, or swapping two adjacent ones, at inference and without fine-tuning, leaves 72-95% of top-1 predictions unchanged. The degradation is localized rather than uniform: the first and last layers are fragile, the middle is not, and swapping hurts less than dropping. Measured layer by layer across four GPT-2 and four Pythia models and confirmed on HellaSwag, ARC-Easy and LAMBADA. Robustness grows with depth, so it is not a small-model artifact. | Active |
| THEORY-038 | A decoder cannot compose over a long context in few layers, because each position forgets what it forwarded | Chen et al. (2024), LIT-464 — causal masking makes a decoder a line of forgetful communicating players, one epoch per layer, and sequential composition needs more epochs than a constant-depth model has. An unconditional bound on what can be expressed, not on what can be learned. | Active |
| THEORY-041 | Left unconstrained, a transformer’s matrices drift into a badly conditioned, rank-deficient shape, and the norms are the part nothing was managing | Loshchilov et al. (2024), LIT-472 — trained GPT embeddings form a hyper-ellipsoid with a high condition number and attention matrices whose singular values suggest rank deficiency; renormalizing after training narrows the gap without closing it. Proposed, because the evidence is correlational: these models are worse conditioned and they also train slower, and nothing yet connects the two. | Proposed |
| THEORY-046 | Uniform-state discrete diffusion is the argmax of a Gaussian diffusion, which is why Gaussian technique transfers to it and not to masked diffusion | Sahoo et al. (2025), LIT-479 — taking the argmax of a Gaussian diffusion’s latents carries its marginals onto those of a uniform-state discrete diffusion, under a reparameterization of the noise schedule, and the discretized process satisfies the defining ODE of a discrete diffusion. So it is one. Masked diffusion has no such preimage. | Active |
| THEORY-050 | Parameters carry memorization and parallel computation carries reasoning | Chen et al. (2025), LIT-486 — the exchange rate between parallel streams and parameters is fitted higher on code (0.39) than on general text (0.33), and downstream a 1.6B model at P = 8 matches a 4.4B model on coding but only a 2.8B one on general tasks. Two independent measurements pointing the same way, and an account the authors themselves call a conjecture. | Proposed |
| THEORY-056 v2 | Distinct state machines agree at chance on long random inputs, so a learner that reads only statistics loses the signal as the sequences get longer | Giapitzakis, Fountoulakis, Nichani and Lee (2025), LIT-495 — reading a uniformly random word drives a random walk on S_N × S_N, and the probability two distinct semiautomata land in the same state is 1/N + error with |error| ≤ (1 − 1/2N)^T. The quantity that identifies which machine you are looking at decays exponentially in sequence length, so more data of this kind is less evidence, not more. | Proposed |
| THEORY-064 | Transformers are biased toward low-sensitivity functions, and that bias is what their robustness is made of | Vasudeva et al. (2024), LIT-525 — the probability that a model’s output changes under a random perturbation of one input token is lower for transformers than for MLPs, CNNs, ConvMixers or LSTMs, on vision and language alike. Low sensitivity is a simplicity bias in established senses, it provably implies robustness in the Boolean case, and it tracks flatness. Proposed: the theory is Boolean and NTK, the experiments are not, and the metric is defined relative to a perturbation distribution nobody has swept. | Proposed |
| THEORY-065 | Self-attention drives tokens into a few clusters, and the spectrum of the value matrix decides the geometry they land in | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), LIT-528 — with weights held fixed, self-attention is an interacting particle system and its tokens cluster. Which clusters is not arbitrary: V = I sends them to the vertices of a convex polytope, a simple positive leading eigenvalue to at most three parallel hyperplanes, V = −I to a single point at the origin. In one dimension the attention matrix provably becomes low-rank and Boolean — the structure Linformer and LoRA assume and impose. | Proposed |
| THEORY-066 | Pure self-attention collapses every token onto one, and skip connections prevent it by keeping short paths alive | Dong, Cordonnier and Loukas (2021), LIT-530 — decompose a self-attention network into paths, one head per layer, and the residual after removing the best rank-1 approximation shrinks at a cubic rate: doubly exponential in depth. Skip connections create short paths — including one that skips everything — and that is what stops it. MLPs slow it in proportion to their Lipschitz constant. Layer normalization cannot help at all, because it is a right multiplication. | Proposed |
| THEORY-067 | A transformer can run a learning algorithm on a model held in its activations, and the architecture admits several, so a construction identifies none | Three groups have constructed transformer weights that run a named learning algorithm in the forward pass on a model carried in the activations: one gradient step per linear-self-attention layer (von Oswald et al., LIT-533), a Sherman–Morrison ridge update in constant depth and O(d²) width (Akyürek et al., LIT-532), and k Iterative Newton steps in k + 8 layers (Fu et al., LIT-535). The frame is what survives the dispute about which algorithm; the corollary is that no construction can settle that dispute, because the architecture admits all three. | Active |
| THEORY-078 | Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked | Sitzmann et al. (2020), LIT-551 — with weights U(±√(6/n)), a sum of arcsine inputs is about N(0,1), and a unit normal through sin is arcsine again, so every layer sees the same distribution and few pre-activations exceed π. The mechanism is verified at initialization for 6 and 50 layers. The historical claim, that this is why periodic activations “failed to robustly outperform” before, is not tested. | Proposed |
| THEORY-081 | A transformer’s feed-forward layers work as key-value memories: each hidden unit’s input weights match input patterns, and its output weights promote the tokens that follow them | Geva et al. (2021), LIT-575 — read FF(x) = f(xK)V as a memory. Keys fire on recognizable prefix patterns, shallow in lower layers and semantic in upper ones. Values, projected to the vocabulary, favor the tokens that follow those patterns. The evidence is 160 keys of one 16-layer model, with value-key agreement of at most 3.5%, and outputs are mostly compositions. This reading is behind ROME, MEMIT and tools that list features by what they “hear” and “predict”. | Proposed |
| THEORY-083 | A message-passing GNN can distinguish no more graphs than the 1-WL test, and reaches that limit only if its neighbor aggregation is injective on multisets, as a sum followed by an MLP is and a mean or max is not | Xu et al. (2019), LIT-580 — each message-passing layer refines node labels as one 1-WL iteration can at best, so 1-WL bounds what the network separates. Injective aggregation reaches the bound. Sums of an MLP over a multiset are injective. Means lose counts and maxes lose multiplicities. It is a theorem, and it explains the paper’s training-fit ordering and the collapse of mean aggregators to chance on featureless graphs. | Active |
| THEORY-096 | A Pre-LN residual stream’s per-layer change decays as one over root k, so deep blocks cannot refine the representation | Xie et al. (2023), LIT-639 — in a Pre-LN transformer the residual stream accumulates unnormalised, so by block k it has variance ~k and a single block’s contribution is a 1/√k share of it. The change in the normalised input between consecutive blocks decays as O(1/√k), and adding a block to an N−1 block model moves the output O(1/√N). The architecture’s benefit and its cost are the same mechanism. | Proposed |
| THEORY-097 | Attention is an approximation of Kanerva’s sparse distributed memory, which is why it needs normalized vectors and a fitted temperature | Bricken and Pehlevan (2021), LIT-641 — attention’s update rule is SDM’s read operation under two conditions, L² normalized vectors and a fitted softmax temperature. Its one non-trivial consequence is a retrodiction: those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], inside the range SDM’s optimality criteria span. | Proposed |
| THEORY-098 | Softmax attention coefficients must disperse as the number of items grows, so no learned attention circuit stays sharp out of distribution | Veličković et al. (2024), LIT-653. With logits bounded and temperature non-zero, every softmax coefficient is at most (1/n)·exp(δ/θ) — and in a Transformer over a finite vocabulary the logits are always bounded, so the bound applies in every attention head. A head that looks sharp in-distribution is not a robust circuit; it is a circuit whose input size has not grown yet. | Active |
| THEORY-101 | A large trained ViT repurposes redundant patch tokens as scratch space for global information, which is what its feature-map artifacts are | Darcet et al. (2023), LIT-662. The high-norm tokens that spoil ViT feature maps are not damage: they sit on patches redundant with their neighbours, they have discarded their own local content, and they carry global information instead. The model found spare capacity and used it. The account’s own prediction — give the model dedicated slots and the artifacts should move there — was tested and holds. | Active |
| THEORY-102 | A causal mask implicitly encodes absolute position, because a token can count how many predecessors it may attend to | Haviv et al. (2022), LIT-665. A decoder-only transformer given no positional encoding recovers absolute position within four layers, and the reason is the mask: a token that can count its attendable predecessors knows its index. The account’s prediction is that removing the mask removes the mechanism, and a bidirectional MLM without an encoding does not merely degrade — it fails, at 147.18 perplexity against 4.00. | Active |
| THEORY-103 v2 | A transformer stores a fact where it needed it, so two-hop composition generalizes only to facts it already saw as a second hop | Wang et al. (2024), LIT-667. Composition and comparison are both two-fact tasks that both arrive by grokking, and only one of them generalizes to facts held out of the training compositions. The difference is where the circuit keeps the facts: composition’s is sequential and ends up with a second copy of the atomic facts in the upper layers, containing only those that appeared as a second hop; comparison’s is parallel and keeps one copy downstairs. So the failure is not an inability to compose — the first hop works on held-out facts — but a missing copy, and tying the two halves of the stack together unlocks it. | Proposed |
| THEORY-125 | One affine autoregressive flow block makes every conditional a single Gaussian and cannot approximate an arbitrary density, while two blocks in opposite orders make every conditional but the last a Gaussian mixture and three make all of them one | Gu et al. (2025), LIT-785, §3.1, Prop. 1 and App. A.1. In one affine autoregressive block each coordinate is a mean plus a scale times fresh Gaussian noise, both functions of the coordinates before it, so every conditional is one Gaussian and no multimodal conditional can be represented. A second block in the opposite order feeds later latents into those parameters and turns each conditional into an infinite Gaussian mixture, except the last coordinate’s; a third restores that one. The one-block half is proved and is what TarFlow (LIT-794) measured, FID 267 at matched depth. The universality half is a sketch. | Proposed |
| THEORY-126 | A more expressive architecture scales better by lowering the data coefficient rather than the exponent, because it can learn more of the discrete tasks in language-model data from the same tokens | Merrill et al. (2026), LIT-799, §4.2 and App. E. The quantization model of scaling laws treats language modelling as many discrete tasks with power-law frequencies. If a fraction ε of tasks is inexpressible, and those tasks need more tokens or parameters or give less loss reduction, lowering ε shifts the loss curve down without changing the exponent. The hybrid’s fitted data coefficient B is lower than the transformer’s (83.7 against 94.9, non-overlapping CIs), and the exponents overlap. But the instantiation that fits was chosen after the fits, and the same paper finds that removing negative eigenvalues barely changes the scaling. | Proposed |
| THEORY-129 | Interleaving attention with a negative-eigenvalue delta-rule recurrence expresses problems neither layer type can express alone, because state tracking and recall come from different layers and one alternation composes them | Merrill et al. (2026), LIT-799, §3.3–3.4 and App. B. Fixed-depth transformers lie in TC⁰ and cannot track state (if TC⁰ ≠ NC¹). Bounded-state recurrences cannot recall from a long context. A Gated DeltaNet with negative eigenvalues tracks state, and attention recalls. One alternation of the two solves state-based recall, which neither can alone, and with padding a hybrid recognizes all of NC¹. The proofs are conditional on a complexity conjecture. The synthetic evidence does not separate expressivity from learnability, and the account does not say the expressive gap is what makes hybrids better language models. | Proposed |