Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Numerics and precision — how many bits, where, and what that costs — number formats, training precision and the failures it causes, post-training quantization, and the interaction between them.
6 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-019 | A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink | Xiao et al. (2023), LIT-191 — attention scores are normalized to sum to one, so a head with nothing it needs to attend to still has mass to shed, and it learns to dump it on the positions every query can see, which under causal masking are the first few tokens. The evidence that it is positional rather than semantic: replacing the first four tokens with linebreaks moves Llama-2-13B from 5.40 to 5.60 perplexity, while evicting them from the cache moves it to 5158. LIT-190 finds the same concentration in the residual stream, and SOTA-134’s gate removes sinks rather than relocating them, which is what the account predicts. | Active |
| THEORY-042 | An update smaller than the lattice spacing is not merely rounded away — it is cancelled exactly, and carrying the remainder makes the discrete path shadow the continuous one within half a grid cell | Xu et al. (2026), LIT-473 §5 — decompose rounding as identity plus error and expand the trajectory: when every step falls short of the grid, the accumulated quantization loss cancels the accumulated ideal update term for term and the weights never move. Stochastic rounding replaces stagnation with a random walk whose variance grows in T. Carrying the remainder bounds the deviation from the ideal path at Δ/2 for all T. | Active |
| THEORY-059 v2 | A low-rank branch corrects quantization because weight spectra are steep and quantization-error spectra are flat | Li et al. (2024), LIT-512 — two propositions bound the output error by the magnitude of weights and activations, not only by their rounding errors. So a rank-r branch helps exactly when the thing it subtracts has a few dominant singular values. A weight matrix does; a quantization error does not, which is why the same trick applied to the error rather than the weights underperforms. | Proposed |
| THEORY-100 | Activation outliers are a product of pre-training optimization choices, not an emergent property of scale | Ahmadian et al. (2023), LIT-656. The sharp post-training quantization failure above ~6B was described as emergent. Hold the architecture fixed and vary weight decay, dropout, gradient clipping and half-precision format, and the failure moves with those instead: 0.09% degradation against 1.36% at 6B, and a 52B model that loses nothing where OPT-66B loses about 42%. The discontinuity is in the recipe, not the parameter count. | Proposed |
| THEORY-104 | A large guidance scale amplifies the model’s derivatives, which narrows a high-order solver’s convergence radius | Lu et al. (2022), LIT-676. Why the fast diffusion samplers invert under guidance. Classifier-free guidance scales the difference between conditional and unconditional predictions, which amplifies the model’s derivatives as well as its output; a k-th order solver is built from k-th order derivatives, so its convergence radius shrinks fastest for the largest k. At a fixed evaluation budget the higher-order method is then worse than the lower, which is what the measured FIDs show — 13.04, 114.62 and 164.74 at 10 evaluations for orders 1, 2 and 3. | Proposed |
| THEORY-128 | A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies | Kozyrev and Maiboroda (2026), LIT-800, §5.3, Table 6, Fig. 1. In Qwen3.8-27B’s Gated DeltaNet layers, run in FP32 lockstep over 32K tokens, 4-bit quantization noise holds state error at a plateau of about 12.6%. A 1% state impulse falls to 1/e in 80–1,382 steps, where the decay gates imply horizons of 1,895–61,659 tokens. The authors attribute the extra forgetting to the delta rule’s overwrite along each new key. No arm removes the delta rule, so the attribution is inferred, not measured. | Proposed |