Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Attention techniques — attention variants and alternative mechanisms, implementation optimizations, context length.
11 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-019 | A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink | Xiao et al. (2023), LIT-191 — attention scores are normalized to sum to one, so a head with nothing it needs to attend to still has mass to shed, and it learns to dump it on the positions every query can see, which under causal masking are the first few tokens. The evidence that it is positional rather than semantic: replacing the first four tokens with linebreaks moves Llama-2-13B from 5.40 to 5.60 perplexity, while evicting them from the cache moves it to 5158. LIT-190 finds the same concentration in the residual stream, and SOTA-134’s gate removes sinks rather than relocating them, which is what the account predicts. | Active |
| THEORY-061 | Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product | Zhai et al. (2023), LIT-523 — with σ = ‖W_K W_Q^T‖₂·‖XX^T‖₂, the minimum attainable attention entropy behaves like Ω(Tσe^{−σ}), and the bound is tight. So a saturated softmax is not bad luck: growing weights force it. Adaptive optimizers make it worse, because the idealized Adam update’s spectral norm grows like √w in the width. Proposed, because a later paper exhibits a stable network in precisely the collapsed state. | Proposed |
| THEORY-062 | What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such | Qi et al. (2025), LIT-521 — an attention map that is sparse but not low-rank has near-zero entropy and trains fine; one that is sparse and low-rank crashes. So entropy is the symptom. The cause offered is spectral energy concentration of W_q^T W_k, which in crashed runs collapses into fewer than 10 directions. Proposed: it is one group’s counterexample to another group’s account, at 300M and below. | Proposed |
| THEORY-065 | Self-attention drives tokens into a few clusters, and the spectrum of the value matrix decides the geometry they land in | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), LIT-528 — with weights held fixed, self-attention is an interacting particle system and its tokens cluster. Which clusters is not arbitrary: V = I sends them to the vertices of a convex polytope, a simple positive leading eigenvalue to at most three parallel hyperplanes, V = −I to a single point at the origin. In one dimension the attention matrix provably becomes low-rank and Boolean — the structure Linformer and LoRA assume and impose. | Proposed |
| THEORY-066 | Pure self-attention collapses every token onto one, and skip connections prevent it by keeping short paths alive | Dong, Cordonnier and Loukas (2021), LIT-530 — decompose a self-attention network into paths, one head per layer, and the residual after removing the best rank-1 approximation shrinks at a cubic rate: doubly exponential in depth. Skip connections create short paths — including one that skips everything — and that is what stops it. MLPs slow it in proportion to their Lipschitz constant. Layer normalization cannot help at all, because it is a right multiplication. | Proposed |
| THEORY-097 | Attention is an approximation of Kanerva’s sparse distributed memory, which is why it needs normalized vectors and a fitted temperature | Bricken and Pehlevan (2021), LIT-641 — attention’s update rule is SDM’s read operation under two conditions, L² normalized vectors and a fitted softmax temperature. Its one non-trivial consequence is a retrodiction: those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], inside the range SDM’s optimality criteria span. | Proposed |
| THEORY-098 | Softmax attention coefficients must disperse as the number of items grows, so no learned attention circuit stays sharp out of distribution | Veličković et al. (2024), LIT-653. With logits bounded and temperature non-zero, every softmax coefficient is at most (1/n)·exp(δ/θ) — and in a Transformer over a finite vocabulary the logits are always bounded, so the bound applies in every attention head. A head that looks sharp in-distribution is not a robust circuit; it is a circuit whose input size has not grown yet. | Active |
| THEORY-117 | A separator token condenses the content of the segment it closes, rather than only receiving attention a head has nowhere else to put | Chen et al. (2024), LIT-748 — SepLLM. Attention in Llama-3-8B-Instruct concentrates on punctuation and whitespace, and the paper reads that as each segment’s content being condensed into its closing separator. Its evidence is what masking costs: keeping only separators, a window and three initial tokens holds GSM8K-CoT at 77.18 against 77.79 with 47% of the KV, where StreamingLLM at the same budget gets 70.89 and keeping every fourth token instead gets 72.71. The record already holds the opposite reading — LIT-414 found heads parking no-op attention on [SEP], periods and commas, whose values are small, and THEORY-019 is the sink account it belongs to. No experiment in either paper separates the two. | Proposed |
| THEORY-127 | QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass | Bertsch et al. (2026), LIT-801, §5.4 and Fig. 7, with the dispersion bound of Veličković et al. (2024), LIT-653. In 26 controlled 7B runs, QK norm costs about 4 to 6 HELMET points at 32K. The QK-norm models have higher attention entropy, less attention-sink mass and less attention on the needle, and sink mass correlates positively with long-context score (R² 0.38). The account: QK norm bounds the logit spread, so a head can be neither sharp at long range nor able to park surplus mass on a sink. All of the evidence is correlational, and QK norm also changed norm order in the cleanest pair. | Proposed |
| THEORY-128 | A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies | Kozyrev and Maiboroda (2026), LIT-800, §5.3, Table 6, Fig. 1. In Qwen3.8-27B’s Gated DeltaNet layers, run in FP32 lockstep over 32K tokens, 4-bit quantization noise holds state error at a plateau of about 12.6%. A 1% state impulse falls to 1/e in 80–1,382 steps, where the decay gates imply horizons of 1,895–61,659 tokens. The authors attribute the extra forgetting to the delta rule’s overwrite along each new key. No arm removes the delta rule, so the attribution is inferred, not measured. | Proposed |
| THEORY-129 | Interleaving attention with a negative-eigenvalue delta-rule recurrence expresses problems neither layer type can express alone, because state tracking and recall come from different layers and one alternation composes them | Merrill et al. (2026), LIT-799, §3.3–3.4 and App. B. Fixed-depth transformers lie in TC⁰ and cannot track state (if TC⁰ ≠ NC¹). Bounded-state recurrences cannot recall from a long context. A Gated DeltaNet with negative eigenvalues tracks state, and attention recalls. One alternation of the two solves state-based recall, which neither can alone, and with padding a hybrid recognizes all of NC¹. The proofs are conditional on a complexity conjecture. The synthetic evidence does not separate expressivity from learnability, and the account does not say the expressive gap is what makes hybrids better language models. | Proposed |