Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Attention techniques — attention variants and alternative mechanisms, implementation optimizations, context length.
70 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-008 v2 | Attention Is All You Need | Vaswani et al. (2017), ARXIV-1706.03762. Introduces transformer architecture. | Active |
| LIT-020 v4 | Pay Less Attention with Lightweight and Dynamic Convolutions | Wu et al. (2019), ARXIV-1901.10430. Alternative to self-attention. | Active |
| LIT-024 v2 | Fast Transformer Decoding: One Write-Head is All You Need | Shazeer (2019), ARXIV-1911.02150. Introduces multi-query attention. | Active |
| LIT-033 v4 | Longformer: The Long-Document Transformer | Beltagy et al. (2020), ARXIV-2004.05150. Efficient attention for long sequences. | Active |
| LIT-074 v3 | FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Dao et al. (2022), ARXIV-2205.14135. IO-aware attention implementation. | Active |
| LIT-100 v2 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | Ainslie et al. (2023), ARXIV-2305.13245. Efficient attention variant. | Active |
| LIT-106 v2 | FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning | Dao et al. (2023), ARXIV-2307.08691. Improved attention parallelism. | Active |
| LIT-112 v2 | Efficient Memory Management for Large Language Model Serving with PagedAttention | Kwon et al. (2023), ARXIV-2309.06180. Introduces vLLM. | Active |
| LIT-115 v2 | Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture | Fu et al. (2023), ARXIV-2310.12109. Efficient alternative to attention. | Active |
| LIT-133 v2 | Kimi Linear: An Expressive, Efficient Attention Architecture | Kimi Team (2025), ARXIV-2510.26692. Kimi Delta Attention extends Gated DeltaNet with a channel-wise forgetting gate; interleaved with MLA at 3:1 in a 48B-total, 3B-active model trained on 1.4T tokens, the hybrid beats full attention on short-context, long-context and RL evaluations while cutting KV cache by up to 75% and reaching up to 6.3× the decoding throughput at 1M tokens. | Active |
| LIT-137 v2 | Gated Delta Networks: Improving Mamba2 with Delta Rule | Yang et al. (2024), ARXIV-2412.06464. Gating erases memory fast and the delta rule writes to it precisely; combining them in one recurrence, with a chunkwise parallel algorithm for the hardware, beats both Mamba2 and DeltaNet on language modelling, retrieval, length extrapolation and long context at 1.3B/100B tokens, and the paper’s own hybrids with sliding-window attention or Mamba2 layers do better still. | Active |
| LIT-138 | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | Qiu et al. (2025), ARXIV-2505.06708. Thirty gating variants compared on 15B MoE and 1.7B dense models over 3.5T tokens: a head-specific sigmoid gate applied after scaled dot-product attention consistently improves quality, stabilises training, tolerates larger learning rates and scales better, and removes attention sinks and massive activations. | Active |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-142 | DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models | DeepSeek-AI (2025), ARXIV-2512.02556. DeepSeek Sparse Attention: a small FP8 lightning indexer scores every preceding token and only the top-k KV entries enter the attention, fitted onto V3.1’s MLA in multi-query mode by continued pretraining with a 2.1B-token indexer warm-up under frozen dense attention — the sparse mechanism V4 compresses further. | Active |
| LIT-143 | Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention | Yuan et al. (2025), ARXIV-2502.11089. A sparse attention trained from the start rather than applied after: coarse token compression for global context, fine top-k block selection for precision, and a sliding window for locality, with kernels balanced for arithmetic intensity; matches or beats full attention on general, long-context and reasoning evaluations with large speedups at 64k in forward, backward and decoding. | Active |
| LIT-161 v2 | Mamba: Linear-Time Sequence Modeling with Selective State Spaces | Gu and Dao (2023), ARXIV-2312.00752. Make the SSM parameters functions of the input and the model can decide what to remember; a hardware-aware recurrent kernel keeps it fast without convolutions. | Active |
| LIT-162 v2 | Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | Dao and Gu (2024), ARXIV-2405.21060. SSMs and attention variants are two decompositions of the same class of structured semiseparable matrices; the duality yields Mamba-2, whose core layer is 2–8× faster. | Active |
| LIT-165 v2 | Mamba-3: Improved Sequence Modeling using State Space Principles | Lahoti et al. (2026), ARXIV-2603.15569. Three changes from an inference-first view — a more expressive discretised recurrence, a complex-valued state update for state tracking, and a MIMO formulation that buys accuracy without decode latency. | Active |
| LIT-174 | DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model | DeepSeek-AI (2024), ARXIV-2405.04434. Where Multi-head Latent Attention comes from: compress the KV cache into a latent vector, cutting it 93.3% and raising maximum generation throughput 5.76×. | Active |
| LIT-176 | Native Hybrid Attention for Efficient Sequence Modeling | Du et al. (2025), ARXIV-2510.07019. Hybridise within the layer instead of between layers: a linear RNN maintains long-term KV slots, a sliding window supplies short-term tokens, and one softmax attends over both. | Active |
| LIT-177 | Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention | Li et al. (2026), ARXIV-2606.26560. The delta rule corrects only at the address it is writing to, so stale content elsewhere can never be removed; EDA adds an erase step at an independently chosen address. | Active |
| LIT-190 | Massive Activations in Large Language Models | Sun et al. (2024), ARXIV-2402.17762. A handful of activations in an LLM run up to 100,000× larger than the rest, at fixed feature dimensions and a few token positions, and they act as fixed biases rather than input-dependent features: setting them to their own mean leaves the model intact, setting them to zero destroys it. They are how the model smuggles an attention bias into a mechanism that has none, and giving attention an explicit learnable bias makes them disappear at no cost in quality. | Active |
| LIT-191 | Efficient Streaming Language Models with Attention Sinks | Xiao et al. (2023), ARXIV-2309.17453. Window attention collapses when the first few tokens leave the cache — Llama-2-13B goes from 5.4 to 5158 perplexity — because a softmax must put its mass somewhere and models learn to dump the surplus on whatever is globally visible, which is the initial tokens. Keeping four of their KV entries alongside a rolling window restores it at no finetuning cost, and a single learnable sink token trained in from the start replaces all four. | Active |
| LIT-192 | Extending Context Window of Large Language Models via Positional Interpolation | Chen et al. (2023), ARXIV-2306.15595. Down-scale the position indices to fit the window the model was trained on, instead of extrapolating past it. LLaMA 7B-65B reach 32768 tokens — 16× — after fine-tuning for under 1000 steps, where naive fine-tuning at the longer length moved the effective window from 2048 to 2560 in more than 10000 batches. The argument is about smoothness: interpolating a learned attention-score function between points it already fits is bounded, extrapolating it is not. | Active |
| LIT-193 | YaRN: Efficient Context Window Extension of Large Language Models | Peng et al. (2023), ARXIV-2309.00071. Position interpolation stretches every RoPE dimension equally, which destroys the high-frequency components and stalls around a scale factor of 8. YaRN scales by wavelength instead — leaving dimensions that never complete a rotation alone, interpolating the ones that do — and adds a temperature on the attention logits implemented for free inside the rotary embedding. Llama 2 to 128k with 10× fewer tokens and 2.5× fewer steps than PI. | Active |
| LIT-194 | Gated Linear Attention Transformers with Hardware-Efficient Training | Yang et al. (2023), ARXIV-2312.06635. Linear attention was slower in practice than optimised softmax attention despite better asymptotics, because nobody had made it I/O-aware. FlashLinearAttention fixes that — faster than FlashAttention-2 as a standalone layer even at 1K — and the algorithm generalises to a data-dependent forget gate. The resulting GLA Transformer matches LLaMA-architecture baselines, beats Mamba on throughput at matched size, and generalises from 2K training to over 20K. | Active |
| LIT-195 | Parallelizing Linear Transformers with the Delta Rule over Sequence Length | Yang et al. (2024), ARXIV-2406.06484. The delta rule fixes what linear attention is worst at — associative recall — but its original algorithm did not parallelize over sequence length, so it never left small scale. A reparameterization as products of Householder matrices makes it chunkwise, and a 1.3B DeltaNet on 100B tokens beats Mamba and GLA. It also reports the hybrid result: interleaving with sliding-window or two global attention layers beats a strong Transformer baseline. | Active |
| LIT-204 | Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models | De et al., Google DeepMind (2024), ARXIV-2402.19427. Two models: Hawk, a pure RNN with gated linear recurrences that exceeds Mamba’s reported downstream numbers; and Griffin, which mixes those recurrences with local attention and matches Llama-2 on over 6× fewer tokens. Scaled to 14B, with the sharding recipe published, matching Transformer hardware efficiency in training and beating it on inference latency and throughput. | Active |
| LIT-207 v2 | The Impact of Positional Encoding on Length Generalization in Transformers | Kazemnejad et al. (2023), ARXIV-2305.19466. Five positional schemes trained from scratch under identical hyperparameters — absolute, T5’s relative bias, ALiBi, rotary, and none at all — and the one that generalizes to unseen lengths is none at all. It also reports that such a model represents both absolute and relative position anyway, which is why removing the encoding removes a constraint rather than information — though the absolute half of that is Haviv et al. 2022 (LIT-665), a year earlier and with the mechanism. | Active |
| LIT-208 | Rope to Nope and Back Again: A New Hybrid Attention Strategy | Yang et al. (2025), ARXIV-2501.18795. The paper that says what NoPE and RoPE layers are each for, by measuring where their attention mass goes: NoPE layers retrieve, RoPE layers attend locally, and the two roles are cleaner when the RoPE layers are confined to a sliding window. The resulting architecture — one full-attention NoPE layer per three windowed RoPE layers, 8B parameters, 5T tokens — beats a full-attention RoPE model on long and short context, at about a quarter of the KV cache. | Active |
| LIT-209 | SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling | Puvvada et al. (2025), ARXIV-2504.08719. The same layout as LIT-208, reached independently and three months later — NoPE global layers interleaved 1:3 with sliding-window RoPE layers — plus the two things that paper does not have: a dynamic scaling of attention scores at inference, which is what keeps the NoPE layers working far past the training length, and a demonstration that an existing 8B RoPE model pretrained on 15T tokens can be converted to this architecture by continued training without losing its short-context scores. | Active |
| LIT-210 | Round and Round We Go! What makes Rotary Positional Encodings useful? | Barbero et al. (2024), ARXIV-2410.06205. Opens up a trained Gemma 7B to ask what RoPE is actually doing, and the answer is not the one RoPE’s own paper gave: attention does not reliably decay with distance, and the model instead splits the frequency band in two — the highest frequencies build robust positional attention heads, the lowest carry semantic content in channels that are provably not robust over long context. Removing the lowest frequencies (p-RoPE) fixes that and improves perplexity at 2B, and p is a dial whose ends are exactly RoPE and NoPE. | Active |
| LIT-225 | Generating Long Sequences with Sparse Transformers | Child et al. (2019), ARXIV-1904.10509. The paper the record’s sliding window comes from. Factorizes causal attention into two steps - a local window of the previous l positions and a second head that reaches across windows - giving O(n*sqrt(n)) and, on two of three datasets, a LOWER loss than the dense attention it replaces. | Active |
| LIT-387 | LoFTR: Detector-Free Local Feature Matching with Transformers | Sun et al. (2021), ARXIV-2104.00680. Drop the keypoint detector: establish dense matches coarsely and refine them, with self and cross attention producing descriptors conditioned on BOTH images. The argument is that the detector is the binding constraint — it cannot produce repeatable interest points in low-texture regions, which is exactly where matching is hard. | Active |
| LIT-413 | Attention Is Off By One | Miller (2023), a blog post. Proposes softmax₁(x)ᵢ = exp(xᵢ)/(1 + Σⱼ exp(xⱼ)) — one added to the denominator — so a head can decline to attend at all, and calls the result QuietAttention. It reports no experiments and says so: “I’m 99.44% sure that it will resolve the outlier feedback loop” is the entirety of the evidence. Filed because it is the clearest statement of the mechanism the record holds in THEORY-019 and because it is what most people have actually read, not because it established anything. | Active |
| LIT-414 | Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing | Bondarenko et al. (2023), ARXIV-2306.12929, NeurIPS 2023. The diagnosis the attention-sink literature rests on, and two architectural fixes with the experiment attached. A head trying to learn a no-op must drive its softmax input ever larger to approximate exact zeros, which is what creates the outliers that break quantization. Clipped softmax and gated attention each remove them: BERT-base goes from 1294 W8A8 perplexity to 4.55, with max infinity-norm from 735 to 20. Its gated attention is a per-head sigmoid on the attention output — the recommendation SOTA-134 makes, two years earlier, as Qiu et al. themselves say. | Active |
| LIT-428 | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | Katharopoulos et al. (2020), ARXIV-2006.16236. Where linear attention comes from, and the identity the record argues about in twelve documents without being able to cite it: drop the softmax for a kernel feature map, associativity reorders the product to O(N), and the autoregressive case is then literally an RNN. | Active |
| LIT-429 | Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence | Peng et al. (2024), ARXIV-2404.05892. RWKV-5 and RWKV-6, and the two changes the line is still built on: the state becomes matrix-valued rather than vector-valued, and the decay becomes a function of the input rather than a learned constant. | Active |
| LIT-430 | RWKV: Reinventing RNNs for the Transformer Era | Peng et al. (2023), ARXIV-2305.13048. The paper the RWKV line starts from, and the first to take the architecture to 14B: AFT’s position bias becomes an exponential decay over relative position, which makes the layer trainable in parallel and runnable as a constant-memory recurrence. | Active |
| LIT-431 | An Attention Free Transformer | Zhai et al. (2021), ARXIV-2105.14103. Drops the query-key product entirely: keys and values are combined with a learned position bias and the query enters element-wise. The record holds it because it is where RWKV’s letters come from — R, W, K, V is this construction renamed. | Active |
| LIT-521 | Taming Transformer Without Using Learning Rate Warmup | Qi et al. (2025), ARXIV-2505.21910 — attention entropy collapse comes in two modes, and only one crashes the model: sparse and low-rank is fatal, sparse alone is benign. The driver is spectral energy concentration in W_q^T W_k. Suppressing it by bounding the learning rate with Weyl’s inequality trains ViT, Swin and GPT without warmup, matching warmed-up AdamW on all five configurations. Read as NOTE-266. | Active |
| LIT-523 | Stabilizing Transformer Training by Preventing Attention Entropy Collapse | Zhai et al. (2023), ARXIV-2303.06296 — proves a tight lower bound on attention entropy that falls like Ω(Tσe^{-σ}) in the spectral norm of W_K W_Q^T, then removes the growth by reparameterizing every linear layer as γ·W/σ(W). A ViT trained this way reaches 82.2% on ImageNet without pre-LN, warmup, weight decay or an adaptive optimizer. Read as NOTE-265. | Active |
| LIT-527 | Leaner Transformers: More Heads, Less Depth | Saratchandran, Teney and Lucey (2025), ARXIV-2505.20802 — a theorem that more attention heads lower the condition number of the attention block, and twelve redesigned architectures trading heads for layers. In the five configurations where the MLP width is held fixed, accuracy holds or improves at 29–53% fewer parameters. Read as NOTE-269. | Active |
| LIT-528 | The emergence of clusters in self-attention dynamics | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), ARXIV-2305.05465 — treat tokens as interacting particles and self-attention as their dynamics. Tokens cluster, and the spectrum of the value matrix decides the limiting geometry: V = I gives the vertices of a convex polytope, a simple positive leading eigenvalue gives at most three parallel hyperplanes, V = −I gives one cluster at the origin. In one dimension the attention matrix provably converges to a low-rank Boolean matrix. Read as NOTE-272. | Active |
| LIT-529 | Linformer: Self-Attention with Linear Complexity | Wang, Li, Khabsa, Fang and Ma (2020), ARXIV-2006.04768 — the measurement the record needed, attached to a method it has already moved past. SVD of the attention matrix in RoBERTa-base and RoBERTa-large over 10k sentences shows a long-tail spectrum at every layer, head and task, skewing further in higher layers; Theorem 1 gives a rank Θ(log n) approximation. Read as NOTE-273. | Active |
| LIT-530 | Attention is not all you need: pure attention loses rank doubly exponentially with depth | Dong, Cordonnier and Loukas (2021), ARXIV-2103.03404 — strip the skip connections and MLPs out of a transformer and its output converges doubly exponentially to a rank-1 matrix with identical rows. Skip connections stop it, MLPs slow it, and layer normalization provably does nothing — it is a right-multiplication, and right-multiplication cannot raise rank. Read as NOTE-274. | Active |
| LIT-533 | Transformers learn in-context by gradient descent | von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov and Vladymyrov (2022), ARXIV-2212.07677, ICML 2023 — the paper that named the mesa-optimizer. Proposition 1 constructs key, query, value and projection matrices for which one linear self-attention layer is exactly one gradient-descent step on an implicit linear model, and a single trained layer is shown to find approximately those weights. Deeper trained models do not match plain gradient descent; they match GD++, gradient descent with an iterative curvature correction — a result of this paper’s own. Read as NOTE-280. | Active |
| LIT-638 | A Length-Extrapolatable Transformer | Sun et al. (2022), ARXIV-2212.10554. Defines attention resolution — a scalar for how well an attention pattern distinguishes token distance, estimable before training — and measures it across the extrapolation boundary. RoPE collapses 0.91 → 0.08 at twice its training length; ALiBi holds 0.81 → 0.88 and is the only incumbent that does. The record holds both papers and four practices about RoPE, and had no measure of this. | Active |
| LIT-640 v3 | Query-Key Normalization for Transformers | Henry, Dachapally, Pawar and Chen (2020), ARXIV-2010.04245. The origin of QK-norm, which thirty documents in this record name and SOTA-192 recommends while sourcing an adopter. ℓ₂-normalize each query and key along the head dimension before the dot product, then scale by a learnable parameter instead of dividing by √d — so the temperature becomes a trained quantity rather than a constant. +0.928 BLEU averaged over five low-resource pairs. | Active |
| LIT-641 | Attention Approximates Sparse Distributed Memory | Bricken and Pehlevan (2021), ARXIV-2111.05498. Attention’s update rule is Kanerva’s sparse distributed memory read, given two conditions: L² normalized vectors and a fitted softmax temperature β. The retrodiction is the interesting part — those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], the range that interpolates between SDM’s optimality criteria. | Active |
| LIT-653 | Softmax is not Enough (for Sharp Size Generalisation) | Veličković, Perivolaropoulos, Barbero and Pascanu (2024), ARXIV-2410.01104. Twenty-three practices in this record name the softmax and no note held it. This one proves a limit on it: in a Transformer over a finite vocabulary, attention coefficients must decay towards zero as the number of items grows, so a head that is sharp in distribution cannot stay sharp on larger inputs. Max retrieval falls from 98.6% at 16 items to 12.4% at 16,384. | Active |
| LIT-655 | Self-attention Does Not Need O(n²) Memory | Rabe and Staats (2021), ARXIV-2112.05682. Six months before FlashAttention: attention in O(1) memory per query, self-attention in O(log n), a practical O(√n) implementation, exact rather than approximate, 59× less memory overhead at sequence length 16,384. And the number the record needed and did not have: on TPU the same algorithm is “within a few percent of the runtime of the standard implementation” — the memory saving is algorithmic and the speedup is not. | Active |
| LIT-661 | Photorealistic Video Generation with Diffusion Models | Gupta, Yu et al., Stanford and Google (2023), ARXIV-2312.06662. A latent video diffusion transformer over a causal image-and-video encoder. It alternates per-frame spatial attention with local spatiotemporal windows. Two single-run ablations bear on the record’s video practices. Adding a separate corpus of ~970M image-text pairs to ~89M text-video pairs takes zero-shot UCF-101 FVD from 598.8 to 344.5 at 419M (Table 5), with compute not stated. And windowed attention matches full self-attention on UCF-101 (FVD 55.3–59.6 against 59.9) while training 1.7x faster (Table 3b). | Active |
| LIT-666 v2 | Sparse Distributed Memory | Kanerva (1988), MIT Press. An associative memory in a high-dimensional binary space: a pattern is written to every fixed-address neuron within a Hamming radius of it, and read back as the thresholded sum over the neurons within that radius of the query. Held because LIT-641 shows attention’s softmax approximates this read — the weighting it induces, the intersection of two hyperspheres, decays near-exponentially in distance — and the record carried that account (THEORY-097) without its source. Filed from secondary accounts; the book has not been read here. | Active |
| LIT-708 | A Controlled Study of Attention-Only Transformers | Ndubuaku et al. (2026), ARXIV-2607.18363. Removing every feed-forward layer from a decoder costs 0.47 nats at matched depth and 0.26 at matched training FLOPs, but only 0.0055 nats (0.27% of loss) at matched parameters once the freed budget goes into attention depth: 20 attention-only layers against 4 standard blocks, 24M parameters, 105B tokens of a synthetic reasoning corpus, with a separate learning-rate sweep for each arm. The remaining gap sits on low-context tokens, where the answer must come from the weights. A pre-registered test on fineweb-edu predicted 0.02–0.05 nats and measured 0.040. Removing QK-norm made the attention-only stack diverge; residual gates and ReZero changed nothing. Everything is at or below 87M parameters. | Active |
| LIT-709 | Primer: Searching for Efficient Transformers for Language Modeling | So et al. (NeurIPS 2021), ARXIV-2109.08668. An evolutionary search over TensorFlow primitives, at a fixed 24-hour training budget, finds two changes that carry most of the gain. One squares the feed-forward ReLU. The other adds a causal width-3 depthwise convolution after each per-head Q, K and V projection (MDHA). The only direct activation comparison has squared ReLU ahead of SwiGLU and ReGLU, in one 110M C4 figure. “4.2×” is against the original ReLU T5 at 537M. Against the SwiGLU + RMSNorm baseline the saving is 2× at equal final quality. On encoder–decoder masked LM, Primer-EZ does no better than that baseline. | Active |
| LIT-711 | Do Transformer Modifications Transfer Across Implementations and Applications? | Narang et al. (2021), ARXIV-2102.11972. About fifty published transformer modifications reimplemented in one codebase (T5 / Mesh TensorFlow, 223M encoder-decoder) with hyperparameters held fixed, scored on C4 pre-training loss, SuperGLUE, XSum, WebQuestions and WMT’14 En-De. Most do not help. The ones that do are small (GLU-variant activations, RMSNorm, untied embeddings), cost parameters or time (Switch/MoE, product-key memory, mixture of softmaxes), or were invented in the same codebase. The Universal Transformer, ReZero, Fixup, cross-layer sharing, the Evolved Transformer and lightweight/dynamic convolutions all come out worse than the vanilla baseline. The caveat that travels least: the positive results are also mostly same-codebase — GLU Variants (LIT-030) was run in this codebase at this exact configuration — so the paper is a strong negative test and a weak positive one. | Active |
| LIT-723 | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows | Liu, Lin, Cao, Hu et al. (2021), ARXIV-2103.14030. A vision transformer with a hierarchical feature pyramid and self-attention confined to shifted local windows, linear in image size rather than quadratic. Nine documents here run experiments on it and none said what it is. Its ablation is the reason to hold it: every design element is measured on three tasks, and the absolute position embedding this record recommends elsewhere helps ImageNet classification (+0.4 top-1) while hurting COCO detection (−0.2 box AP) and ADE20K segmentation (−0.6 mIoU). | Active |
| LIT-726 | Talagrand Meets Talagrand: Upper and Lower Bounds on Expected Soft Maxima of Gaussian Processes with Finite Index Sets | Chu et al. (2025), ARXIV-2502.06709. Take a centered Gaussian process on a finite set T with max variance σ² and minimum separation a. The expected softmax-weighted average g(β) = E[Σ_t X_t e^{βX_t}/Z] satisfies g(β) ≤ √(2σ²·E D(ν_β‖ν₀)) for every β (Thm 1, D = KL from uniform). For β ≥ β*, g(β) ≥ c·a·√(E D(ν_β‖ν₀)), with c the Sudakov constant (1/17 via Marcus–Rosen) (Thm 2). For i.i.d. N(0,σ²) logits there is an exact identity, g(β) = βσ²(1 − E‖ν_β‖²₂) (Lemma 2). The quenched free energy is bounded above by √(2σ²·E D_{1/2}(ν_β‖ν₀)), with an i.i.d. lower bound of (cσ/2)√(E D_{1/2}) (Thms 4–5). | Active |
| LIT-738 | Shortformer: Better Language Modeling Using Shorter Inputs | Press, Smith and Lewis (2020), ARXIV-2012.15832. Training a transformer language model on short subsequences first and then at the full length is faster and better than training at the full length throughout: on WikiText-103 (247M parameters, final length 3,072), 128 tokens for the first 50 epochs gives 17.52 dev perplexity against the baseline’s 18.65. The paper credits the two-stage routine to BERT, which used it only for speed. Also introduces position-infused attention. | Active |
| LIT-747 | Perceiver: General Perception with Iterative Attention | Jaegle et al. (2021), ARXIV-2103.03206 — the Perceiver. A learned latent array of N=512 queries cross-attends to the raw input (50,176 ImageNet pixels), so attention costs O(MN) rather than O(M²), and a deep latent transformer then runs at O(N²) per layer, independent of input size. With 2D Fourier-feature positions and no convolution it reaches 78.0% top-1 on ImageNet without pretraining, unchanged when the pixels are permuted, while ViT-B-16 and ResNet-50 given the same features fall to 61.7 and 39.4. The same architecture runs on audio, video and point clouds. | Active |
| LIT-748 | SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator | Chen et al. (2024), ARXIV-2412.12094 — SepLLM. Let each token attend only to a few initial tokens, the previous n tokens, and every earlier separator token (punctuation and whitespace), on the hypothesis that a segment’s content is condensed into the separator that ends it. Training-free on Llama-3-8B-Instruct it keeps GSM8K-CoT at 77.18 against 77.79 with 47% of the KV cache, where StreamingLLM at the same budget gets 70.89. Trained that way from scratch at 160M it reaches lower loss than full attention at equal FLOPs. | Active |
| LIT-760 | Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks | Lee et al. (2018), ARXIV-1810.00825 — the Set Transformer (ICML 2019). An attention encoder–decoder for set inputs. Its induced set attention block (ISAB) routes a set of n elements through m learned “inducing points”, so each block costs O(nm) rather than O(n²), and its pooling by multihead attention (PMA) aggregates the set with k learned seed vectors in place of mean or max pooling. Attention-based pooling is the larger win on tasks whose outputs interact (amortized clustering), and 16 inducing points matched or beat full self-attention there; each block still maps back to the input size, so a stack of them scales with n at every layer. | Active |
| LIT-763 | Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems | Zhdanov et al. (2025), ARXIV-2502.17019 — Erwin. Build a ball tree over an irregular point set so that every ball at a level holds the same number of points in contiguous memory, then run full self-attention inside each ball, coarsen and refine through the tree in a U-Net, and alternate with a tree built on a rotated copy of the cloud so balls exchange information. Runtime fits n^1.054. Best on cosmology at large training sets, on EAGLE turbulence (3× faster and 8× less memory than EAGLE’s own model), and on ShapeNet-Car; a clear failure on the Airfoil PDE mesh. | Active |
| LIT-783 | Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency | Zheng et al., Tsinghua and NVIDIA (2025), ARXIV-2510.08431. sCM made to run on 14B text-to-image and text-to-video models, with a FlashAttention-2 JVP kernel under FSDP and context parallelism. Pure sCM loses fine detail and temporal coherence there. Adding a DMD loss at weight 0.01 (rCM) fixes it. Four-step Wan2.1 VBench is 84.43 at 1.3B against a re-implemented DMD2’s 84.56, and 84.92 at 14B; GenEval tracks the teacher at 1–4 steps. The diversity advantage over DMD2 is shown in samples only. One run per row. | Active |
| LIT-790 | Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models | Lu and Song, OpenAI (2024), ARXIV-2410.11081. Continuous-time consistency models had been unstable. A trigonometric flow parameterization, identity time input with positional embeddings, normalized tangents, learned loss weights and a Flash Attention JVP make them train to 1.5B parameters. Continuous time beats every discrete step count tried. Two-step FID is 2.06 on CIFAR-10, 1.48 on ImageNet-64 and 1.88 on ImageNet-512, against teachers at 2.01, 1.33 and 1.73. Every model, “consistency training” included, starts from the pretrained diffusion model. | Active |
| LIT-799 | Olmo Hybrid: From Theory to Practice and Back | Merrill et al., Ai2 and Lambda (2026), ARXIV-2604.03444. Olmo 3 7B with its sliding-window layers replaced by Gated DeltaNet (negative eigenvalues), at 3:1 with full attention, trained to 6T tokens. It reaches Olmo 3’s MMLU in 49% fewer tokens and beats it in every domain after mid-training, though data mix, schedule and mid-training batch also changed. Scaling fits from 60M to 1B give the hybrid a lower data coefficient (83.7 against 94.9, non-overlapping CIs); the projected 1.3–1.9× token savings have CIs that include 1 at every size but 3B. A theorem shows hybrids express a task neither parent can. Removing negative eigenvalues barely changes the scaling, which weakens the expressivity explanation. | Active |
| LIT-800 | Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM | Kozyrev and Maiboroda, Minima (2026), ARXIV-2609.04098. Public 4-bit builds of Qwen3.8-27B kept its 48 Gated DeltaNet layers in 8 or 16 bits for fear that errors in a recurrence compound. Quantizing all 496 linear layers to NVFP4 W4A4 by calibration alone stays within seed noise of BF16 on five task suites (average −0.52) at 17.5 GiB. The perplexity cost is the largest of the recipes compared, +0.72 at 4K, but it shrinks along a 32K window. In replayed layers the gate projections everyone protected are the least sensitive. State error holds at a plateau of about 12.6% over 32K tokens, and a state impulse is erased within hundreds to a few thousand steps. One model, one format. | Active |
| LIT-801 | Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension | Bertsch, Soldaini, Gormley, Neubig, Hajishirzi, Lo and Groeneveld, Ai2 and CMU (2026), ARXIV-2608.10296. Twenty-six 7–8B models, 140B tokens each on identical data, extended identically to 64K. QK norm, GQA, sliding-window layers and a 4K pretraining length each cost long-context ability, and together they compound: HELMET at 32K runs from 56.4 down to 29.9. Counting how many of the four a model has predicts that score with in-sample R² 0.67 (0.61 leave-one-out). Pretraining loss predicts it at R² 0.29. QK norm is the largest single cost, about 4 to 6 points. GQA plus sliding windows costs about 9, against 1.1 for windows alone. One seed per configuration, and initialization alone moves scores by 7.7% of the range on average. | Active |
| LIT-802 | Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | Hatamizadeh, Choi and Kautz, NVIDIA (2026), ARXIV-2605.22791. The delta rule’s scalar beta decides two things at once: how much to erase along the key and how much of the value to write. Splitting it into a channel-wise erase gate on the key and a channel-wise write gate on the value, on top of KDA’s channel-wise decay, gives the best average of six recurrent mixers at 1.3B/100B tokens and matched state size (53.11 against 52.39 for Mamba-3 MIMO and 52.28 for KDA; 53.97 hybrid). One run each. The erase gate carries most of the gain. Its multi-key retrieval lead is real, but Gated DeltaNet beats it on the word-needle task at 4K, 60.6 against 31.8. | Active |