Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Attention techniques — attention variants and alternative mechanisms, implementation optimizations, context length.
29 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-050 v2 | Scale attention weights by 1/sqrt(head_dim) | Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762. | Active |
| SOTA-085 v4 | Use flash attention for all attention computations when hardware supports it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-086 v4 | Tiling size should match hardware SRAM size | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-087 v4 | Recompute attention during backward pass instead of storing it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-106 v3 | Use flash-attention-2 over original flash-attention when available | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Active |
| SOTA-107 v3 | Keep sequence lengths multiple of 128 for best performance | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — The 128 is traced and belongs to a different quantity. Every occurrence in the source is a head dimension or a block size; none is a sequence length, and “multiple of” and “divisible” appear zero times |
| SOTA-108 v3 | Pad attention masks to block boundaries for better hardware utilization | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — Not in the cited paper. “padding”, “padded”, “pad”, “block boundar” and “divisible” appear zero times each; the body’s reasoning about skipping fully-masked blocks is correct and is not the source’s |
| SOTA-109 v3 | Prefer GQA to MQA or MHA | Ainslie et al. (2023), LIT-100 — group the query heads and give each group one key/value head: multi-query’s cache saving without multi-query’s quality loss, and uptrainable from an existing multi-head checkpoint. Still the default for a model not paying MLA’s implementation cost. | Active |
| SOTA-110 v3 | Consider Monarch Mixer in place of attention for very long sequence tasks | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Active |
| SOTA-111 v4 | Use Monarch Mixer where attention’s quadratic cost is what bottlenecks training | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Active |
| SOTA-112 v4 | Combine Monarch Mixer layers with standard attention rather than replacing it wholesale | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Rejected — The source is attention-free and argues for replacement, not combination. “hybrid” occurs zero times in it; the hybrid argument belongs to the attention/SSM literature elsewhere in this record |
| SOTA-131 v4 | When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip) | Kimi Team (2025), LIT-132 — MuonClip carried a 1T/32B MoE through 15.5T tokens with zero loss spikes where plain Muon let attention logits pass 1000; confirmed at 2.8T in LIT-131. | Active |
| SOTA-132 v3 | Interleave linear-attention layers with global attention at about 3:1 instead of using full attention throughout | Kimi Team (2025), LIT-133 — three Kimi Delta Attention layers per gated-MLA layer beat full MLA at 48B/1.4T while cutting KV cache 75%; the layout LIT-131 ships at 2.8T with 69 KDA and 24 MLA layers. | Active |
| SOTA-134 v2 | Gate each attention head’s output with a sigmoid after the scaled dot-product | Qiu et al. (2025), LIT-138 — the best of 30 gating variants at 15B MoE and 1.7B dense over 3.5T tokens: better quality, more stable training, larger tolerable learning rates, no attention sinks; shipped in every Qwen full-attention layer since Qwen3-Next. | Active |
| SOTA-135 v2 | Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes | Yang et al. (2024), LIT-137 — beats Mamba2 and DeltaNet across language modelling, retrieval and long context at 1.3B/100B; the recurrence the Qwen hybrids use for three layers in four, and the one Kimi Delta Attention extends. | Active |
| SOTA-138 v2 | Train sparse attention natively with a learned top-k indexer, warmed up under dense attention | DeepSeek-AI (2025), LIT-142 — a small FP8 indexer scores past tokens and only the top-k enter attention; initialised for 2.1B tokens with dense attention on and the rest frozen; V4 repeats the pattern with a 1T-token dense warm-up. | Active |
| SOTA-139 | Extend the context length in stages during pretraining rather than training at the target length from the start | DeepSeek-AI (2026), LIT-139 — 4K, then 16K, 64K and 1M over 32–33T tokens, with sparse attention introduced at the 64K stage; Kimi K3 (LIT-131) reaches 1M the same way. | Active |
| SOTA-147 v2 | Compress the KV cache into one shared latent vector instead of sharing key and value heads | DeepSeek-AI (2024), LIT-174 — Multi-head Latent Attention projects keys and values into a single low-rank latent and caches that, cutting the KV cache 93.3% and raising maximum generation throughput 5.76× against the same team’s dense 67B. Every DeepSeek model since is built on it. | Active |
| SOTA-153 v4 | Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position | Yang et al. (2025), Puvvada et al. (2025) and the Kimi Team (2025) — in a model that already interleaves full attention with a cheap local mixer, the full-attention layers do not need a positional encoding: the local layers carry position and recency, and the global layers do retrieval better without one. Three groups, three different local mechanisms, the same one-global-per-three-local layout. The payoff is that extending the context needs no RoPE rescaling, because there is no positional parameter left to rescale. | Active |
| SOTA-161 v4 | Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling | Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out. | Active |
| SOTA-176 | Hybridise inside the layer: long-term slots, a sliding window, and one softmax over both, with the window size as the only dial | Du et al. (2025), LIT-176 — long-term context in key-value slots updated by a linear RNN, short-term context from a sliding window, and a single softmax attending over both together, so the long/short weighting is per-token, per-head and content-dependent with no fusion parameters. Every layer stays structurally identical and one hyperparameter slides from purely linear to full attention. | Proposed |
| SOTA-177 v2 | Decouple the erase address from the write address in a delta-rule recurrence | Li et al. (2026), LIT-177 — the delta rule corrects what is stored at the current write address before writing there, so stale information at a different address can only decay passively. EDA runs a targeted erase along a learned erase direction first, then the ordinary corrective write. Best at both 2.5B dense and 25B-A2.8B MoE, and the gain persists after 80B tokens of long-context midtraining. | Proposed |
| SOTA-192 v9 | Normalize the queries and keys before the attention dot product | Active | |
| SOTA-208 | Factorize long-sequence attention into a local window and a second head that escapes it | Child et al. (2019), LIT-225 — ARXIV-1904.10509. Superseded: the field went to exact attention made fast (SOTA-086, SOTA-087) and then to sparsity that is LEARNED rather than fixed (SOTA-138). What survives is the connectivity requirement — a window alone is not the design — which every hybrid the record recommends is an instance of. | Superseded — by SOTA-138 |
| SOTA-248 | Stretch and clip the softmax so an attention head can output exact zeros | Bondarenko et al. (2023), LIT-414 — a head that wants to do nothing has to drive its softmax input to infinity to approximate exact zeros, and that is what creates the activation outliers that break INT8. Stretch the softmax to (γ, ζ) and clip back to (0,1) and zeros become reachable from a finite input. On BERT-base: W8A8 perplexity 1294 → 4.55, max infinity-norm 735 → 20, and the FP16 model gets slightly better rather than worse. Evidence is BERT-base, OPT-125M and ViT-S/16 — small and encoder-heavy. | Proposed |
| SOTA-322 | Trade attention heads for depth: more heads conditions the attention block, and the layers you drop cost nothing | Saratchandran, Teney and Lucey (2025), LIT-527 — more heads lower the condition number of the attention block, which is part of what depth was buying. In the five configurations where the MLP width is held fixed, raising head count and cutting layers holds or improves accuracy at 29–53% fewer parameters — TNT-B goes 65.4M → 30.9M at identical Top-1. | Proposed |
| SOTA-390 v3 | In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost | Proposed | |
| SOTA-391 | Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary | Sun et al. (2022), LIT-638 — attention resolution scores how well an attention pattern distinguishes token distance, and is estimable from the encoding before a run. Measured at the training length and twice it: RoPE 0.91 → 0.08, ALiBi 0.81 → 0.88. The record has four practices about rotary encoding and held no measure of this. | Proposed |
| SOTA-436 | Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size | Jaegle et al. (2021), LIT-747 — the Perceiver. A learned array of 512 latents cross-attends to all 50,176 ImageNet pixels, at a cost of O(MN) rather than O(M²), and a 48-block latent transformer then runs at a cost independent of input size. It matched ResNet-50 and ViT-B/16 at 78.0% top-1 without convolutions, did not beat them, and ran unchanged on audio, video and point clouds. The case for doing it is cost and modality-independence, not accuracy, and the cost advantage is argued from complexity, not measured against a baseline. | Proposed |