attention-techniques.
22 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-005 | FlashAttention | Attention is bounded by HBM traffic rather than FLOPs. Tiling to on-chip SRAM and recomputing the attention matrix in the backward pass gives exact attention in Θ(N²d²M⁻¹) HBM accesses against standard attention’s Θ(Nd + N²) — and Proposition 3 proves no exact algorithm beats that across all SRAM sizes. | Read |
| NOTE-007 | FlashAttention-2 | FlashAttention reached only 25–40% of peak FLOPs/s; the loss was work partitioning between thread blocks and warps, not the algorithm. Three changes — fewer non-matmul FLOPs, parallelism over sequence length, better warp partitioning — give ~2× and 50–73% of peak. | Read |
| NOTE-011 | Attention Is All You Need | Dispenses with recurrence and convolution entirely. The 1/√d_k scaling has a stated reason: with unit-variance components, q·k has variance d_k, and unscaled dot products push the softmax into regions of extremely small gradients. | Read |
| NOTE-014 | Monarch Mixer | One sub-quadratic primitive — Monarch matrices — along both sequence length and model dimension. Matches BERT-base/large on GLUE with up to 27% fewer parameters and 9.1× throughput at 4K, and beats ViT-b by 1% at half the parameters. Attention-free, not hybrid. | Read |
| NOTE-016 | Fast Transformer Decoding: multi-query attention | Incremental decoding is bounded by the memory bandwidth of reloading the keys and values, not by arithmetic. Share one key/value head across all query heads: much faster decoding, “only minor quality degradation”. | Read |
| NOTE-021 | Grouped-Query Attention | MQA is fast and costs quality, and training a separate model for inference is undesirable. Two results: uptrain an existing multi-head checkpoint with 5% of pretraining compute, and use an intermediate number of key-value heads — quality near multi-head at speed near MQA. | Read |
| NOTE-023 | PagedAttention and vLLM | KV cache memory is huge, grows and shrinks dynamically, and was managed as one contiguous block — so fragmentation and duplication capped the batch size. PagedAttention borrows OS paging: near-zero waste, sharing within and across requests, 2–4× throughput at equal latency. | Read |
| NOTE-059 | Longformer: The Long-Document Transformer | Replaces full self-attention with a sliding window plus a small set of task-chosen global positions, giving linear cost in sequence length. The global tokens are the design decision that matters — they are where the task’s inductive bias is stated, and they are why the pattern can be adapted per task rather than fixed. | Read |
| NOTE-065 | Pay Less Attention with Lightweight and Dynamic Convolutions | Shows a depthwise convolution with softmax-normalized weights, shared across channel groups and with a kernel fixed at 31 or less, matches self-attention on translation — 29.7 BLEU on WMT En-De. Dynamic convolution predicts the kernel from the current token alone, so the mixing weights depend on position rather than on the pair of positions being mixed. | Read |
| NOTE-173 | LoFTR: Detector-Free Local Feature Matching with Transformers | Remove the keypoint detector and match densely, coarse-to-fine, with descriptors conditioned on both images by cross attention. The argument is a failure mode rather than an efficiency: detectors cannot emit repeatable points in low-texture regions, so no downstream matcher can recover what was never proposed. | Read |
| NOTE-265 | A proved bound under the entropy-collapse observation, and a reparameterization that removes four crutches | Read on its own merits after being triaged past once. The record already held the failure this paper is about and two remedies for it, and had neither the inequality that makes the failure inevitable nor this third remedy. The accuracy numbers are nearly flat; what changes is how many hyperparameters have to be right. | Read |
| NOTE-266 | Not all entropy collapse crashes a model, and the discriminator is low-rankness | Read after the paper it contradicts. The contribution is a distinction the record did not have: an attention map can be sparse with near-zero entropy and the run is fine, provided it is not also low-rank. That makes entropy the symptom and spectral energy concentration in the query-key product the cause, and it makes a warmup-free optimizer fall out. | Read |
| NOTE-269 | Heads traded for depth, and the five rows where only those two things move | Read on its merits after being triaged past. The headline is an architecture recommendation and the interesting part is the mechanism: more heads lower the condition number of the attention block, which is the same quantity THEORY-041 says drifts the wrong way when nothing manages it. Half the configurations confound the trade by also halving the MLP; the other half do not, and those are the result. | Read |
| NOTE-272 | An idealization that checks its own assumptions against ALBERT, and clusters within a few layers | Read a second time, in full, after being declined on a skim. The decline said the t → ∞ frozen-weights idealization put it too far from a trained model to be evidence about one, and that it underwrites no practice. Both halves were wrong: the record’s own schema says a theory need underwrite nothing, and the paper checks its hypotheses on ALBERT, gives the discrete-time analogue, and says clustering arrives within a few layers. | Read |
| NOTE-273 | The observation outlived the method, and it is the observation the record needed | Read for one figure. The record holds two practices resting on attention being low rank and held no source for the claim; the record also already holds the verdict that this paper’s method was overtaken. Those are different things and the reading keeps them apart. | Read |
| NOTE-274 | The trunk under a term the record was using for two different things | Read because four documents in this record lean on “rank collapse” and none of them held its source. Reading it produced a finding that was not the point of the unit: the record carries that term for two distinct phenomena, and the conflation is the field’s rather than ours. | Read |
| NOTE-280 | The construction is exact, and the trained model it describes is one layer deep | Read as the trunk of the mesa-optimization line. Proposition 1’s weight construction is exact and a single trained linear-self-attention layer approximately finds it. The result that matters most for what came after is four pages later and rarely quoted: deeper trained models do not match gradient descent, they match GD++ — gradient descent on data transformed by I − γXXᵀ — which is the paper’s own evidence that plain GD is the wrong description beyond one layer. | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |
| NOTE-366 | Primer | Squaring the feed-forward ReLU and adding a causal width-3 depthwise convolution after each per-head Q/K/V projection makes decoder-only LMs reach a vanilla baseline’s quality in 1.75–2.3× less compute at 35M, at least 1.8× at 110M and 4.2× at 537M. That last figure is at fixed size, where the factor grows with training length. At 1.9B, against a GELU baseline, it takes ⅓ of the compute. Both changes cost step time and are paid for in the accounting. Squared ReLU beats SwiGLU in exactly one figure, at 110M. Against the SwiGLU + RMSNorm baseline the saving is about 2×, and it vanishes on encoder–decoder masked LM. | Read |
| NOTE-367 | Narang et al., transformer modifications | About fifty transformer modifications reimplemented in one T5/Mesh TensorFlow codebase at 223M parameters, with hyperparameters fixed, on four task families. Most do not beat the vanilla pre-norm, relative-attention baseline. The winners are GLU-variant activations, RMSNorm, untied embeddings, and sparse experts or other variants that cost parameters or time. The Universal Transformer, ReZero, Fixup, block sharing and the Evolved Transformer all lose. The positives are largely same-codebase re-runs, GLU Variants included, so the paper is strong evidence against the losers and weak evidence that the winners transfer. | Read |
| NOTE-368 | Attention-only transformers | A necessity test of the feed-forward layer with three matchings. Deleting it in place costs 0.47 nats. At matched FLOPs the cost is 0.26 nats. At matched parameters, with the budget moved into attention depth, the cost is 0.0055 nats, and it shrinks with tokens. What remains is concentrated on low-context tokens, where recall has to come from the weights. QK-norm is what keeps the attention-only stack trainable. At or below 87M parameters, on a synthetic reasoning corpus. | Read |
| NOTE-378 | Talagrand Meets Talagrand | Take a centered Gaussian process on a finite set T with max variance σ² and minimum separation a. The expected softmax-weighted average g(β) = E[Σ_t X_t e^{βX_t}/Z] satisfies g(β) ≤ √(2σ²·E D(ν_β‖ν₀)) for every β (Thm 1, D = KL from uniform). For β ≥ β*, g(β) ≥ c·a·√(E D(ν_β‖ν₀)), with c the Sudakov constant (1/17 via Marcus–Rosen) (Thm 2). For i.i.d. N(0,σ²) logits there is an exact identity, g(β) = βσ²(1 − E‖ν_β‖²₂) (Lemma 2). The quenched free energy is bounded above by √(2σ²·E D_{1/2}(ν_β‖ν₀)), with an i.i.d. lower bound of (cσ/2)√(E D_{1/2}) (Thms 4–5). | Read |