anthology-of-the-sota

LIT-008: Attention Is All You Need

2 min read

Properties7
StatusActive
Version2
Tagsmodel-architecture, attention-techniques
Compared againstLIT-709 — Primer: Searching for Efficient Transformers for Language Modeling
Cited as “introduced by” bySOTA-050 — Scale attention weights by 1/sqrt(head_dim)
Cited as “paper” byNOTE-011 — Attention Is All You Need
Cited as “source” bySOTA-050 — Scale attention weights by 1/sqrt(head_dim)

Graph View

Vaswani et al. (2017) — ARXIV-1706.03762

Key takeaways

  • Relating two positions takes a constant number of operations, against ConvS2S’s linear and ByteNet’s logarithmic growth in distance — which is what “makes it more difficult to learn dependencies between distant positions” in those models
  • The cost of that is “reduced effective resolution due to averaging attention-weighted positions”, and multi-head attention exists to counteract it
  • The 1/√d_k scaling has a stated reason. Unscaled, “the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients”. The footnote gives the arithmetic: with q and k components independent, mean 0, variance 1, q·k has mean 0 and variance d_k
  • Dot-product over additive attention is chosen on a practical argument — “much faster and more space-efficient, since it can be implemented using highly optimized matrix multiplication code” — not a quality one
  • The paper hedges C3 as “we suspect”, which is weaker than the flat statement the field repeats

Standing in the anthology

The source of SOTA-050.

Read in full for #114; the reading is NOTE-011. The practice is confirmed, and gains the reason behind the formula it states.

The variance argument is a footnote in the original and is the whole justification. A reader who knows only 1/√d_k cannot tell whether it generalises; a reader who knows that q·k has variance d_k can.

Primer (LIT-709) measures the output of an architecture search in the compute needed to reach this architecture’s final quality. Primer-EZ, which adds only a squared-ReLU feed-forward activation and a causal depthwise convolution after each head’s Q, K and V projections, needs 1.75–2.34× less compute than a vanilla Transformer at 35M on LM1B, depending on codebase and hardware, and at least 1.8× less at 110M on C4 and PG19. The full Primer reaches the original T5’s quality at 537M in 4.2× less, a fixed-size figure that grows with training length. All of it is decoder-only language modelling. On encoder–decoder masked LM, Primer-EZ does no better than a SwiGLU-and-RMSNorm Transformer.

Backlinks