Contribution
“The first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution” (p. 2), which trains in 12 hours (base) or 3.5 days (big) on eight P100 GPUs and sets new translation records.
Key insight
Attention separates which positions are read from what is read: “An attention function can be described as mapping a query and a set of key-value pairs to an output … The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key” (pp. 3–4). Every pair of positions is one operation apart (Table 1).
Assumptions
- Encoder and decoder stacks of N = 6 identical layers, d_model = 512, residual connections and layer norm, LayerNorm(x + Sublayer(x)) (§3.1).
- Causal masking in the decoder self-attention (§3.1, §3.2.3).
- Position enters only through added encodings PE(pos, 2i) = sin(pos / 10000^{2i/d_model}), PE(pos, 2i+1) = cos(·) (§3.5).
Key results
- Scaled dot-product attention (eq. 1, p. 4): Attention(Q, K, V) = softmax(QKᵀ/√d_k)V. The scale counters dot products whose variance grows as d_k for unit-variance components (footnote 4).
- Multi-head (p. 5): MultiHead(Q, K, V) = Concat(head₁, …, head_h)W^O, headᵢ = Attention(QWᵢ^Q, KWᵢ^K, VWᵢ^V); h = 8, d_k = d_v = 64. “Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.”
- Three uses (§3.2.3): encoder–decoder attention, encoder self-attention, masked decoder self-attention.
- Complexity (Table 1): self-attention O(n²·d) per layer, O(1) sequential operations and path length; recurrent O(n·d²), O(n), O(n).
- Translation (Table 2, §6.1): EN–DE 28.4 BLEU (big), 27.3 (base); EN–FR 41.8 (big) in Table 2 and the abstract. §6.1 says “a BLEU score of 41.0” for the same model, an inconsistency within v7.
- Ablations (Table 3): single-head is 0.9 BLEU worse than the best; too many heads also hurt; reducing d_k hurts (“determining compatibility is not easy”); learned positional embeddings ≈ sinusoids.
- Parsing (Table 4): WSJ-only 91.3 F1; semi-supervised 92.7.
- “Self-attention could yield more interpretable models … individual attention heads clearly learn to perform different tasks” (p. 7), shown only by visualizations.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Attention-only models match or beat recurrent and convolutional ones on WMT 2014 at lower cost | strong | Table 2 |
| C2 | Multiple heads help over one | moderate | Table 3 rows (A), single runs on the dev set |
| C3 | Self-attention gives constant path length between positions | strong | construction, Table 1 |
| C4 | Heads learn interpretable, distinct functions | weak | a few visualizations |
Concepts
- Query, key, value — vectors; the weights come from query–key compatibility, the content from the values.
- Head — one attention function on learned projections of Q, K, V.
Connections
Bahdanau et al. (additive attention), end-to-end memory networks, ByteNet and ConvS2S, the parallel non-recurrent predecessors. In the record the circuit analysis of attention is Elhage et al. (LIT-1534, THEORY-988).
Bearing on the record
- TERM-064’s note of 2026-11-02 (“QKV, MVC and legibility”). The paper
supports its structural reading: the weights depend on queries and keys
alone and the output is the weighted values, so selection and transport are
separate factors of eq. (1). Three refinements:
- The words “QK circuit” and “OV circuit” are Elhage et al.’s (LIT-1534); Vaswani et al. speak of a compatibility function and values. The note’s attribution “(Vaswani et al., … Elhage et al.)” reads as if both used the split; only the second names it.
- The note’s “exact only with attention patterns frozen” is right and is visible in eq. (1): Q, K and V are all projections of the same input, so changing the input changes both what is selected and what is moved.
- The paper’s own justification of multiple heads, that one head averages away information from different subspaces (p. 5), is a point about a single bounded observer’s limits that the note can use: a head’s softmax is a convex combination, so one head cannot both select sharply and report from several positions.
- No instruction for ML practice beyond architecture. The paper is a method paper whose practice content (warm-up schedule, label smoothing, checkpoint averaging) belongs to the anthology.
Limitations
- Translation and parsing only; no language modeling.
- Single runs; no variance reported.
- The interpretability remark rests on visualizations.
Open questions
The authors’: restricted local attention for long inputs; other modalities; less sequential generation.