nucleation

LIT-1959: Attention Is All You Need

1 min read

Properties5
StatusActive — read in full 2026-11-03 (NOTE-1501), from arXiv v7; worth reading as the source of query–key–value attention, whose factorization into compatibility weights and transported values is the structure TERM-064's note of 2026-11-02 compares to legibility. The QK/OV vocabulary is Elhage et al.'s, not this paper's.
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-1501 — Vaswani et al. — Attention is all you need
Cited as “source” byTHEORY-1342 — In Transformer attention the weights each query puts on the positions depend only on query–key compatibility, softmax(QKᵀ/√d_k), and the output is those weights applied to the values, so for a fixed input what a head selects and what it transports are separate factors; because queries, keys and values are all projections of the same input, the separation holds only with the input, and so the pattern, held fixed

Graph View

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017). Attention Is All You Need. arXiv preprint; NeurIPS 2017. arXiv:1706.03762.

Read in full on 2026-11-03 (NOTE-1501; THEORY-1342).

Key takeaways

  • Attention(Q, K, V) = softmax(QKᵀ/√d_k)V: which positions are read is set by queries and keys, what is read by values (pp. 3–4).
  • Multiple heads because one head’s convex averaging cannot attend to several subspaces at once (p. 5).
  • Its training recipes are practice, the anthology’s.

Standing in the record

The source of query–key–value attention, the other side of the owner’s U23 analogy. The record holds the QK/OV analysis of it (LIT-1534, THEORY-988) but not the paper.

Backlinks