Source
Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762.
Why the square root, specifically
The dot product of two vectors whose components are independent with unit variance has variance equal to their dimension, so raw attention logits grow like the head dimension. Feed those into a softmax and it saturates: one weight goes to nearly one, the rest to nearly zero, and the gradient through the softmax vanishes.
Dividing by √d_head restores unit variance, which keeps the softmax in the range where it still has a gradient. The scaling is not a tuning constant — it is the factor that makes the initialisation-time variance independent of head dimension, which is why it is the same in every implementation and why nobody tunes it.
The practice originates in LIT-008, which defines scaled dot-product attention
as softmax(QKᵀ/√d_k)V, with the scaling part of the operation rather than a
trick applied to it. Its footnote is the argument above: with q and k
components independent, mean 0 and variance 1, q·k has variance d_k. The
paper offers the saturation account as a suspicion (“we suspect”), backed by
additive attention outperforming unscaled dot-product attention at larger
d_k, not by a measurement of softmax gradients.
Where it stops being sufficient
At scale it does not hold on its own. Attention logits can still grow during training as the query and key projections align, and the practices that bound them — SOTA-131’s QK-Clip, QK-norm and its relatives — exist because the initialisation-time argument says nothing about step 100,000.
That is the useful reading of this practice today: a necessary normalisation that everything assumes, plus the observation that it is not the whole of the problem it addresses. Filed under model-stability rather than attention-techniques for that reason, which is one of the seams the unbound-lineage report keeps returning.