Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov and Max Vladymyrov (2022), ICML 2023, PMLR 202:35151–35174 — ARXIV-2212.07677
Key takeaways
- The identity (Proposition 1). With W_K = W_Q = [[I_x, 0], [0, 0]], W_V = [[0, 0], [W₀, −I_y]] and P = (η/N)I, a linear self-attention step sends every token e_j = (x_j, y_j), the query included, to (x_j, y_j − ΔW x_j), where ΔW = −(η/N) Σ_i (W₀x_i − y_i)x_iᵀ is one gradient step on L(W) = (1/2N) Σ ‖Wx_i − y_i‖². A gradient step on the weights is re-expressed as a change to the data, and the attention layer makes exactly that change. The updated model is never stored; its effect appears only in the query token’s output.
- Found, not only constructible. A single LSA layer trained on random linear-regression tasks reaches the loss of one tuned gradient step, its predictions and input sensitivities align with the construction’s (cosine near 1), and a scale correction plus linear interpolation between trained and constructed weights keeps the loss unchanged, in and out of the training input range.
- Deeper is not plain gradient descent. Recurrent two-layer and five-layer LSA models beat K steps of gradient descent and align instead with GD++, a step on data transformed by H(X) = I − γXXᵀ with η and γ tuned per step: a curvature correction.
- MLP then attention (Proposition 2) gives gradient descent on a linear read-out of deep features, i.e. kernel regression with k(x, y) = m(x)ᵀm(y); trained models on sine-wave regression resemble a meta-learned MLP with one gradient step on its output layer.
- Building the tokens (Proposition 3, §4). If inputs and targets come in separate tokens, a first (softmax) layer can copy them together and a second layer then performs the step; trained two-layer models do this, matching one GD step, not two.
Standing in the record
Held in both records under ADR-013. The anthology holds it as ANTH-LIT-533,
read for what the forward pass of a transformer computes, and has rejected
the identification of in-context learning with gradient descent
(ANTH-THEORY-068) while keeping the frame that a transformer can run a
learning algorithm on a model carried in its activations (ANTH-THEORY-067).
It is filed here at the owner’s request on 2026-10-09, from the bibliography
of the owner’s manuscript What Survives Translation? (work
what-survives-translation), which considered it and dropped it from the
final reference list. The second question, the one this record asks, is the
reading on this record’s terms: what Proposition 1 says about the
difference between conditioning a fixed system on a context and changing
the system’s parameters. That reading is NOTE-621, and it is a source
of THEORY-169.
The two readings do not disagree. The anthology’s rejection concerns the claim that trained transformers in general do gradient descent; this record uses only the exact construction and the single-layer result, which the anthology’s entry keeps.