Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto and Carlos Guestrin (2024), ICML 2025, PMLR 267:57503–57522 — ARXIV-2407.04620
Key takeaways
- Every sequence layer is a state, an update rule and an output rule. An RNN compresses context into a fixed-size state; self-attention keeps the whole context (the KV cache) and scans it. A TTT layer makes the state the weights W of a model f, the update W_t = W_{t−1} − η∇ℓ(W_{t−1}; x_t), and the output z_t = f(θ_Q x_t; W_t). Even at test time the layer trains a fresh sequence of weights for every input sequence.
- Two loops. The inner loop trains W on reconstruction, ℓ(W; x_t) = ‖f(θ_K x_t; W) − θ_V x_t‖²; the outer loop trains θ_K, θ_V, θ_Q (and the rest of the network, the initial weights and a token-dependent learning rate) on next-token prediction. Outer-loop parameters are hyper-parameters of the inner learning problem; W is state, not a parameter.
- Theorem 1. Linear f, batch gradient descent with η = 1/2 and W₀ = 0 gives z_t = Σ_{s≤t} (θ_V x_s)(θ_K x_s)ᵀ(θ_Q x_t), which is linear attention.
- Theorem 2. The Nadaraya–Watson estimator with kernel exp((θ_K x)ᵀθ_Q x′) and labels θ_V x_s, as a nonparametric inner learner, gives softmax self-attention. Training is storing the context; prediction is kernel smoothing over it.
- What changes the results. Mini-batch inner updates (b = 16) instead of batch gradient descent give the largest gain (perplexity 14.05 → 12.35 in the 125M ablation); LayerNorm and a residual in f the second.
- Long context. TTT-Linear and TTT-MLP keep reducing perplexity with token index out to 32k, like the Transformer and unlike Mamba. TTT-MLP is costly in wall-clock time, which the authors leave open.
Standing in the record
Filed on 2026-10-09 at the owner’s request, from the bibliography of the
owner’s manuscript What Survives Translation? (work
what-survives-translation), which considered it and dropped it from the
final reference list. Read on its own terms (NOTE-620).
It carries anthology-candidate: as an architecture paper its subject is
one anthology topics hold, and the anthology names it without filing it.
What this record takes from it is the two theorems, which state attention
as learning on the context. They are a source of THEORY-169.
The name “test-time training” covers two things: here, an inner model whose weights are a layer’s state; in Akyürek et al. (LIT-796), a temporary update to the network’s own weights. Akyürek et al. say so in a footnote.