nucleation

LIT-857: Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws

4 min read

Properties4
StatusActive — read 2026-10-09 (NOTE-661); worth reading as a solvable model in which the spectrum of trained attention weights is derived, not just measured: for empirical risk minimization of one tied softmax attention layer on Gaussian sequences, with n of order d², the learned query–key map is, in law, a rescaled ReLU of the target plus a semicircle noise bulk shifted by a threshold, so its outliers are recovered target directions and its bulk is finite-sample noise. Weight decay on the factor is exactly a nuclear-norm penalty on the product. With a power-law target, modes are recovered one by one as the threshold moves, which gives heavy-tailed learned spectra and power-law learning curves whose exponents match the LASSO rates. The main results are replica/AMP "Claims", not theorems, checked against Adam runs at d = 100–400.
Tagslearning-theory, mathematics, anthology-candidate
Cited as “paper” byNOTE-661 — Single-Head Attention in High Dimensions
Cited as “source” byTHEORY-180 — In a tied single-head attention layer trained by ridge-penalized empirical risk minimization on high-dimensional Gaussian sequences, weight decay is a nuclear-norm penalty on the query–key map, and the learned map is a soft-thresholded noisy copy of the target: its outliers are recovered target directions and its bulk is finite-sample noise, THEORY-726 — Weight decay on factors is a nuclear-norm penalty on their product: with tied weights λ‖W‖²_F = λ‖WWᵀ‖_*, and for untied factors the least (‖U‖²_F + ‖V‖²_F)/2 over UVᵀ = M is ‖M‖_*, so for any loss that sees the weights only through the product, square-penalized training is nuclear-norm-penalized training of a product of bounded rank

Graph View

Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala and Lenka Zdeborová (2025), arXiv preprint, v2 2026 — ARXIV-2509.24914

Key takeaways

  • An exact (non-rigorous) account of ERM in one attention layer. The model is f(x) = softmax_β((x W Wᵀ xᵀ − centring)/√(dp)) x, query and key tied, value the identity, trained on square loss plus λ‖W‖²_F. Inputs are T i.i.d. Gaussian tokens in ℝ^d; targets are the same form with a rank-κ₀d matrix S0 plus symmetric Gaussian noise of variance Δ on the pre-activations. As d, n, p → ∞ with α = n/d², κ = p/d and κ₀ fixed and T finite, Claim 3.1 gives the global minimum’s train loss and test error through a six-dimensional variational problem Φ(Σ, m, q, Σ̂, m̂, q̂), valid when a replicon condition holds (evaluated numerically, always satisfied in the plots). The derivation maps the problem to multi-token matrix sensing with GOE sensing matrices and an AMP algorithm; the paper calls it a claim with a proof sketch, not a theorem.
  • Weight decay is a nuclear norm on the attention map (Appendix A, an exact identity): with S = WWᵀ, ‖W‖²_F = ‖S‖*, and for untied factors min (‖U‖²_F + ‖V‖²_F)/2 over UVᵀ = M is ‖M‖*. Consequences in the model: every width p/d ≥ 1 − F(2λ̃/m̂) attains the same error, so mild rank constraints cost nothing; and training the factors beats training S directly with a Frobenius penalty at its best λ, though the latter is strictly more expressive. For p0 ≪ d the factorized model needs n = O(d·p0) samples, the unfactorized one n = O(d²) (Appendix D: in the noiseless ridgeless limit its error is Q0(1 − 2α)).
  • The learned spectrum, in closed form (Claim 4.1). In law, WᵀW/√(pd) = η·ReLU(S0 + δZ − εI), Z ∼ GOE(d), ReLU on eigenvalues, with η = m̂/Σ̂, δ = √q̂/m̂, ε = 2λ̃/m̂ from the same variational problem. δ is a finite-sample noise level, ε a regularization threshold. As α grows the spectrum passes through rank collapse (a delta at zero), bleed-out and outliers, and for a Marchenko–Pastur target splits into two bulks (Figure 1). The pattern is in “qualitative agreement” with spectra measured in trained transformers; nothing more is claimed.
  • Thresholds. For λ → 0⁺ the interpolation threshold is α_interp = ∂₁J(δ̄, 0)/(2δ̄(T² + T − 2)), and for Δ = 0 a perfect-recovery threshold α_perf is given by Eqs. (16)–(18), both by reducing softmax to linear attention below interpolation (Appendix C). Curves for different T collapse when α is rescaled by (T² + T − 2)/2, the number of constraints each sample imposes.
  • Spectrum and error, term by term (Section 4, Conjecture E.1). For targets with eigenvalues decaying faster than i^(−1/2), the excess error splits into overfitting (the noise bulk), underfitting (target directions still in the bulk), approximation (error along recovered outliers) and a mismatch term. Outliers are the learned features.
  • Scaling laws from sequential recovery (Section 5, Appendix F). For S0 with eigenvalues √d·i^(−γ), γ > 1/2, the threshold ε moves down the target spectrum as n grows, admitting weaker modes one at a time; the accumulated outliers make heavy-tailed learned spectra. The excess error scales as n^(−f(γ)) with regime-dependent exponents (Eqs. 24–25), e.g. (n/d)^(−1+1/(2γ)) for d ≪ n ≪ d² and small λ, the same rates as for LASSO and matrix compressed sensing in Defilippis et al., and at optimal regularization the classical minimax rate n

Standing in the record

Filed on 2026-10-09 at the owner’s request, with no stated context, in a batch with arXiv 2502.09863, 2504.12916, 2205.10343 and 2410.17770. It cites the last of these, Staats, Thamm and Rosenow’s random-matrix analysis of transformer weights, as its empirical reference for the spectra it derives.

Read on 2026-10-09 (NOTE-661). It is the source of THEORY-180, the record’s statement of the thresholded spectral law and what it does not reach. It sits with the record’s other proportional-limit accounts of learning: Martin and Mahoney’s statistical mechanics of generalization (LIT-672), and the exact double-descent papers LIT-679 and LIT-683 behind THEORY-106. Its interpolation peak (Figure 6) is a neighbouring case of THEORY-106 in a model with learned features rather than random ones; see the NOTE.

Anthology. Not held there. It is ML theory about attention, scaling laws and emergence, all subjects anthology topics hold, and it draws a practical comparison (factorized query–key training against direct training of the product), so anthology-candidate.

Backlinks