Max Staats, Matthias Thamm and Bernd Rosenow (2024), NeurIPS 2025 — ARXIV-2410.17770
Key takeaways
- The null and its two edges (Section 3). For an m × n matrix with i.i.d. zero-mean entries and q = n/m fixed, the singular values follow the Marchenko–Pastur law on [ν₋, ν₊], ν± = σ̃(1 ± √(1/q)). Freshly initialized transformer matrices match it closely. A square matrix has ν₋ = 0, so learning can only push values out above the bulk; a rectangular one has ν₋ > 0, so it can push them out below as well. The paper reads agreement with the law as randomness and departures as learning.
- Departures carry data directions, in some matrices (Section 4, Appendices A–C). The overlap O_k of each right singular vector with the eigenvectors of the covariance of the activations entering the matrix (WikiText; BookCorpus in Appendix B) rises above a random-vector 3σ band for the above-bulk outliers generally, and for the below-bulk outliers of the Up-Projection (Pythia, Llama) and Gate-Projection (Llama). It does not for the Attention-Output matrix in any of the three models, nor for Llama’s Down-Projection, nor for the below-bulk outliers of Llama’s rectangular Key and Value.
- The ablation (Section 5, Tables 1, 2 and 5). Zeroing one decile of rank-ordered singular values in every matrix of one type: in square matrices the damage falls monotonically from the largest decile down; in rectangular ones the smallest decile is always worse than some larger ones. Llama-3 8B on GSM8K (43.2%): the smallest decile of the Down-Projection leaves 2.0%, worse than every decile but the largest; the same decile of the square Attention-Output leaves 40.0%.
- Order of operations (Figure 6, BERT on BoolQ, RTE, SST-2). Pruning the smallest decile and then fine-tuning mostly recovers; fine-tuning and then pruning does not. The authors use this to reconcile Hsu et al. (small values matter) with Sharma et al. (removing them helps).
- A solvable mechanism (Section 6, Appendix F). A Gaussian Gibbs ensemble over the first-layer weights W of a two-layer linear student learning a rank-one linear teacher has mean W₀ ∝ v uᵀ and row covariance Σ = α𝟙 − α²β/(1 + αβ) vvᵀ: the loss suppresses the noise along the second-layer direction v. That negative spike in the noise lets the teacher’s singular value sit below the bulk, and the student’s singular vectors overlap the teacher’s more the farther the outlier sits below the bulk, falling to chance where it merges.
Standing in the record
Filed on 2026-10-09 at the owner’s request, with no context stated; it came in a batch with arXiv 2509.24914, 2502.09863, 2504.12916 and 2205.10343. It is read here on its own merits. See the curation entry of that day.
It is also held in the Anthology of the SOTA as ANTH-LIT-517, read there (ANTH-NOTE-263) for what the bottom of the spectrum implies for low-rank-plus-quantization methods and for its account of steep weight spectra. The second question here, per ADR-013, is the reading on this record’s terms: what the paper shows about using a random-matrix null to locate information in a learned matrix, and how far that inference holds. Its point for this record is that the null spectrum, not the magnitude order, is the reference: the Frobenius-optimal truncation of Eckart and Young (LIT-330) discards by size, while this paper locates information by distance from the bulk, at either edge (NOTE-659). It sits in the line of random-matrix readings of trained networks that the record holds through Martin and Mahoney (LIT-672), the HTSR grokking study (LIT-369) and Papyan’s Hessian spectra (LIT-617, LIT-619), whose bulk-plus-outliers picture THEORY-078 states for the Fisher.