nucleation

LIT-624: Towards Understanding the Spectral Bias of Deep Learning

4 min read

Properties4
StatusActive — read 2026-10-03 (NOTE-483): main text and Appendix A in full, the proofs in Appendix B followed at the level of their lemmas. Worth reading as the theorem that makes "spectral bias" a statement about NTK eigendirections: for a two-layer ReLU network trained by gradient descent, the residual projected onto the top r_k eigenfunctions of the NTK integral operator shrinks like (1 − λ_(r_k))^T (Theorem 4.2), for any target, with sample size governed by the eigengap λ_(r_k) − λ_(r_k+1). On uniform spherical data the eigenfunctions are spherical harmonics, the odd degrees above 1 have eigenvalue zero, and µ_k = Ω(max(k^(−d−1), d^(−k+1))) for even k (Theorem 4.3).
Tagslearning-theory, representation-learning, anthology-candidate
Cited as “paper” byNOTE-483 — Towards Understanding the Spectral Bias of Deep Learning
Cited as “source” byTHEORY-084 — Because the Gauss–Newton Fisher JᵀJ and the NTK Gram matrix JJᵀ share their non-zero spectrum, the lazy-regime Fisher of a ReLU network on spherical data inherits the NTK's harmonic-degree blocks, and a spectral threshold on it is well posed only at gaps between blocks, THEORY-086 — In the kernel regime gradient descent fits the target eigenspace by eigenspace of the neural tangent kernel, at rates set by their eigenvalues; on uniform spherical data the eigenspaces are the harmonic degrees, and for fully connected ReLU networks the non-zero eigenvalues decay as k^(−d) at every depth

Graph View

Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou and Quanquan Gu (2019), IJCAI 2021 — ARXIV-1912.01198

Key takeaways

  • Training decomposes along the eigenfunctions of the NTK integral operator L_κ. Let V_(r_k) hold the first r_k eigenfunctions evaluated on the sample, where r_k counts the multiplicities of the first k distinct eigenvalues. Then n^(−1/2)‖V_(r_k)ᵀ(y − ŷ(T))‖ ≤ 2(1 − λ_(r_k))^T n^(−1/2)‖V_(r_k)ᵀ y‖ + ε (Theorem 4.2). This holds when n ≥ Ω̃(ε^(−2) max{(λ_(r_k) − λ_(r_k+1))^(−2), M⁴r_k²}) and the width m is polynomial in T, λ_(r_k)^(−1) and ε^(−1). Unlike most NTK results it assumes nothing about the target: high-frequency content elsewhere does not stop the top eigendirections from being learned.
  • On uniform data on S^d the eigenfunctions are spherical harmonics and the distinct eigenvalues µ_k are indexed by degree (Theorem 4.3). µ₀ and µ₁ are Ω(1), µ_k = 0 for odd k ≥ 3, and for even k, µ_k = Ω(k^(−d−1)) when k ≫ d but Ω(d^(−k+1)) when d ≫ k. The second regime is the authors’ improvement over Bietti and Mairal’s Ω(k^(−d−1)), and they argue it is the practical one.
  • The sample and width needed to learn degree k grow exponentially, in d when k ≫ d and in k when d ≫ k (Corollaries 4.6–4.7). Remark 4.8 argues this is unavoidable, because the number of harmonics of degree below k, r_k, is itself Ω(d^(k−1)), and n samples cannot pin down more than n components. Experiments with a 4,096-unit two-layer network on 1,000 points in R¹⁰ show the residual’s degree-1, 2 and 4 components converging in that order, even when the higher degrees are given larger amplitude, and roughly the same ordering on three non-uniform input distributions (§5).

Standing in the record

Filed on 2026-10-03 at the owner’s request, with Bietti and Bach on the eigenvalue decay of deep NTKs (LIT-608) and Basri et al. on the convergence time per frequency (LIT-612). The owner listed the three beside work on Fisher-spectrum block structure, Papyan’s spectra of deep networks, and model comparison by evidence, which are being filed in parallel.

Of the three, this is the one with a dynamical theorem about a finite network that is not assumed infinitely wide, and it is the one that says most directly what a spectral cut means. Theorem 4.2 is stated for the first r_k eigenfunctions, a set closed under multiplicity, and its sample requirement scales with the inverse square of the gap between the r_k-th and (r_k + 1)-th eigenvalues. A cut through a degenerate block, or between two nearly equal eigenvalues, is exactly where the guarantee fails. A thresholding procedure on this spectrum is well posed at the gaps between degree blocks and nowhere else. The paper does not draw that conclusion; I do, in NOTE-483.

Basri et al. (LIT-612) reached the parity result independently: the bias-free two-layer ReLU NTK has eigenvalue zero on every odd degree k ≥ 3. Cao et al. give it as part of Theorem 4.3 and design their experiments around it. Basri et al. go further and show the null space disappears when a bias is added. No relation is declared between the two. They are near-simultaneous treatments of one object from Arora et al.’s linearised dynamics, which the record does not hold, and neither builds on the other. Bietti and Bach (LIT-608) cite this paper among the works on spectral properties of wide networks and give the exact asymptotic decay where this paper gives lower bounds.

The same picture of learning one eigendirection at a time, largest first, is Simon et al.’s account of self-supervised learning (LIT-242): a linearised Barlow Twins learns the top eigenmodes of a contrastive kernel in sequence. That is the kernel-PCA counterpart of this paper’s kernel regression.

It carries anthology-candidate because its subject is the training dynamics of neural networks, which anthology topics can hold. The anthology holds the NTK paper (ANTH-LIT-360), the learning curves that follow from an NTK spectrum (ANTH-LIT-328), and two later papers that rely on spectral bias as a known effect: Tancik et al. on Fourier features (ANTH-LIT-550) and Vasudeva et al. on low-sensitivity transformers (ANTH-LIT-525).

Backlinks