nucleation

LIT-1261: On the Spectral Bias of Neural Networks

1 min read

Properties5
StatusActive — read 2026-10-20 (NOTE-983) in full; worth reading for the experiment in which a ReLU network fits low frequencies before high ones even when the high ones are ten times stronger (THEORY-759), for the Fourier analysis of piecewise-linear networks (THEORY-761), and for the observation that the frequencies that are hard to learn depend on how the data's manifold is embedded. Read it knowing that its theorem is proved only for compactly supported networks, that the link from the theorem to the order of learning is an upper bound on speed and not a proof of order, that the volume result in Appendix D does not follow, and that its "manifold complexity" is confounded with the curve's length.
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-983 — Rahaman et al. — On the Spectral Bias of Neural Networks
Cited as “source” byTHEORY-759 — A six-layer, 256-unit ReLU network trained by full-batch Adam to regress a sum of sinusoids at 5 to 50 cycles on [0, 1] fits the components in order of frequency, and keeps that order when the amplitudes rise tenfold with frequency, so in this setting the order of acquisition is set by the learner and not by the strength of the target's components, THEORY-761 — A continuous piecewise-linear function on R^d with compact support has a Fourier transform that, along any fixed direction û, is bounded by C(û)k^(−2), and by C(û)k^(−d−1) whenever û is orthogonal to no edge of its linear regions, which is almost every direction; C(û) is not uniform and grows without bound near the exceptional directions; for a ReLU network without compact support the statement is not established

Graph View

Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F. A., Bengio, Y. & Courville, A. (2019). On the Spectral Bias of Neural Networks. Proceedings of the 36th International Conference on Machine Learning, PMLR 97: 5301–5310. Preprint: arXiv:1806.08734 (22 June 2018).

Read in full on 2026-10-20 in NOTE-983. The proved Fourier bound is THEORY-761, and the experimental order of fitting is THEORY-759.

Why this is here

A2 listed it as a contrastive reading: an order of acquisition by frequency, which need not be an order by semantic generality. Bears on QUESTION-217, beside THEORY-086. NOTE-483 and NOTE-777 cite it without a LIT. The anthology does not hold it (only its ANTH-NOTE-296 mentions it), and it is held here for this record’s question about orders of acquisition (ADR-036), flagged anthology-candidate.

Backlinks