Source
Rahaman et al. (2019), “On the Spectral Bias of Neural Networks”, ICML, LIT-1261, §3.1 Experiment 1, Figs. 1, 2 and 10, and Appendix A.1, read in full in NOTE-983. The figures were read from renders at low resolution, so the heatmaps were read qualitatively.
What was actually shown
The target is λ(z) = Σ_i A_i sin(2πk_i z + φ_i), with k_i = 5, 10, …, 50 and φ_i drawn uniformly. It is sampled on 200 equally spaced points of [0, 1]. A 6-layer, 256-unit ReLU network is trained for 80 000 full-batch Adam steps. Every 100 steps the single-sided DFT of the network on the grid is taken at each k_i. The plot shows |f̃(k_i)|/A_i, averaged over 10 runs that differ in the phases. In setting (a), A_i = 1. In setting (b), A_i = 0.1, 0.2, …, 1, so the 50-cycle component is ten times the 5-cycle one. In both, the normalised fit fills in from low frequency to high (Fig. 1). In the paper’s words, “lower frequencies (i.e. smaller k_i’s) are regressed first, regardless of their amplitudes”. The spectral norm of every layer rises throughout training.
What could have come out otherwise: in (b), an order by strength, as in a deep linear network (THEORY-186), would fit the 50-cycle component first. It came last.
What this does not say
- Not a theorem. The paper’s account is Eq. 9: the speed at frequency k is at most O(k^(−Δ)) at each instant. Its constant grows as the weights do. It bounds high frequencies from above and gives no lower bound for low ones, so it does not imply the order. The proved part of the paper is a bound on the Fourier content of piecewise-linear functions (THEORY-761), which is not about training.
- Not for any amplitude ratio. A tenfold ratio did not reverse the order. Whether a larger one would is untested.
- Not beyond this setting. It covers one architecture, Adam rather than gradient descent, a 1-D input on a grid, and frequencies up to half the Nyquist limit. The paper’s MNIST experiments use proxy frequencies and do not vary amplitude against frequency.
- Not “one at a time”. The heatmap shows overlapping fills, not stages. Nothing here says the components are learned in separate transitions, as in THEORY-186.
- Not a refinement order. Fourier bands are not nested representations: no band is a function of another. This is an order of acquisition only.
- Not a property of the target alone. The paper’s §4 shows that the same latent signal is fitted more easily on a curve the input space crosses faster. “Frequency” here is frequency in the network’s input coordinates.
Connections
- QUESTION-217: answers in part. In A2’s terms this is an acquisition order with no structural partner. Set beside THEORY-186, it shows that what orders acquisition need not be the target’s strength. In a deep linear network with white inputs the learner has no preference, and the data’s singular values decide. Here the learner’s frequency preference overrode a tenfold difference in strength. So Conjecture A’s “effective eigenvalues” must be the learner’s and the data’s together, and the record has no formula for that product outside the kernel regime.
- THEORY-086. In the kernel regime the order is by NTK eigenvalue, independent of the target’s amplitudes. That is the same pattern, proved, in a regime this experiment is not known to be in.
- THEORY-481. In linear attention the order is by input variance, another case where it is set by the data. This entry is the contrasting case.