anthology-of-the-sota

LIT-550: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

3 min read

Properties7
StatusActive
Tagsrepresentation-and-encoding, vision-and-graphics, analysis-and-evaluation
ExtendsLIT-435 — NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Compared againstLIT-511 — Image-GS: Content-Adaptive Image Representation via 2D Gaussians
Cited as “introduced by” bySOTA-331 — Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution
Cited as “paper” byNOTE-296 — Fourier features
Cited as “source” bySOTA-331 — Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution, THEORY-077 — A coordinate MLP learns high frequencies slowly because its tangent kernel's spectrum falls off fast, and a sinusoidal input mapping works by making that kernel stationary with a bandwidth you choose

Graph View

Tancik, Srinivasan, Mildenhall, Fridovich-Keil, Raghavan, Singhal, Ramamoorthi, Barron, Ng, UC Berkeley, Google Research, UCSD (2020) — ARXIV-2006.10739

Key takeaways

  • Why NeRF needed positional encoding. In the kernel (NTK) regime, a network’s training error decays along each kernel eigenvector at a rate proportional to its eigenvalue. A coordinate MLP’s NTK has eigenvalues that fall off rapidly with frequency, so high-frequency content is learned “extremely slowly”, which is the spectral bias
  • What the mapping does: γ(v) = [a_j cos(2πb_jᵀv), a_j sin(2πb_jᵀv)] induces a kernel that depends only on v₁ − v₂, so it is stationary, and whose spectrum is set by the b_j and a_j. Too narrow underfits high frequencies. Too wide overfits and aliases
  • Only the scale matters. Random Fourier features drawn from Gaussian, uniform, log-uniform or Laplacian distributions trace the same test-error curve against the standard deviation of the frequencies (Figure 4). Sixteen sampled frequencies match a dense basis
  • Table 1 (PSNR, or IoU for 3D shape): no mapping, basic, positional encoding (NeRF-style, log-spaced on-axis) and Gaussian on image, 3D shape, CT, MRI and simplified NeRF tasks. Gaussian is best on all of them. For example: natural images 19.32 / 21.71 / 24.95 / 25.57, NeRF 22.41 / 23.16 / 25.28 / 25.48

Standing in the anthology

The explanation for the step NeRF (LIT-435) called necessary. Mildenhall et al. reported that the network without positional encoding oversmoothed. This paper, from an overlapping group, says why and generalizes the encoding. Filed from #163 as “[theory] fourier features”. The account is THEORY-077 and the recommendation is SOTA-331.

The theory is a kernel-regime argument, and THEORY-076 says the kernel regime is exactly where networks do not learn features. The two are consistent. Fourier features act on the input, before any feature learning, and the paper checks the kernel prediction against trained 4-layer, 1024-wide networks (Figure 3), where it holds. But it is not a claim about feature-learning dynamics.

The language-model echo is analogy. YaRN (LIT-193) motivates its NTK-aware RoPE interpolation with this argument (“RoPE is close to a Fourier feature”). This paper studies MLPs on 1–3-dimensional coordinates, not attention over token positions, and the record does not treat it as evidence for SOTA-151.

Measured by others at fixed size. Image-GS (LIT-511) includes a Fourier-feature network (“FFN”) among six neural image representations, run from official code with only model size changed — 154 KB against Image-GS’s 160 KB for 2K×2K images — and its 2D Gaussians win across the evaluated bitrate range. That is a test of compact value fitting, where SOTA-205’s explicit structures win; it says nothing for or against the spectral-bias account here.

Read — NOTE-296.

Backlinks