anthology-of-the-sota

LIT-551: Implicit Neural Representations with Periodic Activation Functions

3 min read

Properties6
StatusActive
Tagsmodel-architecture, model-stability, vision-and-graphics
Compared againstLIT-435 — NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, LIT-511 — Image-GS: Content-Adaptive Image Representation via 2D Gaussians
Cited as “introduced by” bySOTA-332 — When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network
Cited as “paper” byNOTE-295 — SIREN
Cited as “source” bySOTA-332 — When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network, THEORY-078 — Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked

Graph View

Sitzmann, Martel, Bergman, Lindell, Wetzstein, Stanford (2020) — ARXIV-2006.09661

Key takeaways

  • Sine activations for coordinate networks. φ(x) = sin(Wx + b) at every layer. The derivative of a SIREN is a SIREN, so gradients and Laplacians of the fit are as expressive as the fit. ReLU networks are piecewise linear and have a zero second derivative everywhere
  • The initialization is the method. With inputs arcsine-distributed and weights U(−√(6/n), √(6/n)), each pre-activation is approximately N(0, 1) and each sine output is again arcsine, at every depth. Most pre-activations stay below π, so frequency content grows only slowly with depth. The first layer uses sin(ω₀ · Wx + b) with ω₀ = 30 so it spans several periods over [−1, 1]. The authors report that weights “not carefully chosen” gave poor accuracy and slow convergence
  • Checked at initialization: in a 6-layer, 2048-wide SIREN, activation histograms match the predicted normal and arcsine distributions layer by layer, the maximum frequency grows slowly, and gradient statistics stay constant with depth. The same holds at 50 layers (supplement, §1.4)
  • Results: it fits an image and its gradient and Laplacian where ReLU, tanh, ReLU with positional encoding and RBF-ReLU do not get the derivatives right (Figure 1). On a 300-frame 512×512 video it reaches 29.90 dB against 25.12 for ReLU. It fits a room-scale SDF with one 5-layer MLP, and solves Poisson, Helmholtz and wave problems supervised only through derivatives

Standing in the anthology

Filed from #163, whose item asks why periodic activations used to struggle. The paper’s answer is initialization, and the record holds it as THEORY-078 at Proposed. The authors verify that their initialization does what they derive. They do not train earlier periodic networks, or SIRENs with standard initializations, and show that this is where those fail. The practice it supports is SOTA-332, scoped to derivative supervision.

Measured by others at fixed size. Image-GS (LIT-511) includes SIREN and Fourier-feature networks (LIT-550) in a same-size image-compression comparison, and all three coordinate MLPs lose to the Gaussian primitives. That is about fitting pixel values, where SOTA-205’s explicit structures win. SIREN’s distinctive claim is about derivatives, which that benchmark does not test.

Compared with NeRF’s encoding. The paper’s own image-fitting baselines (Figure 1) include a ReLU MLP with the positional encoding of NeRF (LIT-435), which it cites as concurrent work. Only that baseline and SIREN reproduce the image itself, and only SIREN also reproduces its gradient and Laplacian: the two fix the same high-frequency failure, one at the input and one in the activation, and only the activation carries over to derivatives.

The RoPE question in #163 (“leads into RoPE maybe? probably not”): the paper does not discuss attention or sequence position, and the record draws no link.

Read — NOTE-295.

Backlinks