anthology-of-the-sota

SOTA-332: When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network

2 min read

Properties6
StatusProposed
Consensusunreplicated
Tagsmodel-architecture, model-stability, vision-and-graphics
SourceLIT-551 — Implicit Neural Representations with Periodic Activation Functions
Introduced byLIT-551 — Implicit Neural Representations with Periodic Activation Functions
Explained byTHEORY-078 — Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked

Graph View

Source

Sitzmann et al. (2020), LIT-551 — SIREN. Read as NOTE-295. The initialization account is THEORY-078.

The practice

When the loss constrains the network’s gradients or higher derivatives, as in a signed distance function fit with an Eikonal term, a Poisson or Helmholtz solve, or gradient-domain image editing:

  • Use sin as every hidden activation. Any derivative of a SIREN is a SIREN, so the derivatives are as expressive as the fit. A ReLU network’s second derivative is zero everywhere, and tanh’s derivatives were “often not well behaved” in the paper’s comparison
  • Initialize hidden weights U(−√(6/n), √(6/n)). This holds pre-activations near N(0, 1) at every depth
  • Scale the first layer: sin(ω₀ · W₀x + b₀), with ω₀ = 30 in every experiment in the paper. It sets the frequency range, so it belongs to the signal, and 30 is a starting point
  • Keep inputs in [−1, 1]

What was measured

The architecture, the initialization and the recommendation to use them for derivative supervision all originate in LIT-551, SIREN. Besides the results below, it checked that its initialization does what it derives: in 6- and 50-layer SIRENs at initialization, pre-activations match N(0, 1) and sine outputs match the arcsine distribution layer by layer, with gradient statistics constant across depth.

  • Image fitting (Figure 1): ReLU with positional encoding matches SIREN on the image and not on its gradient or Laplacian. ReLU, tanh and RBF-ReLU miss the image too
  • Video, 300 frames at 512²: 29.90 dB against 25.12 dB for ReLU
  • SDF from oriented points: much finer detail than a ReLU SDF, with a room fit by one 5-layer network
  • Poisson reconstruction from gradients or Laplacians alone, and Helmholtz and wave solutions

Conditions

  • The claim is about derivatives. For fitting values alone, the record prefers explicit structures (SOTA-205), and an outside same-size benchmark (LIT-511) has SIREN losing to them
  • No comparison against Fourier features, the concurrent input-side fix (LIT-550, SOTA-331). Which is better for derivative supervision is open in the record
  • The initialization is not optional, according to the authors, but its necessity is asserted, not ablated (THEORY-078)

Known implementations

  • SIREN reference implementation (Stanford)

Backlinks