Source
Karras et al. (2019), LIT-560 — StyleGAN2. Read as NOTE-301.
The practice
When a per-sample style scales a convolution’s input channels
(w'_ijk = s_i · w_ijk):
- Do not normalize the activations per sample and channel. Instance normalization forces every feature map to unit statistics and throws away their relative magnitudes, and the generator learns to put a sharp spike somewhere to control its statistics
- Demodulate the weights:
w''_ijk = w'_ijk / √(Σ_{i,k} w'_ijk² + ε). Assuming unit-variance independent inputs, this gives unit-variance outputs without looking at the data, so the magnitude information survives. Implement it with grouped convolutions - Move bias and noise outside the style block, onto normalized data
What was measured
The diagnosis and the fix originate in LIT-560, StyleGAN2, which traced the droplets to instance normalization by showing that removing normalization removes them, then replaced it with demodulation to keep style mixing. Droplet artifacts disappear from images and from intermediate activations (Figure 3). FFHQ FID 4.40 → 4.39, and PPL 212 → 175, in Table 1, from one training run per configuration.
Conditions
- The statistics assumption (i.i.d. unit-variance inputs) is what makes it “expected” normalization, and it is not enforced
- Evidence of the artifact is visual. The FID does not register it, which is also the paper’s point about FID