anthology-of-the-sota

SOTA-422: Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there

2 min read

Properties6
StatusProposed
Consensusemerging
Tagsgenerative-modeling, training-optimization
SourceLIT-689 — Common Diffusion Noise Schedules and Sample Steps are Flawed, LIT-635 — Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
Introduced byLIT-689 — Common Diffusion Noise Schedules and Sample Steps are Flawed
Compared againstSOTA-266 — Connect data and noise on a straight line, and sample the training timesteps from a logit-normal rather than uniformly

Graph View

Source

Lin, Liu, Li and Yang (2023; WACV 2024), LIT-689, with the controlled comparison in Emu Video (LIT-635).

What to do

  1. Shift and scale √ᾱ_t linearly so that √ᾱ_T = 0 and √ᾱ_1 is unchanged. For a cosine schedule, drop the 0.999 clip on β.
  2. Train with v-prediction (SOTA-195). At zero SNR, ε-prediction “becomes a trivial task”, and the v-target at t = T is the clean sample.
  3. Sample from t = T (SOTA-416), and write the sampler in the x₀ form. Converting v to ε at ᾱ_t = 0 is a division by zero.

Why

The leaked signal at t = T is the lowest-frequency part of the image, chiefly each channel’s mean. The model learns to respect it, and at inference it gets pure noise with zero mean, so outputs are pulled to medium brightness. That account is the source’s argument. No experiment isolates it, and the source never measures brightness numerically.

Evidence

Emu Video’s controlled arm at 512px video wins 96.8% on quality and 88.3% on faithfulness against “the standard noise schedule”. That arm also switches ε to v, which is why this practice pairs the two. Lin et al.’s only number, COCO FID 22.96 → 21.66, bundles this fix with trailing spacing and guidance rescale, so it is not cited for this practice alone.

Conditions

  • VP schedules only. “variance-exploding formulation cannot truly reach zero terminal SNR.”
  • Flow matching gets this for free, and beat it. Movie Gen compared flow matching against v-prediction diffusion with zero terminal SNR, everything else fixed, and flow matching won. See SOTA-266. This practice is for anyone staying on a VP diffusion model.
  • Compatible with SOTA-263’s resolution shift. That moves the whole curve, and this fixes the endpoint. Nobody has compared them.
  • Guidance may need rescaling afterwards. The source reports that classifier-free guidance “becomes very sensitive” at zero terminal SNR. See SOTA-423.

Known implementations

  • diffusers rescale_betas_zero_snr. It is adopted by CogVideoX and by Marigold’s v1-1 configuration, which is adoption per DP-005.

Backlinks