anthology-of-the-sota

SOTA-423: When classifier-free guidance over-exposes a latent diffusion model, rescale the guided prediction toward the conditional prediction's standard deviation

2 min read

Properties5
StatusProposed
Consensusunreplicated
Tagsgenerative-modeling, inference-optimization
SourceLIT-689 — Common Diffusion Noise Schedules and Sample Steps are Flawed
Introduced byLIT-689 — Common Diffusion Noise Schedules and Sample Steps are Flawed

Graph View

Source

Lin, Liu, Li and Yang (2023; WACV 2024), LIT-689, §3.4 and §5.3.

What to do

x_cfg      = x_neg + w · (x_pos − x_neg)
x_rescaled = x_cfg · std(x_pos) / std(x_cfg)
x_final    = φ · x_rescaled + (1 − φ) · x_cfg

The source uses w = 7.5, φ = 0.7. Its sweep found φ between 0.5 and 0.75 “produces the most appealing results”. At φ = 1 images are “overly plain”.

Why

A large guidance weight inflates the prediction’s scale, and the result over-exposes. SOTA-202 handles this for pixel-space models by clamping to the training range. A latent has no known range, so this rescales to the conditional prediction’s own spread instead. “Inspired by” Imagen, “applicable to both image-space and latent-space models”.

The rescale is the fourth of LIT-689’s fixes, introduced there with w = 7.5, φ = 0.7 as the setting that “works great” and supported by a sweep of five φ values on three prompts. Its one number is COCO FID 21.66 against 22.96 for the same data with no fixes (10k captions, DDIM 50 steps), and that gain bundles the rescale with zero terminal SNR, v-prediction and trailing steps.

Conditions

  • Never compared with SOTA-202. The source cites dynamic thresholding as its inspiration and does not run it, so no compared_against is declared.

  • The evidence is qualitative. Five values of φ on three prompts. Its one numeric result (COCO FID) bundles it with three other fixes.

  • It is partly a guidance-strength change. Pulling toward the conditional std lowers effective guidance, and FID usually improves as guidance drops. Nothing in the source separates the two.

  • Motivated by zero terminal SNR, not shown to need it. The source says guidance “becomes very sensitive” as terminal SNR approaches zero. It never shows over-exposure, or the fix, on a model without it (SOTA-422).

Known implementations

  • diffusers guidance_rescale.

Backlinks