Hang et al. (2023; ICCV 2023) — ARXIV-2303.09556
Key takeaways
The weighting. Write the loss in x0-space, where DDPM’s ε-MSE carries
an implicit weight of SNR(t) = α²/σ² (App. B). Replace that weight with
min(SNR(t), γ), capping it so the low-noise steps stop dominating. On the
other targets the same weighting is min(γ/SNR, 1) for ε and
min(SNR, γ)/(SNR + 1) for v. γ = 5 is the default.
What it starts from is DDPM’s (LIT-036) training setup: one denoiser shared
across all timesteps, trained on the simplified ε-MSE loss with fixed reverse
variances (§3.1, Eq. 4). Its change is to read that loss as an x0 loss with
weight SNR(t), and so as one fixed choice of per-timestep weighting in a
multi-task problem, and to replace that weight with a capped one. On ε
prediction the plain DDPM loss is what the paper calls the SNR strategy.
The comparison it ran (ImageNet 256 latent, ViT-B, Fig. 5), against the
record’s existing advice. It covers constant, SNR, LIT-067’s truncated
max(SNR, 1) and Min-SNR-5, each under x0, ε and v. Min-SNR-5 converges
fastest under all three targets. With ε output, constant and
max(SNR, 1) weighting diverge. Both put weight 1/SNR on the ε loss,
unbounded at the noisy end. The 3.4× is time-to-FID-10 for x0 + Min-SNR-5
against x0 + constant, which is the weakest baseline in the figure.
Against the ε default, the gain is mostly early (UNet, Table 1):
| iterations | 200K | 400K | 600K | 800K | 1M |
|---|---|---|---|---|---|
| ε baseline (ε-MSE) | 8.55 | 5.43 | 4.64 | 4.35 | 4.21 |
| ε + Min-SNR-5 | 7.32 | 4.98 | 4.48 | 4.24 | 4.14 |
| x0 baseline | 25.93 | 15.41 | 11.54 | 9.52 | 8.33 |
| x0 + Min-SNR-5 | 7.99 | 5.34 | 4.69 | 4.41 | 4.28 |
Table 1 does not state the baselines’ weighting. Read as the plain loss on each target, which Fig. 5’s curves suggest, the large effect is on x0-prediction, which was badly weighted to begin with. For the ε-MSE that DDPM made standard, the end-of-training gain is 0.07 FID, single runs.
γ barely matters below 20 (Table 2). UNet-ε gives 4.30 / 4.14 / 4.14 / 4.12 at γ = 1 / 5 / 10 / 20, so its best value there is 20, not 5.
On EDM (App. D.1.1, Fig. 9). Multiplying EDM’s loss by min(SNR, 5)/SNR
converges faster over the first 200M training images. This is the paper’s
only comparison against LIT-075’s own weighting: EDM’s official ImageNet-64
code and settings, the same batch size and optimizer, sampled with EDM’s
18-step Heun sampler, with the weighting the only change. The authors stop there
“due to the limit of compute budget”, short of EDM’s full schedule.
The headline FIDs are system results, not weighting results. 2.06 on ImageNet 256 is a ViT-XL at ~7M iterations with x0 + Min-SNR-5, classifier-free guidance at 1.5 and EDM’s Heun sampler at 50 steps. It is compared with LIT-448’s published 2.27. The “3.3× faster than DiT” (2.08 at 2.1M iterations) sets their run against DiT’s reported one. It is not a re-run.
Traps
- “3.4× faster” does not mean “3.4× faster than what you are doing”. It is measured against constant-weighted x0-prediction. Against ε-prediction with plain MSE, the table above is the evidence.
- The explanation is weaker than the recipe. “Conflicting gradients across timesteps” rests on two things. Fig. 2 fine-tunes on three timestep bins and watches the loss at others move. Fig. 4 shows Min-SNR-5 scores near the per-iteration Pareto solver on a regularized min-norm objective, Eq. 11. That is a proxy the authors chose, and nothing ties it to FID independently.
- Min-SNR is not a monotone weighting. In LIT-692’s frame it is
sech(λ/2)·min(1, γe^{−λ}), so THEORY-107’s noise-augmented-likelihood reading does not apply to it. - ViT runs use AdamW β = (0.99, 0.99), from U-ViT, and a vanilla ViT with the timestep and class as tokens. Nobody should copy the absolute FIDs to a different backbone.
Standing in the anthology
Filed from #290’s catalogue triage. The record already named it twice: in
LIT-692’s weighting table and in THEORY-107’s list of non-monotone
weightings, without holding the paper. It is the one direct comparison the
record has of LIT-067’s truncated-SNR weighting against an alternative.
SOTA-195’s weighting paragraph now says so. No practice recommends
Min-SNR, and this filing does not add one. The instruction is clear, but the
evidence is one group, and for ε-prediction the gain is mostly in the first
few hundred thousand iterations.