Source
Hoogeboom, Heek and Salimans (2023), LIT-660 — ARXIV-2301.11093, and Chen (2023), LIT-659 — ARXIV-2301.10972. They are concurrent, and each cites the other.
Esser et al. (2024), LIT-449 — ARXIV-2403.03206. SD3 derives the shift again for rectified flow, measures the sampling-time value by human preference, and notes that its shift is “similar to (Hoogeboom et al., 2023)”.
The correction
A noise schedule is usually treated as a property of the model: chosen once, carried across runs. But what a timestep does depends on how much signal there is to destroy, and an image with more pixels has more. Add the same noise to a 256×256 image and a 1024×1024 one and the second is less corrupted, because the redundancy across neighbouring pixels survives.
So a timestep t at one resolution corresponds to a different timestep at
another if the two are to corrupt equivalently, and the correspondence can be
written down from the pixel count. The usual pipeline — pretrain at low
resolution, finetune at high — carries the schedule across unchanged, which
means the high-resolution stage trains against a schedule that under-corrupts
throughout. The same error occurs without a second stage. The cosine
schedule was designed at 32 and 64, and using it to train at 256 from
scratch is where the error was first measured.
The direction of the error is predictable, which is what makes this a practice rather than an observation: higher resolution wants more noise at the same nominal timestep.
Why this is easy to miss
Nothing breaks. The model trains, the loss falls, and the samples are plausible — the failure is that the high-resolution stage spends its budget on a corruption range it was not supposed to be in. That is the same shape of quiet failure the record records elsewhere for over-large batches (SOTA-061) and for unadjusted preconditioning (SOTA-188): the run completes and is simply worse than it should have been, and the deficit gets attributed to data or to scale.
Relation to the schedule material the record holds
THEORY-027 says the continuous-time bound is indifferent to the
schedule’s shape given its endpoints. This practice is not in tension with
that: a resolution shift changes which corruption level a given t produces,
which is a statement about the map from time to signal-to-noise, and the
endpoints move with it. What the invariance licenses is exactly this kind of
reparameterization — if the shape were part of the model, shifting it would
be a modelling change rather than a correction.
What it is worth
Simple diffusion, Table 2. Pixel-space ImageNet, where only the reference resolution of the cosine schedule changes:
| resolution | unshifted (FID train / eval) | best shift | shifted (train / eval) |
|---|---|---|---|
| 128 | 2.96 / 3.38 | to 32 | 2.26 / 2.88 |
| 256 | 7.65 / 6.87 | to 32 | 3.76 / 3.71 |
The shift halves FID at 256, and the gain grows with resolution, which is the direction the argument predicts.
Chen, Table 4. The same shift written as input scaling, x0 → b·x0. At 256 with a 1−t schedule, b = 1 gives 7.21 and b = 0.4 gives 3.52. The best b falls from about 1.0 at 64 to 0.6 at 128 and 0.4 at 256.
SD3, Fig. 6. A human-preference sweep of the shift at sampling time, on a model trained at 1024². It shows “a strong preference for samples with shifts greater than 1.5 but less drastic differences among the higher shift values”.
The direction is measured; the magnitude is not derived
Every source derives a shift from the pixel count, and none uses the derived value:
- SD3. The derivation gives α = √(m/n), which is 4 for 256² → 1024². SD3 uses 3.0, chosen by preference, and the preference is flat above 1.5.
- Simple diffusion. The reference resolution is picked empirically. Shift-to-32 and shift-to-64 differ by 0.18 FID at 256, and the authors recommend 64 “because it performed slightly better in early iterations”.
- Chen. The optimum is found by sweeping b, and the sweep is not monotone between neighbouring values.
- FLUX.2’s latent-space sweep (LIT-572) applies the same √(m/n) argument to latent channels. SD’s best training shift is 1, and FLUX.2 prefers 4.63 against a predicted 2.82. That is SOTA-346’s territory, and it shows the same pattern.
So the practice is to shift in the predicted direction and then tune the amount. The formula is a starting point, not an answer.
Conditions
- Where it is measured. The controlled measurements are pixel-space class-conditional ImageNet, trained from scratch at each resolution. SD3 is the latent case, and only sampling-time. Nobody has reported the pretrain-at-256, finetune-at-1024 path with and without the shift, which is the situation this document was written about. The mechanism is the same, and the measurement is adjacent.
- Guidance interacts with it. Simple diffusion’s shifted schedule “can only tolerate little guidance” (Table 9). For text-to-image it interpolates between two shifts instead.
- What counts as equivalent corruption is a modelling choice. All three sources match signal destroyed per pixel, with slightly different arguments (pooling, redundancy, a constant image). A different notion of equivalence would give a different map.
- Video adds duration. Open-Sora 2.0 scales the shift with T×H×W, and LTX-Video with token count. Both extend the argument to time without testing it.
Relation to zero terminal SNR
The fix comes from Lin et al. (LIT-689), and their reason is not
resolution. It is the per-channel mean that leaks through Stable Diffusion’s
terminal SNR (√ᾱ_T = 0.068) at 512px. Emu Video (LIT-635) adds the
resolution argument, “the residual signal is higher for high resolution video
frames”, for the same fix. It rescales the
schedule so the final step is pure noise, and its controlled comparison at
512px wins 96.8% on quality against the standard schedule. That fixes the
endpoint. This practice moves the whole curve. The two are compatible, and
nobody has compared them.
Known implementations
- simple diffusion, Chen (2023) with RIN, Stable Diffusion 3
- Stable Video Diffusion, which raises P_mean from −1.2 through 0 to 1.0 as resolution rises, LTX-Video, and Open-Sora 2.0 (adoption, per DP-005)