Source
Saharia et al. (2022), LIT-073 — Imagen, where high classifier-free guidance weights were destroying images and the cause turned out to be arithmetic rather than aesthetic.
The mechanism
Training data lies in [−1, 1], so the model’s x̂₀ prediction should too.
High guidance weights push it outside. On its own that would be a bounded
error — but the model is applied iteratively to its own output, so the
out-of-range prediction is fed back in as input, and the excursion compounds
step over step. LIT-073’s description: the sampling process “produces unnatural
images and sometimes even diverges”.
A small out-of-distribution excursion is not small when it is the next step’s input.
The fix
Dynamic thresholding. At each sampling step, set s to a percentile of
|x̂₀|; if s > 1, clip x̂₀ to [−s, s] and divide by s. Saturated pixels
are pushed inward at every step rather than accumulating.
The percentile matters: static clipping to [−1, 1] also bounds the range and
flattens everything that was already near the edges. The dynamic version rescales
instead, so the distribution is compressed rather than truncated.
Reported to improve both photorealism and image-text alignment, especially at very large guidance weights — so it is not a quality-for-safety trade.
The precondition: the sampler has to be carrying x̂₀
This practice says to clamp the x̂₀ prediction at every step, which assumes
there is an x̂₀ to clamp. A diffusion ODE solver can be written on the noise
prediction ε̂ instead, and a high-order one then has intermediate stages where
no data prediction exists — so the clamp has nowhere to attach.
LIT-676 reaches this mechanism independently, from the solver side rather than from Imagen’s, and draws the design consequence: it solves the ODE for the data prediction model specifically because “thresholding methods are further available to keep the samples bounded”. SOTA-410 is that recommendation.
The practical form: this practice and a noise-prediction high-order sampler are not compatible, and the incompatibility bites hardest at the large guidance weights where this practice was needed in the first place. Two documents in this record found the same mechanism from opposite directions, which is the strongest support either of them has.
The general form
The record already holds this instrument twice, in two other places:
- SOTA-158 bounds the activation range in low precision, because error accumulates through a network’s depth.
This is the third: bound the output, because a sampler consumes its own predictions. The common structure is a quantity that is fed back into the process that produced it, where an excursion is not corrected but compounded.
Where else this structure appears and the record says nothing: autoregressive decoding, and agent loops that consume their own outputs.
Conditions
Diffusion sampling with classifier-free guidance, images, 2022. The percentile is a hyperparameter with no principle behind it.
Requires a known training range — [−1, 1] here — so “the training range” has
to be a thing you can name. That is easy for pixels and not obvious for a latent.
The generalisation to other iterative self-application is the shape of the
argument, not something LIT-073 establishes.
Known implementations
- Imagen; standard in guided diffusion samplers since