Source
Kingma et al. (2021), LIT-446 — ARXIV-2107.00630.
The practice follows from the theorem
THEORY-027 says the continuous-time variational bound sees the noise schedule only through its endpoint signal-to-noise ratios. Everything between them is free.
A free parameter should be spent on something. The two endpoints get optimized against the bound, because that is what they affect. The shape gets optimized against the variance of the Monte Carlo estimator of the loss — a target that exists only because the bound itself is indifferent to it. Lower estimator variance means faster optimization at the same bound.
A low-discrepancy sequence for sampling the time variable, rather than i.i.d. uniform draws, cuts the variance further and costs nothing.
The derivation and the recommendation are both Kingma et al.’s, in LIT-446. They parameterize the log-SNR as a monotone network, fit its endpoints to the VLB and its shape to the estimator’s variance, and report markedly faster optimization at the same bound on CIFAR-10. The likelihoods they reach on CIFAR-10 and ImageNet 64×64 beat the autoregressive models that had led those benchmarks.
The endpoints turn out to matter a lot
The clearest evidence that this is not bookkeeping is the Fourier-feature result. Appending high-frequency channels of the input to the denoiser buys large likelihood improvements — but only with a learned schedule. Under Ho et al.’s fixed schedule the maximum log-SNR is pinned near 8 and the likelihood stalls above 4 bits per dimension; learning the endpoints takes it to about 13.3 and the fine-scale features become usable.
So a hand-designed schedule was capping what the architecture could express, in a way nobody would have found by looking at the architecture.
Why the field did not take this up
SOTA-188 is the recommendation that actually propagated: concentrate the training noise in the middle of the range, because the extremes teach little. That is also a use of the freedom this practice establishes, aimed at a different target — where perceptual quality comes from, rather than how fast the bound converges.
The two are not in conflict and nobody has run them against each other. It is an open question whether the variance-minimizing schedule and the compute-allocating one land in the same place; two different arguments both conclude “spend effort in the middle”, which is suggestive and is not a measurement.
The one practice measured directly against this one is SOTA-412. MuLAN took VDM’s learned scalar schedule as its baseline, on likelihood, and a per-dimension schedule conditioned on a learned latent improved the bound at equal steps: 2.65 → 2.60 bits/dim on CIFAR-10 and 3.72 → 3.71 on ImageNet-32. It reached VDM’s 2.65 in 2M steps rather than 10M. Its ablations put a scalar input-conditioned schedule and a time-only multivariate one level with VDM, so the scalar schedule recommended here is beaten only by adding both the latent and the per-dimension shape.
Conditions, and why this is Proposed
Continuous time and the unweighted bound. The invariance the practice rests on is a statement about a limit. Every practical recipe the record holds uses a weighted loss and finitely many steps, where the schedule’s shape does affect what is optimized.
Likelihood is the objective throughout — CIFAR-10 and ImageNet 64×64, bits per dimension. The record’s other diffusion practices are about perceptual generation, where likelihood is known to be a poor proxy, and nothing here says a variance-minimizing schedule improves samples.
One group, 2021, and no implementation in this record learns a schedule. The variance-minimizing target is also specific to this estimator; how much of the gain survives a different time-sampling strategy is unexamined, which matters because the low-discrepancy sampler is itself one such change.
Known implementations
- None in the record. Every diffusion model it holds uses a fixed schedule.