Key takeaways
- The change of variables is the whole paper. Integrate the continuous-time diffusion loss with respect to the signal-to-noise ratio rather than with respect to time. Because the SNR function is monotone and therefore invertible, this is a legitimate substitution — and after it, the schedule appears only in the integration limits.
- So the continuous-time VLB is invariant to the schedule’s shape, given its endpoint SNRs. The generative distribution is too, up to a trivial rescaling of the latents. Variance-preserving and variance-exploding processes, treated as different model classes in the literature, are equivalent in continuous time.
- The invariance is a licence, not a curiosity. Once the shape does not affect the bound, it can be spent on something else. Here it is spent on minimizing the variance of the Monte Carlo estimator of the loss, with the endpoints separately optimized against the VLB itself. Faster optimization, same bound.
- Fourier features on the colour channels, appended to the denoiser’s input, buy large likelihood improvements — and only when the SNR is learned. With Ho et al.’s fixed schedule the maximum log-SNR is pinned near 8 and the likelihood stalls above 4 bits/dim; learning the endpoints takes it to about 13.3, and the fine-scale features become usable. The same features gave no benefit in a PixelCNN++.
- State-of-the-art likelihoods on CIFAR-10 and ImageNet 64×64, beating the autoregressive models that had held those benchmarks for years. Also a bits-back compression scheme reaching near-optimal lossless rates.
- The discrete-time loss is an upper Riemann sum of the continuous-time integral, so more timesteps is always at least as good — which is a clean answer to a question the literature had been settling empirically.
Standing in the anthology
It supplies the account under a practice the record already holds. SOTA-188 says to sample training noise from a log-normal concentrated in the middle of the range, because the extremes teach little. That is a recommendation about where to spend training compute along the noise axis, and the record has carried it without an answer to the obvious objection: if you change where you sample, have you changed the model?
This says no — in continuous time, and for the unweighted bound. The schedule shape is not part of the model, so choosing it is an optimization decision and nothing else. THEORY-027 is that argument with its limits stated.
It also fills the continuous-time gap in the record’s diffusion line. LIT-036 is discrete-time DDPM, LIT-075 is EDM’s design-space study with its own continuous formulation, and nothing between them said why the two could be discussed as the same object. This is the paper that proved they are.
Two later papers measure themselves against it. Stochastic interpolants (LIT-644) list VDM in their Table 2 as a quoted row, not a retrained one. There VDM’s CIFAR-10 NLL of 2.65 bits/dim and FID of 7.41 beat InterFlow’s 2.99 and 10.27. On ImageNet-32, InterFlow’s 3.48 is below VDM’s 3.72, though that paper’s own text gives 3.45. MuLAN (LIT-677) is the controlled comparison. On this paper’s U-Net and settings, its per-pixel schedule conditioned on a learned latent moves the CIFAR-10 bound from 2.65 to 2.60 at 10M steps, and reaches 2.65 in 2M steps. Its scalar-schedule ablation and its schedule-swap test reproduce the invariance here rather than overturn it.
The bits-back result and the Fourier features are the parts the record has no home for yet. The Fourier-feature finding is the more interesting of the two, because its conditional form — useless unless the schedule is learned — is a small worked example of the invariance being load-bearing rather than decorative.