anthology-of-the-sota

SOTA-203: Sample a diffusion model with a higher-order ODE solver on the weights you already trained

4 min read

Properties6
StatusActive
Version3
Consensusuniversal
Tagsgenerative-modeling
SourceLIT-076 — DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, LIT-038 — Denoising Diffusion Implicit Models, LIT-676 — DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Introduced byLIT-076 — DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

Graph View

Source

Lu et al. (2022), LIT-076 — DPM-Solver, which supplies the solver; and Song et al. (2020), LIT-038 — DDIM, which supplies the permission to change the sampler at all.

Why this is available

The enabling fact is not about solvers. LIT-038 derived a family of non-Markovian forward processes that share DDPM’s training objective, so the generative process is a choice made after training rather than a consequence of it. Sampling is therefore a separate design problem, attackable on weights somebody else trained.

That separation is what the record was already relying on without holding it. SOTA-188’s source treats sampler improvements transferring to a pretrained network — ImageNet-64 from FID 2.07 to 1.55, no retraining — as its evidence for the design-space framing. That transfer is available because of LIT-038.

The claim

The diffusion ODE is semi-linear: an analytically solvable linear term plus an integral of the neural network. A general-purpose solver approximates both. Solving the linear part exactly and approximating only the neural integral is what buys the step count.

  • 10-20 NFE, training-free: 4.70 FID at 10 NFE and 2.87 at 20 on CIFAR-10.
  • 4-16x faster than the previous best training-free samplers.
  • Beats Runge-Kutta of comparable order, which is the direct test of the argument rather than a benchmark win.
  • DDIM is exactly DPM-Solver-1. The widely used sampler is the first-order case — the least accurate member of the family without guidance. This is the reason to state the practice as “higher-order” rather than as a product name, and the qualifier is load-bearing: under guidance the ordering reverses and DDIM becomes the most accurate of them at a small budget. See the condition below.

Conditions

  • Works on continuous- and discrete-time pretrained models, under linear and cosine noise schedules. Discrete-time models accepting continuous time inputs is a stated hypothesis that works empirically; the explanation offered is a guess.
  • The exact solution does not depend on the noise schedule between endpoints (Proposition 3.1). A schedule choice is choosing where the approximation error lands, not which solution you converge to — which is a sharper reading of EDM’s tuned rho than EDM offers.
  • Quality at 10 NFE is comparable, not equal, to a long run. Where fidelity matters more than latency, spend the steps.

The condition this practice was missing: guidance

Everything above is measured without guidance. Guided sampling at a large scale is how conditional models are actually run — 7.5 is the recommended setting for Stable Diffusion — and there the recommendation inverts. LIT-676, ImageNet 256×256 at classifier guidance 8.0, FID:

sampler10 NFE152025
DDIM — order 113.0411.2710.219.87
DPM-Solver-2114.6244.0520.339.84
DPM-Solver-3164.7491.5964.1129.40

What 8.0 is on this model. The table is Dhariwal & Nichol’s 256×256 classifier-guided model (LIT-699), whose authors tuned its scale for 25 DDIM steps at 2.5 (FID 5.44). 8.0 is a deliberate stress test at about three times that, and a classifier-gradient scale is not the same knob as Stable Diffusion’s CFG weight of 7.5, however close the numbers look. The shipped-regime evidence is LIT-676’s Stable Diffusion comparison; the ImageNet table is where the failure is cleanest, not where it is typical, and its FIDs are not the model’s quality.

Monotone in the wrong direction, and two other solver families fail the same way. THEORY-104 is the account: guidance amplifies the model’s derivatives, a k-th order method is built from k-th order derivatives, and the convergence radius narrows fastest for the largest k.

So this practice applies to unguided sampling, and SOTA-410 is the guided case — second order, multistep, on the data prediction. The two are one recommendation split by regime rather than rivals.

Why the gap survived here. This document’s implementations line reads “default or selectable sampler in essentially every diffusion serving stack”, which is precisely where guidance is on by default. A practice can name its deployment setting and still not have checked the setting’s own defaults.

What this does not cover

Distillation and few-step training are a different route to the same goal and are not compared here: this practice costs a solver, that one costs a run. LIT-067 and LIT-093 are the record’s holdings on the other route.

Known implementations

  • Default or selectable sampler in essentially every diffusion serving stack.

Backlinks