LIT-676: DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models

Lu, Zhou, Bao, Chen, Li and Zhu (2022) — ARXIV-2211.01095

Key takeaways

The gap it opens with. Guided sampling with a large guidance scale is how text-to-image models are actually run, and the paper’s charge is that nobody had checked the fast samplers there:

Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before. In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows large.

The table that settles it. ImageNet 256×256, discrete-time model of Dhariwal & Nichol, classifier guidance scale 8.0, no thresholding, FID (lower better):

sampler	10 NFE	15	20	25	250
DDIM — order 1	13.04	11.27	10.21	9.87	9.37
PNDM — order 2	99.80	37.59	15.50	11.54	—
DPM-Solver-2	114.62	44.05	20.33	9.84	—
DPM-Solver-3	164.74	91.59	64.11	29.40	—
DEIS-3	66.86	24.48	12.98	10.87	—
DPM-Solver++(2S)	12.20	9.85	9.19	9.32	—
DPM-Solver++(2M)	14.44	9.46	9.10	9.11	—

Where 8.0 sits for this model. Dhariwal & Nichol (LIT-699) tuned the same 256×256 model’s classifier scale for 25 DDIM steps at 2.5, giving FID 5.44, from a sweep that stopped at 3.5 (their App. I, Table 14). So 8.0 is about three times past the model’s tuned scale, deep in the region where guidance is trading FID for precision, and the DDIM row’s 9.87 at 25 steps is that trade as much as it is the solver. That is what a stress test of guided sampling should do, and it does not touch the ordering between solvers — but the FIDs are not the model’s quality, and a classifier scale of 8.0 is not the same knob as Stable Diffusion’s CFG weight of 7.5.

At 10 NFE the ordering is monotone in the wrong direction — 13.04, 114.62, 164.74 for orders 1, 2, 3. DPM-Solver-2 needs 25 evaluations to match what DDIM does in 25, and DPM-Solver-3 is still four times worse there. Meanwhile DPM-Solver++(2M) reaches 9.10 at 20 NFE, better than DDIM manages at 250.

Both baselines are rows of this table. DDIM (LIT-038) is the first-order sampler, and the comparison shows it beating every unmodified higher-order method at 10 NFE under this guidance scale, while DPM-Solver++(2S) edges it there (12.20 against 13.04) and (2M) passes it from 15 NFE on. DPM-Solver (LIT-076) is the same first author’s earlier solver, which reached 4.70 FID at 10 NFE on unguided CIFAR-10 and showed DDIM to be its first-order case; run here at guidance 8.0 its second- and third-order variants are the catastrophic rows, and DPM-Solver++ is the revision that keeps order 2 and changes the parameterization and step scheme.

Two causes, and only one of them is new to this record.

Convergence radius. A large guidance scale amplifies both the model’s output and its derivatives. High-order solvers are built out of high-order derivatives, which are “generally more sensitive to the amplifications”, so the convergence radius narrows and the solver needs a much smaller step size to converge at all. At a fixed evaluation budget the higher-order method is therefore worse than the lower-order one. Filed as THEORY-104.

Train–test mismatch. Large guidance pushes the conditional noise prediction away from the true noise, so the converged x₀ “fall[s] out of the bound” of the data range. This record already holds that mechanism, as SOTA-202, from Imagen — where high guidance weights were destroying images and the cause was arithmetic. The contribution here is what it implies for solver design, below.

The fix, in three parts.

1. Solve the ODE for the data prediction x_θ rather than the noise prediction ε_θ. The paper’s reason is that this is what makes bounding available: “thresholding methods are further available to keep the samples bounded”. You cannot clamp x̂₀ at every step if your solver’s state is not x̂₀. That is the connection SOTA-202 could not make on its own.
2. Multistep rather than singlestep, to reduce the effective step size and so work inside the narrowed convergence radius — DPM-Solver++(2M) against (2S).
3. Stop at order 2. Stated flatly: “For k ≥ 3, as discussed in Sec. 3, high-order solvers may be unsuitable for large guidance scales, thus we mainly consider k = 2 in this work”. The ladder does not continue.

Where the headline lands. 15 to 20 steps for high-quality guided samples, in pixel space and latent space. On COCO with Stable Diffusion the comparison is run at guidance 7.5, “which is the recommended setting for stable-diffusion” — so the evaluation regime is the shipped one rather than a favourable one.

Standing in the anthology

Re-measured into first place among the remaining #342 Tier B items, from last by mention count. The reason is SOTA-203, which is Active with consensus: universal, recommends sampling “with a higher-order ODE solver on the weights you already trained”, and mentions guidance nowhere — in a document whose own implementations line reads “default or selectable sampler in essentially every diffusion serving stack”, which is exactly where guidance is on by default.

The practice is not wrong; its scope was never written down. What this note supplies:

• the bound, carried onto SOTA-203, including the inversion of its own rhetorical point about DDIM being the family’s weakest member;
• SOTA-410, the recommendation for the guided regime;
• THEORY-104, the convergence-radius account;
• and the parameterization consequence for SOTA-202, which had the mechanism and not its implication for solvers.

What it does not cover. Orders above 2 under guidance are left open by the authors rather than ruled out — “leave the solvers for higher orders for future study”. Everything is image diffusion, pixel- and latent-space.

The gap this note named is closed. It was filed saying the record did not hold classifier-free guidance (2207.12598) and that this made it the most-referenced unheld paper in the cluster. It is now LIT-693, filed the same day and partly because this note said so. The antecedent on the other side — Dhariwal & Nichol, whose model this paper’s entire ImageNet table is run on — followed it as LIT-699, so both ends of the guidance lineage are now held and this note names no gap that is still open.