LIT-790: Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models

Lu and Song, OpenAI (2024), ICLR 2025 — ARXIV-2410.11081. Known as “sCM”. Read at v2 (1 Mar 2025), main text and Appendices A–G; v1 is 14 Oct 2024.

Key takeaways

• TrigFlow (§3, Eqs. 3–5). x_t = cos(t)·x₀ + sin(t)·z on t ∈ [0, π/2], with the consistency model f = cos(t)·x_t − sin(t)·σ_d·F_θ, a single Euler step of the probability-flow ODE. It keeps EDM’s unit-variance coefficients, and is a special case of flow matching and of v-prediction.
• Where the instability came from (§4.1, Eq. 7, Fig. 4). The training signal is the tangent df/dt along the ODE. The unstable part is the time derivative ∂_t F, through the time transform and the embedding. EDM’s log-tan time transform blows up near t = π/2, and Fourier embeddings at scale 16 oscillate. The fixes are c_noise(t) = t, positional embeddings, and “adaptive double normalization” in place of AdaGN.
• Controlling the gradient (§4.2, Eq. 8, Fig. 5a–b). The tangent is normalized by ‖df/dt‖ + 0.1 (clipping works too). A learned per-time weight, as in EDM2, replaces hand-set weighting. In ImageNet-512 distillation curves, normalization or clipping improves one- and two-step FID well clear of raw tangents (“substantial improvements”; Fig. 5a). The learned weight’s gain shows at two steps, and its one-step curves nearly coincide with the unweighted ones (Fig. 5b). The tangent’s unstable term is also warmed up over 10K iterations, an “optional technique” with no curve. The sentence reporting its effect appears to name the wrong technique; see the hedges.
• Continuous beats discrete (Fig. 5c, App. E). Discrete-time models improve as the step count N rises to 1,024 and degrade beyond it, from numerical precision. The continuous-time model beats every N.
• Results (Tables 1–2). CIFAR-10, one / two steps: sCT 2.85 / 2.06, sCD 3.66 / 2.52. ImageNet-64: sCT 2.04 / 1.48, sCD 2.44 / 1.66. ImageNet-512, 1.5B parameters: sCD 2.28 / 1.88, sCT 4.29 / 3.76. The teacher is 1.73 at 63 × 2 evaluations.
• Distillation scales with the teacher (Fig. 6). sCD’s FID ratio to its teacher is roughly constant across five model sizes, and smaller at two steps than one. sCT is more compute-efficient at 64 px and less so at 512 px. sCD reaches the teacher’s quality in under 20% of the teacher’s training compute; 20K iterations already give good samples (§5.2).
• Against score distillation on diversity (Fig. 7, EDM2-M on ImageNet-512). One-step VSD, the DMD objective, raises precision and lowers recall as guidance increases, ending in “severe mode collapse”. VSD is the objective DMD descends. Two-step sCD’s precision and recall stay close to the teacher’s across guidance scales. Read from the plot (no values are tabled): one-step sCD is plotted too, and at guidance 1.0 its recall is about 0.70 against about 0.65 for one- step VSD, so the gap is there at matched step count and before guidance is raised. The arm that sums the two losses at equal weight tracks VSD’s recall, not sCD’s.
• Engineering for scale (§5.1, App. F). The JVP is rearranged to avoid FP16 overflow near t = 0 and π/2. A Flash-Attention-style kernel computes attention and its JVP in one pass.

Where the hedges are

Per DP-010:

• “Consistency training (CT), by contrast, trains CMs from scratch” (§2.2). In this paper it does not. “We always initialize the CM from the EMA parameters of the teacher diffusion model”, for sCT as well as sCD (App. G). On ImageNet-512 that diffusion model trained 376K–1,048K iterations at batch 2,048 before sCT’s 100K (Table 5). sCT here is fine-tuning a diffusion model without using its ODE. Later tables that file sCT under “from scratch” (IMM, MeanFlow) misread it.
• “Narrowing the gap … to within 10%” (abstract). On CIFAR-10 it is 2% (2.06 against 2.01) and on ImageNet-512 9% (1.88 against 1.73). On ImageNet-64 it is 11% (1.48 against 1.33).
• The ImageNet-512 teachers are re-trained here, with TrigFlow and a corrected latent normalization (Table 2, App. G). At XL and XXL they beat published EDM2 (1.80 against 1.85, 1.73 against 1.81). sCD students are conditioned on the guidance scale and scored “under optimal guidance scales”, chosen per step count (Table 6).
• “The tangent normalization does not affect sample quality but may reduce some gradient spikes” (§4.2). The sentence closes the paragraph on tangent warmup, and as written it contradicts the same section’s Fig. 5a, where normalization or clipping gives “substantial improvements” in FID over raw tangents. It most likely means the warmup, an optional addition that has no curve of its own. That reading is the record’s, not the paper’s: the warmup’s effect on sample quality rests on that one sentence, and normalization’s on Fig. 5a.
• Most ablations are curves on one setting (Fig. 5, ImageNet-512 distillation, 50K iterations). The positional-embedding and normalization fixes are shown through gradient norms on CIFAR-10 (Fig. 4), not FID.
• No seeds or intervals. The diversity comparison is the only precision and recall reported, as curves.

Which comparisons are like for like

• sCD against its own teacher is matched: same architecture, same batch, initialized from it.
• The VSD comparison (Fig. 7) is run here at EDM2-M size with tuned weighting and proposal distributions (§5.2). It is the paper’s one controlled comparison against another distillation method. The text contrasts one-step VSD with two-step sCD, but the figure also plots one-step sCD, which is the like-for-like arm.
• Tables 1 and 2 are mostly copied rows, including DMD2 at 1.28 and the CD, PD and iCT rows.

Standing in the anthology

Its Fig. 7, where distribution matching alone loses recall, is why SOTA-447 keeps the consistency term primary.

It extends Consistency Models (LIT-093), whose Remark 10 derived the continuous-time gradient this paper finally makes trainable (§2.2, Eq. 2). LIT-093 and iCT had both found that limit unstable. The iCT paper (LIT-786) is its main baseline. sCT beats iCT-deep at one step on ImageNet-64, 2.04 against 3.25, and keeps iCT’s dropout placement. Unlike iCT, which trains from random weights, it starts from a diffusion model.

Its teachers are EDM (LIT-075) on CIFAR-10, the 2.01 its “within 10%” compares against, and EDM2 (LIT-714) on ImageNet, re-trained in TrigFlow form. Its ImageNet-64 table carries progressive distillation (LIT-067) at 10.70 and 4.70, as re-implemented by Heek et al., and DMD2 (LIT-646) at 1.28 in one step, ahead of sCD’s 2.44. The VSD arm of Fig. 7 is DMD’s (LIT-643) distribution-matching gradient without the regression loss. The recall drop it shows is the record’s first measurement of that objective’s diversity cost against a forward-direction distiller on one backbone.

The score-regularized consistency paper (LIT-783) scales this method to 14B-parameter video models. It finds pure sCM fails on fine detail there, and adds a DMD term.

Two from-scratch models later set their CIFAR-10 numbers against sCT’s. Inductive Moment Matching (LIT-775) reports two-step 1.98 against sCT’s 2.06, and MeanFlow (LIT-784) one-step 2.92 against 2.97. Both file sCT as trained from scratch, which by App. G here it is not: it starts from the pretrained diffusion model.

On SOTA-206 it is support. sCD and sCT sample at one or two steps from one model, and two steps close most of the gap to the teacher at every size.

On SOTA-204 it is the strongest evidence against. Fig. 5c removes the discretisation rather than annealing it, and the continuous-time model beats every fixed N tried. It is one setting, a distillation run started from a pretrained diffusion model, so it does not test training from random weights, which is where iCT found the curriculum mattered.

Filed without a NOTE: the takeaways come from one full reading of v2’s main text and appendices. Figs. 5–7 have no tabled values and are described, not quantified.