Source
Song et al. (2023), LIT-093 — Consistency Models, where the schedules
N(.) and mu(.) are reported as necessary for good performance when training
in isolation, with the bias/variance reasoning given explicitly (Fig. 3d,
Appendix C).
The claim
When a training objective approximates a continuous target by discretising it into steps, the number of steps is not a constant to be tuned once. It sets a bias/variance trade that moves over training:
- Coarse early. Few steps give a target that is biased but cheap and low-variance, which is what an untrained model can use.
- Fine late. More steps give a faithful target whose extra variance a converged model can absorb.
Fixing the count picks one point on that trade for the whole run.
Why this is Proposed
The mechanism is general — the argument names nothing specific to consistency training, and the same shape applies anywhere a loss approximates a target through a step count. But the evidence is one paper, one setting, and the schedules there are reported as necessary rather than isolated against a well-tuned fixed count.
The record’s other schedule practices are all about learning rates (SOTA-140) and batch sizes. A schedule on a discretisation is a different object, and until 2026-10 this was the only instance the record held. One instance of a general-looking mechanism is how an over-general practice gets filed, so it waits. The second instance, below, is the same method from the same author, so it still waits.
The comparison it lacked, and a counter
iCT (LIT-786) runs the comparison the source did not. In consistency training from random weights on CIFAR-10, a curriculum that doubles N from 10 to 1,280 at fixed intervals beats a constant N and four other shapes (square-root, linear, square and cosine) with the same endpoints (its Fig. 3c). FID improves as a power law in N until it saturates. It is one setting, curves, one run each, and its bias/variance reasoning is carried over from the source rather than tested. Song is first author of both.
sCM (LIT-790) contests the premise. Its continuous-time consistency model removes the discretisation rather than annealing it. Discrete-time models improve as N rises to 1,024 and then degrade from numerical precision, and the continuous-time model beats every N tried (its Fig. 5c, App. E). If the continuous target is trainable, there is no step count to schedule. Its run is a distillation started from a pretrained diffusion model, in one setting, so it does not test training from random weights, which is where iCT found the curriculum mattered. It compares against fixed N, not against a schedule.