Source
Chen et al. (2024), LIT-554 — Diffusion Forcing. Read as NOTE-298.
The practice
When a model generates continuous tokens one step at a time (video frames, trajectories, actions) and must run past the length it was trained on:
- Train with an independent noise level for every token in the sequence, with a causal model and an ordinary denoising loss. This covers clean-history next-token prediction and equal-noise full-sequence diffusion as special cases
- At rollout, mark the generated history as slightly noisy
(
0 < k ≪ K) and do not treat it as clean ground truth. The model has seen noisy history in training, so its own small errors do not push it out of distribution - Choose the schedule at inference. Keep the near future cleaner than the far future when planning, so uncertainty grows with the horizon, and guide over the whole horizon if there is a reward
What was measured
Both halves of the recipe originate in LIT-554, Diffusion Forcing, which framed per-token noise as a continuous mask covering teacher forcing and full-sequence diffusion as special cases, and proposed the slightly noised history for long rollouts. Its evidence, all on a small convolutional RNN:
- Minecraft and DMLab video: coherent to 1,000 frames, where teacher-forced and causal full-sequence baselines on the same RNN diverge. Figure 3 only, no metric
- D4RL Maze2D: average reward 141.7 against Diffuser’s 119.5, which needs a hand-coded controller and gets 8.7 executing its own actions
- A real robot task needing memory: 80% success. 76% with occluded observations, marked as noisy, against 48% for a next-frame baseline
What has been measured since
Self Forcing (LIT-629, Table 2) ran the first controlled comparison at transformer scale by a group other than the authors, on Wan2.1-1.3B with 5s clips scored by VBench total. The table is two matched pairs, not one setup. The many-step rows are 50×2-step models fine-tuned on 70K videos generated by Wan2.1-14B. The distilled rows are 4-step DMD students. Each comparison below is within a pair. It bears on the two halves differently.
- Training half, per-token noise against teacher forcing: split. In the many-step rows diffusion forcing loses (82.95 against 83.58 chunk-wise, 77.24 against 80.34 frame-wise). After DMD distillation it wins (82.76 against 82.32, 80.56 against 78.12). Both lose to training on the model’s own rollouts (84.31, 84.26).
- Rollout half, noised history: not tested. The paper describes the inference setting for diffusion-forced models as one “where context frames are clean and the current frame is noisy” (§1), and labels autoregressive inference “Clean KV” (Fig. 2c). The distilled diffusion-forcing row replicates CausVid, which caches clean outputs. The text doesn’t say how the many-step rows were sampled.
CausVid (LIT-631) adopts per-chunk noise for its student and its causal teacher, and also rolls out on clean context. So it adopts the training half only.
Neither result meets promote_when. It asks for quality against rollout
length with this practice’s full recipe, and Table 2 measures neither.
Conditions
- A small convolutional RNN. The authors name transformers and scale as future work
- The anti-drift claim is qualitative in the source
- Continuous tokens. This is not a recommendation for language models. The discrete-token evidence on the same mismatch is measurable but modest, and none of it shows divergence. Scheduled Sampling (LIT-649) has the largest effect, on its speech task: the teacher-forced baseline has the best next-step error (15.0) and a decoding error of 46.0, against 34.5 with scheduled sampling (its Table 3). MIXER (LIT-647) gains 1 to 3 points, but it also optimizes the test metric, so the gain is not isolated to exposure bias. Professor Forcing (LIT-642) finds no difference on word-level Penn Treebank and attributes that to dependency length; its strongest result is on continuous handwriting