Source
Sahoo, Deschenaux, Gokaslan, Wang, Chiu and Kuleshov (2025), LIT-479 — read as NOTE-228. LM1B and OpenWebText at GPT-2 scale.
The claim, and the regime it is about
The record already recommends masked diffusion for language modelling (SOTA-157, SOTA-254). This does not contest that. On likelihood, masked diffusion is ahead in the source’s own tables — MDLM wins 6 of 7 zero-shot datasets and both training corpora.
What it adds is a crossover. Below roughly 32 function evaluations, distilled uniform-state diffusion generates better text than distilled masked diffusion; above it, masked diffusion wins. So the choice depends on the step budget, and the record previously had no way to say that because it held only one of the two families.
Why the ordering flips
Masked diffusion unmasks many tokens independently and cannot revise a token once it has been emitted. At a small step count that means committing to choices made with little context and living with them, which shows up as incoherence.
Uniform-state diffusion is self-correcting — a later denoising step can overwrite an earlier one. At a large step budget that flexibility buys little and costs likelihood; at a small one it is the whole difference.
This is a structural property of the two processes, not a tuning artefact, which is the reason to expect the crossover to survive where a tuned number would not.
What cannot revise is the plain masked sampler, not the masked path. Discrete Flow Matching (LIT-792, Thm. 4) adds a corrector to a masked-path model that re-masks already-emitted tokens and still samples the right marginals, at the price of extra steps. So revision is available to masked diffusion, and the source compares against masked diffusion sampled without it. Whether a corrector-equipped masked model beats distilled uniform-state diffusion at 8–16 steps is untested: that paper samples at 256 NFE and above and does not distil.
What it takes to get there
Distillation is not optional. The 8- and 16-step results are after Discrete Consistency Distillation — five rounds, starting at a discretization step of 2 and doubling. The undistilled model needs 1024 steps.
Use the denoising model, not an EMA copy, as the distillation teacher. This goes against standard practice in consistency models and the source ablates it: the non-EMA teacher gives the better distilled model.
The Greedy-Tail sampler buys the last factor of two — 16 steps to 8 — with slightly better generative perplexity and lower entropy. Each distillation round improves both quality and diversity under Greedy-Tail, which is not true of ancestral sampling.
And train with the Gaussian-guided curriculum (THEORY-046), which is what makes the base model good enough to distil from. It roughly halves training time to a given perplexity against the previous uniform-state model.
Conditions
Measure generative perplexity in float64, and report entropy beside it. Masked diffusion models can show misleading Gen PPL with low diversity under low-precision sampling — a known artefact the source cites and guards against by running every sampling experiment in double precision. It also reports entropy: MDLM distilled with SDTT matches an autoregressive model’s Gen PPL at 5.4 entropy against 5.6, a diversity loss the perplexity number alone hides. A comparison run without both of these is not evidence about this practice. A second lab measures the same effect. Discrete Flow Matching’s float64 rerun (LIT-792, Table 5) raises every discrete model’s generative perplexity 2–3.5 times over its float32 value, puts all of them at about twice the autoregressive baseline, and shows float32 entropy below the data’s (7.2 against 7.7), the low-precision signature.
Scale. LM1B and OpenWebText, GPT-2-sized models. A latency budget that makes few-step generation worth choosing is set at serving scale, and there is no evidence there.
One lab, which also wrote the baseline. MDLM is from the same group. That makes the comparison likely to be competently run and not independent.
The crossover point is one number from one setup. “About 32 function evaluations” is where the curves cross in this paper’s figure, against this paper’s distilled MDLM. Nothing says where it sits for a different model, a different distillation budget, or a different sampler.
Likelihood is not the thing being recommended. Duo beats an autoregressive transformer on 3 of 7 zero-shot datasets and loses the other four by wide margins — 82.05 against 89.35 on PTB, 25.75 against 33.57 on Wikitext. That number belongs to the paper, not to this recommendation.