Gong et al. (2024) — ARXIV-2410.17891
Key takeaways
- The premise is an asymmetry of supply. Diffusion language models have been studied at smaller scale than autoregressive ones and training them from scratch at scale is hard; open-weight AR models are abundant. So adapt what exists.
- Continual pretraining, under 200B tokens, converting GPT2 and LLaMA across 127M–7B into DiffuGPT and DiffuLLaMA.
- The method rests on connections between the AR and diffusion objectives, which is what makes this a conversion rather than a re-initialisation with extra steps.
- Resulting models outperform earlier diffusion language models and are competitive with their AR counterparts, on language modeling, reasoning and commonsense benchmarks.
- The capability that justifies the exercise: filling in the middle without prompt re-ordering, plus in-context learning and instruction following. Infilling is the thing an AR model structurally cannot do well, so it is the payoff for changing paradigm rather than a bonus.
Standing in the anthology
Sources SOTA-232, and pairs with something the record already holds. SOTA-157 records the other route — train as a masked diffusion model from the start, from LIT-217 (LLaDA). This is the same choice met from the other side: keep the autoregressive pretraining and convert afterwards.
That pairing is an instance of a shape this record meets repeatedly: build it that way from scratch, or adapt an artifact you already have. Quantization has it — SOTA-163 and SOTA-185 compress a finished model, and the ternary line trains in the target format instead. Adaptation has it — SOTA-123 pretrains a tiny model from scratch on the target distribution rather than fine-tuning one. The record has no document naming that shape, which is what DP-007 predicts of a pattern everybody uses and nobody introduced.