Source
Gong et al. (2024), LIT-381 — ARXIV-2410.17891.
The claim
Diffusion language models are hard to train from scratch at scale; autoregressive checkpoints are abundant. Continual pretraining under 200B tokens, on a stated correspondence between the AR and diffusion objectives, converts one into the other — demonstrated on GPT2 and LLaMA from 127M to 7B, released as DiffuGPT and DiffuLLaMA.
The converted models beat earlier diffusion LMs and are competitive with the AR models they came from.
The recommendation to convert rather than train starts with LIT-381. Gong et al. framed it as a question of supply, adapting the abundant AR checkpoints because diffusion LMs are hard to train from scratch at scale, and reported DiffuGPT and DiffuLLaMA competitive with their AR originals on language modeling, reasoning and commonsense benchmarks while gaining infilling.
Why do it at all
Parity is not a reason. If the converted model only matched the original, converting would be a cost with no return.
The return is infilling without prompt re-ordering. An AR model can be made to fill a gap by rearranging the prompt so the suffix comes first, and that workaround is paid at every call and constrains how the context can be built. A diffusion model does not need it. In-context learning and instruction following survive the conversion, so the capability is added rather than traded for.
The other branch
SOTA-157 is the same decision from the other end — train as a masked
diffusion model from the start, from LIT-217. Both are Proposed, which
is the honest state: the record holds the choice and recommends neither side.
What separates them is when the decision is made and what it costs to change. Training as diffusion commits the whole pretraining budget in advance; converting is under 200B tokens spent against a checkpoint that already exists, which makes it the cheaper way to be wrong.
Conditions
Proposed, and the reason is supply rather than doubt about the measurement:
one group, one method, no production system in this record serving a
converted model. The released weights make it checkable, which is the main
thing raising it above a paper claim.
The 200B-token figure is a ceiling from these experiments, not a law, and the largest conversion shown is 7B. Nothing here says what conversion costs at frontier scale, and the compute argument for converting rather than training is strongest exactly where it has not been demonstrated.