Nie et al. (2025) — ARXIV-2502.09992
Key takeaways
What is being challenged is an identification, not a benchmark. The capabilities everyone attributes to LLMs — in-context learning, instruction following, scalability — are routinely attributed to autoregressive modelling specifically. LLaDA holds the paradigm fixed (pretrain, then SFT) and swaps only the factorization, which is what makes the comparison an argument about that identification rather than about a model.
The mechanism. A forward process masks tokens; a reverse process, parameterized by a Transformer, predicts them. Training optimizes a lower bound on the likelihood, so it is a principled generative model rather than a denoising heuristic — the paper is explicit that this is what gives it probabilistic inference.
The reversal-curse result is the sharpest one. Autoregressive models trained on “A is B” notoriously fail at “B is A”. LLaDA surpasses GPT-4o on reversal poem completion. That is a small task, and it is the one place where the difference in factorization predicts the difference in behaviour in advance — which makes it evidence about the mechanism rather than about scale.
The comparison is against self-constructed ARM baselines, plus published LLaMA3 8B numbers. “Comparable to our self-constructed ARM baselines” is the honest form of the claim and should be read as written: this is a demonstration that the paradigm scales, not a report that it wins.
Standing in the anthology
It is the assumption under the whole registry, named. Every practice in this record — schedules, optimizers, sparsity, attention variants, context extension, the RL post-training spine — is advice about training an autoregressive transformer. None of them says so, because nothing in the record has ever needed the qualifier. LLaDA is why the qualifier now exists to be written.
Some of the registry would survive the swap untouched: AdamW and its decoupled decay, µP, warmup-stable-decay, mixture-of-experts routing and data packing are about optimization and capacity, not about the factorization. Others are load-bearing on it — fill-in-the-middle training, multi-token prediction, speculative decoding and KV-cache compression all exist because generation is left-to-right and cached. Which practices sit on which side is a real question the record has never had to answer.
The practice is SOTA-157, filed Proposed. No frontier report in
this record trains a diffusion language model, and LLaDA at 8B against its
own ARM baselines does not establish that one should — which is what
Proposed is for rather than a reason to file nothing. Its promote_when:
asks for what would settle it: a released model above 8B trained this way, or
a matched-compute comparison with both arms swept, from a second group. The
split above is recorded in that practice too, as the adoption cost nobody has
costed.