Source
Nie et al. (2025), LIT-217 — ARXIV-2502.09992.
What is being challenged is an identification rather than a benchmark. The capabilities everyone attributes to LLMs — in-context learning, instruction following, scalability — are routinely attributed to autoregressive modelling specifically. LLaDA holds the surrounding paradigm fixed and swaps only the factorization, which is what makes the result an argument about that identification.
A forward process masks tokens; a reverse process, parameterized by a Transformer, predicts them; training optimizes a lower bound on the likelihood. It is a principled generative model rather than a denoising heuristic, and that is what gives it probabilistic inference.
Conditions, and what is not established
Reported: LLaDA 8B is competitive with LLaMA3 8B on in-context learning, and after SFT shows instruction-following in multi-turn dialogue. It surpasses GPT-4o on reversal poem completion — the sharpest result here, because it is the one place where the change in factorization predicts the change in behaviour in advance rather than after the fact.
The comparison is against self-constructed ARM baselines plus published LLaMA3 numbers, and “comparable to our self-constructed ARM baselines” is the honest form of the claim. This is a demonstration that the paradigm scales, not a report that it wins, and one group has made it at one size.
What MDLM and SEDD support, and what they do not
The practice’s title makes two choices, and until 2026-09-25 the record held evidence for only one of them.
Masked, rather than another discrete diffusion. Sahoo et al., LIT-702, and Lou et al., LIT-701, are the evidence. SEDD trains absorbing-state and uniform-state models on the same architecture and recipe, and absorbing wins on every table (LM1B ≤32.79 against ≤40.25). MDLM derives the masked-diffusion objective LLaDA trains with — a schedule-weighted average of masked-LM losses — and beats SEDD at matched training on every likelihood table it reports. LIT-479’s uniform-state model narrows the gap and does not close it. That half of the recommendation is well supported, at GPT-2 scale.
Rather than autoregressively. Neither paper supports it, and one argues the other way. MDLM retrains an autoregressive baseline on its own backbone and data, which makes it the only same-architecture comparison in the record from a group other than LLaDA’s. AR wins in-domain: LM1B 22.32 against ≤27.04 at 33B tokens and 20.86 against ≤23.00 at 327B; OpenWebText 17.54 against ≤23.21. That is with the AR arm trained for half the optimizer steps, because “matched tokens” counts only the masked tokens a diffusion model is scored on. MDLM wins 3 of 7 zero-shot sets, which its authors offer as a hypothesis about robustness, and its own checklist lists underperforming autoregression as the limitation. SEDD’s “beats GPT-2” is 3 of 5 sets against the released model, trained on different data. Its retrained LM1B baseline is 31.98 exact against SEDD’s ≤32.79 bound.
None of that meets promote_when:. It is 110M parameters, the steps are not
matched, and neither arm was swept. It is recorded because it is the nearest
thing to the requested test that exists, and at small scale it favours the
baseline. The case for this practice over autoregression still rests on
LIT-217’s demonstration at 8B and on SOTA-254’s conditional, not on the two
papers that made masked diffusion work.
What adopting this would cost, which nobody has costed
Most of this registry is advice about training an autoregressive transformer, and none of it says so. Under this practice the registry splits, and the split has not been checked by anyone:
- Carries over unchanged — the optimizer and schedule material (AdamW’s decoupled decay, µP, the peak-LR power law), the capacity material (mixture-of-experts routing and granularity), data packing. These are about optimization and capacity, not about the factorization.
- Does not carry over — fill-in-the-middle training, multi-token prediction, speculative decoding, KV-cache compression, and context extension by RoPE rescaling. Each exists because generation is left-to-right and cached; under a masked-diffusion decoder some are unnecessary and some are incoherent.
That sort is one reader’s first pass and the boundary runs through the attention material, where a practice can be about the cache or about the pattern. It is recorded here because a recommendation whose adoption cost is unknown should say so rather than imply the cost is zero.
A second line of evidence, conditional where this one is not
Prabhudesai et al., LIT-442, give the first reason to prefer this objective that is not a capability comparison. Refitting LIT-166’s data-constrained scaling law with the objective swapped, the half-life of data reuse is 512.85 for masked diffusion against 31.93 for autoregressive training — so under a fixed corpus the diffusion model keeps extracting signal from repeated data roughly sixteen times longer.
It comes with a boundary, which is why it is filed as its own practice rather than folded in here. Below a computable critical compute threshold the autoregressive model is better by a wide margin — 7.07 against 10.65 in validation loss at the single-epoch compute-optimal point. SOTA-254 carries that conditional form.
Nothing in it moves this practice’s own promote_when:. The evidence is
about data efficiency at unique-token budgets below 100M, and what this one
asks for is scale.
Known implementations
- LLaDA 8B. No frontier report in this record trains this way.