Gat, Remez, Shaul, Kreuk, Chen, Synnaeve, Adi and Lipman, Meta FAIR and Weizmann Institute (2024), NeurIPS 2024 — ARXIV-2407.15595. Read at v2 (5 Nov 2024), main text and Appendices A–I; v1 is 22 Jul 2024. Table 5, the double-precision rerun, is in v2.
Key takeaways
- The flow-matching recipe on a CTMC (§2, Thms. 2–3). Each token jumps with a “probability velocity”. Conditional velocities for a convex-mixture path, conditioned on a source–target pair, marginalize to one that generates the marginal path. With the mixture path (Eq. 9) the generating velocity has exactly the continuous x-prediction or ε-prediction form (Table 1). Sources are all-mask or uniform. Couplings are independent (“U”) or a partial-mask conditional coupling (“C”).
- Corrector sampling with a guarantee (Eq. 26, Thm. 4). Mixing the forward velocity with a backward one gives a valid velocity. With αₜ − βₜ = 1 it generates the same path while re-masking some unmasked tokens, and with αₜ − βₜ = 0 it is an MCMC corrector at fixed t. It was applied only to the 1.7B models, tuned on perplexity and entropy (App. D, Fig. 7).
- The scheduler matters, and is chosen empirically (App. D, Fig. 6, Fig. 8). For text, κₜ = t² (the cubic family with a = 0, b = 2) is best on GPT-2 perplexity. On discrete CIFAR-10 the cubic scheduler is best, and FID ranges from 5.04 to 9.75 across train/eval scheduler pairs without a corrector.
- MaskGIT is this with a biased sampler (§3, Prop. 6). With an all-mask source the denoiser is time-independent. MaskGIT’s confidence-ordered unmasking wins at low NFE and loses at high NFE on discrete CIFAR-10 (Fig. 3), which reaches FID 3.63 at 1,024 NFE.
- Unconditional text at 150M (Table 2, OpenWebText, no temperature or corrector, float32). Mixture path at 1,024 NFE: Llama-2 / Llama-3 / GPT-2 generative perplexity 22.5 / 33.8 / 29.3, entropy 7.2. SEDD: 27.2 / 43.9 / 40.5, entropy 7.6. Campbell et al.: 28.7 / 46.5 / 43.0. The 1,024-step autoregressive model: 31.4 / 54.8 / 45.3. Data entropy is 7.7.
- Double precision changes the picture (App. G, Table 5, v2). Sampling with a float64 categorical sampler, per Zheng et al. (2024), the mixture path at 1,024 NFE scores 50.1 / 101.3 / 97.5 with entropy 8.0. SEDD scores 56.1 / 117.7 / 109.5 and Campbell 50.9 / 102.9 / 103.4. The autoregressive row is unchanged at 31.4 / 54.8 / 45.3. Every discrete model is now about twice the autoregressive perplexity, and the lead over Campbell is under 1 point on Llama-2.
- At 1.7B (Tables 3–4, Llama-2 and CodeLlama data mixes, with corrector and temperature). Conditional text: Llama-3 perplexity 9.7 at 512 NFE (entropy 6.7), against 22.3 for a 1.7B autoregressive model (entropy 7.2). Code: HumanEval pass@1/10 6.7/13.4 and MBPP 6.7/20.6, against 14.3/21.3 and 17.0/34.3 for the autoregressive model. Giving the model the true answer length raises HumanEval pass@1 to 11.6.
Where the hedges are
Per DP-010:
- “Improves generative perplexity compared to previous discrete diffusion and flow models” (abstract) rests on Table 2, sampled in float32. The double-precision rerun the authors added in v2 shrinks the margin over Campbell et al. to 0.8–5.9 points and puts every discrete model behind the autoregressive baseline. Lower float32 entropy (7.2 against data 7.7) is the signature of the low-precision effect Zheng et al. describe. The abstract was not updated.
- “Surpassing a 1.7B autoregressive model” (§1, Table 3) compares a sampler tuned with corrector, temperature and scheduler against an autoregressive model whose sampling settings are not given. The discrete model’s entropy is lower (6.7 against 7.2). The table is not rerun in double precision.
- “Significantly closing the gap” on code is about half the autoregressive model’s pass rates at equal size and data (Table 4). The autoregressive model’s training budget is not stated.
- Latency. At 256 NFE it is about 2.5× faster per HumanEval example than autoregressive decoding at batch 1 (19.97 against 50.94 s). At 512 NFE they are equal. The authors note autoregressive throughput is better with a KV cache (App. F).
- No few-step regime. Text uses 256–1,024 NFE for 1,024 tokens. The conclusion says discrete flow matching “still does not achieve the level of sampling efficiency achieved by its continuous counterpart”.
- One run per row, 1,000 samples per perplexity number.
Which comparisons are like for like
- Table 2 fixes 150M parameters, OpenWebText and plain sampling. The baseline numbers’ provenance is not stated. Table 5 reruns the same rows in double precision.
- Tables 3–4 compare against an autoregressive model on the same data mix and size. The training budget and sampling settings for that model are not given.
- Discrete CIFAR-10 (Fig. 3) compares three recipes the authors trained and tuned themselves (App. G.2): MaskGIT’s cosine scheduler and confidence unmasking, Campbell et al.’s linear scheduler, and their cubic scheduler with a corrector. Each got a temperature search, and the last two a corrector search.
Standing in the anthology
It extends Flow Matching (LIT-630) to discrete state spaces. The path, the marginalization trick and the regression to a generator carry over, and the generating velocity has the same denoiser form. It is the discrete half of the later guide (LIT-777).
It is compared against SEDD (LIT-701), the score-entropy discrete diffusion model. In float32 it beats it at every NFE (Table 2), and in double precision it still does, by smaller margins (97.5 against 109.5 GPT-2 perplexity at 1,024 NFE). The record’s SEDD note already flags sampling precision as unaddressed. Table 5 here measures the effect for SEDD as well: its GPT-2 perplexity goes from 40.5 to 109.5.
It is compared against MaskGIT (LIT-498), reimplemented and trained by the authors on discrete CIFAR-10. Prop. 6 shows MaskGIT is this paper’s masked case with a different sampler. Its confidence-ordered unmasking helps at low NFE and hurts at high NFE (Fig. 3).
On SOTA-289 it bears on the mechanism and on the measurement, not on the crossover. That practice says masked diffusion “cannot revise what it has already emitted”. Here the revision is available to a masked path. Thm. 4’s corrector re-masks emitted tokens and still samples the right marginals, at the price of extra steps. What cannot revise is the plain masked sampler, not the masked path. Whether a corrector-equipped masked model beats distilled uniform-state diffusion at 8–16 steps is untested, because this paper does not go below 256 NFE and does not distil. It does not meet promote_when. It independently confirms the practice’s measurement condition. Duo (LIT-479) runs in float64 and reports entropy, and this lab’s own float64 rerun raises every discrete model’s perplexity 2–3.5 times over its float32 value.
On SOTA-157 it is a data point against, not a test. At 1.7B and equal data, the masked model reaches about half the autoregressive model’s code pass rates. The autoregressive baseline’s budget and tuning are not reported, which is what that practice’s promote_when asks for.
MDLM (LIT-702) is the masked-diffusion language model the record holds. It is not among this paper’s baselines.
Filed without a NOTE: the takeaways come from one full reading of v2, main text and Appendices A–I. Figs. 3 and 5–8 are read only through values the text or grid labels give.