Source
Gloeckle et al., Meta (2024), LIT-163 — ARXIV-2404.19737.
At each position, ask the model to predict the following n tokens through n independent output heads on a shared trunk. It is an auxiliary objective, not a replacement: next-token prediction stays, and the extra heads are supervision on top of it. That framing is what keeps the change cheap.
Why this is a design choice and not a trick
Two properties, and both are the kind that usually fail:
- The benefit grows with model size. Most auxiliary objectives help small models and wash out; this does the opposite.
- It survives multi-epoch training, which is where auxiliary supervision normally stops paying.
Reported at no overhead in training time, for code and natural language alike; strongest on generative benchmarks, with 13B models solving 12% more HumanEval and 17% more MBPP than comparable next-token models. Small algorithmic tasks suggest the mechanism: multi-token prediction favours induction heads.
All of this is LIT-163’s measurement, and the recommendation is its too: the paper proposes n heads on a shared trunk as an auxiliary loss for pretraining and reports the size-scaling, the multi-epoch result and the HumanEval and MBPP gains against next-token models it trained for comparison. The adopting reports ship the head without re-measuring any of it.
The double motive, which nobody has separated
It is sold as a training objective that improves the model and pays for itself, and adopted as an inference mechanism — models trained with 4-token prediction run up to 3× faster at decode, because the extra heads are a draft model for speculative decoding you did not have to train separately. That mechanism is SOTA-227, filed under #87; when this practice was written the record held no document for it, so the sentence above rested on nothing a reader could follow.
Which of the two is doing the work at frontier scale is not something any adopting report isolates. That matters for reading this practice: if you want only the training benefit, the evidence for it is this one paper; if you want the draft heads, the adoption is overwhelming and the training claim comes along for free either way.
Overwhelming for the head, that is, not for its use as a draft. Not every adopter says it drafts with the head: DeepSeek-V4’s full report (NOTE-361) keeps MTP only as an auxiliary loss, with its weight annealed from 0.3 to 0.1 at learning-rate decay, and never mentions speculative decoding. Which of the other adopters’ own texts state the inference use has not been re-checked here.
Filed late, and the reason is the same as the deduplication one
Five model reports in this record ship an MTP head and each mentions it as settled. A step everyone agrees on generates no argument, so nothing prompted a practice — the registry was missing nodes at the too-obvious end as well as the too-new end.
Known implementations
- DeepSeek-V3 and V4, Kimi K3 (fine-tuned into an EAGLE-3-style single-layer draft model), Qwen3-Next, Qwen3.8-27B.
- Against, by silence: Olmo 3, Llama 3, the Falcon line.