nucleation

LIT-1896: Multi-Token Prediction Done Right

1 min read

Properties5
StatusActive — read in full 2026-10-31 (NOTE-1470); worth reading for its exact split of the multi-token-prediction gradient into self-distillation toward the model's own next-token head and a data-target residual (THEORY-1274), and for what the split does not carry: in the DeepSeek-style heads of its main results the student sees the same tokens as its teacher, so it is not the information projection the exchange read into it.
Version2
Tagsrepresentation-learning, learning-theory, anthology-candidate
Cited as “paper” byNOTE-1470 — Teoh — Multi-token prediction done right
Cited as “source” byTHEORY-1262 — In multi-token-prediction pre-training at 167M to 1.2B parameters, the part of the MTP gradient that pulls toward the sampled target carries about four-fifths of the MTP gradient reaching the shared trunk with roughly a quarter of the signal-to-noise ratio of the self-distillation part, and removing it from the trunk improves next-token loss, speculative decoding and zero-shot accuracy by small margins, THEORY-1274 — Under a shared linear output head, the cross-entropy gradient of a multi-token-prediction head at its hidden state is exactly the gradient of a forward KL toward the model's own position-aligned next-token prediction, held fixed, plus the next-token cross-entropy gradient at the aligned state, which alone carries the sampled target; the identity is about gradients and computations, not information, and in heads fed the intervening tokens the student and its teacher condition on the same evidence

Graph View

Teoh, J. (2026). Multi-token prediction done right. arXiv:2610.38331.

Read in full in NOTE-1470.

Why this is here

The owner linked it at U15: “maybe framing this recursive production as self-distillation could be helpful”. A15 and A16 built on its decomposition the reading of self-distillation as conditional projection: the student reproduces the teacher averaged over what the student cannot see. That is QUESTION-361’s starting point. The reading confirms the decomposition and A16’s routing, and corrects the premise. In the DeepSeek-style heads behind every main result, the MTP head is fed the intervening tokens and conditions on the same prefix as its next-token teacher. Teacher and student differ in depth of computation, not in information. The conditional-projection limit, with its irreducible conditional mutual information, belongs to the privileged-context case of Hossain et al. (LIT-1883, THEORY-1259). Here it is zero (THEORY-1274).

Backlinks