Source
Teoh (2026), LIT-1896, §6.1–6.3, Figs. 3–6, Tables 2–6 and Appendix E, pp. 7–12 and 21–23 of arXiv 2610.38331v1; read in NOTE-1470.
What was actually shown
On frozen checkpoints of LLaMA-style models (167M, 332M, 689M, 1.2B; K = 3 DeepSeek-style heads), the residual component had about 4× lower gradient SNR than the KL component, the ratio widening from 4.1× to 4.6× with scale (Fig. 3). Its share of the MTP gradient reaching the trunk, measured as the projection ⟨g_hard, g_res⟩ / ‖g_hard‖², was about 80% at every scale. A unit hidden-state step along it raised NTP loss; one along the KL component lowered it (Fig. 4a). Trained with the residual removed, models reached lower NTP loss at every scale and lower MTP loss from 689M (Fig. 4b–c). They had higher acceptance length in every speculative-decoding cell (Table 2) and higher average zero-shot accuracy (Table 3), and led on seven of eight fine-tuning cells at 1.2B (Table 4). At 332M the advantage held across λ_MTP ∈ {0.25, …, 1} and K ∈ {3, …, 8} (Fig. 5).
What could have come out otherwise: the residual could have carried useful signal that the trunk needed, in which case removing it would have raised NTP or MTP loss.
What this does not say
- That correction against the data is unnecessary. Soft-MTP keeps cross-entropy at the shared output head. The explicit-KL variant, which removes it there as well, had a lower-rank readout (321 against 437, Fig. 6) and was weaker on most metrics (Table 6).
- That the effect grows with scale beyond the tested range. Four points, one run each.
- That the SNR gap causes the gain. The two are reported side by side; no intervention isolates the mechanism.
- Anything about other self-distillation settings. The finding is about MTP heads with a shared head and a same-model teacher.