Why it is a question
Teoh (LIT-1896) shows that the gradient of the multi-token-prediction loss at a prediction head’s hidden state decomposes exactly into a forward-KL term toward the model’s own next-token prediction, held fixed, plus a residual (THEORY-1274). A model’s own outputs specify a target that constrains how another part of it is trained. That is the shape of recursive production: a realization generating a specification that constrains its successor (CLAIM-477).
The teacher there knows no more than the student. In Teoh’s main heads the student is fed the intervening tokens and conditions on the same prefix as its teacher. They differ in depth of computation, not in information (NOTE-1470). The exchange’s A15 and A16 read it as a less-informed student, and that premise was wrong. The routing A16 described holds, but the KL gradient also reaches the shared trunk that computes the teacher, so the student does pull on the teacher there.
What distillation can carry, when the teacher knows more. The limit A15 §3 derived is published as Hossain et al.’s Proposition 1 (LIT-1883, THEORY-1259). If the teacher sees information B and the student only A ⊆ B, the student’s loss E KL(q_B ‖ p_A) splits into E KL(q_B ‖ E[q_B | A]), which no student can remove, and E KL(E[q_B | A] ‖ p_A). The best student reproduces the teacher averaged over what the student cannot see. For a calibrated teacher the irreducible part is I(Y; B | A), and the student is on average at least as uncertain as its teacher. Iterated through shrinking information sets, inheritance keeps exactly what survives every restriction (A16). Agreement with the teacher is coherence, not fidelity to the world (TERM-097): “perfect agreement with an increasingly degraded teacher” is possible.
Iterated self-distillation also changes what is kept round by round, by two distinct mechanisms. In Mobahi et al. nothing is withheld from the student, and the regularizer acting again shrinks the solution to zero (THEORY-1252). In Wu et al. each round projects onto a fresh random subspace, so noise and then signal are forgotten (THEORY-1269); repeating one projection loses nothing further.
Three things keep it open.
- Endogeneity. “The fact that teacher and student belong to one model does not make that model an autonomous self-maintaining entity” (A15). In Teoh’s setting the optimizer that runs the distillation is outside the model. In Hossain et al.’s the teacher is the frozen initial model, so it is inheritance from a fixed ancestor, not a model teaching itself. What would make the distillation the organization’s own is the question CLAIM-414 and CLAIM-522 make hard: matched trajectories, even under state intervention, do not show where the producer sits.
- Teaching as well as learning. “A successor can be a perfect student and a poor teacher” (A16 §8). Among successors equally faithful on a declared probe family, some keep the capacity to reconstruct after damage to themselves and some do not. CASE-161 shows the separation for governing rules. Whether it is visible in the geometry of the set of equally faithful realizations, or only in their maintenance dynamics, is open.
- Continuity. A faithful successor is not thereby the predecessor continuing. That is QUESTION-118’s question, and fidelity is not lineage (CLAIM-329).
What would count as an answer
A system in which a realization produces the specification its successor is built to, with the producing mechanism inside a declared boundary; a test, by intervention on that mechanism, of whether the distillation is the system’s own; and a comparison of successors matched on fidelity that differ in what they can pass on after a second injury. The constructions of the exchange (CASE-190) go part of the way: a copying catalyst the organization keeps regenerated, with a student whose memory is erased and rebuilt. Its student never becomes a teacher, so the succession of realizations is not yet there.