nucleation

LIT-1895: Self-Distillation Amplifies Regularization in Hilbert Space

1 min read

Properties5
StatusActive — read 2026-10-31 (NOTE-1478), body and appendices, with the proofs of Appendix C skimmed; worth reading as the clean case of iterated self-distillation losing everything through the regularizer alone, on the same inputs with nothing withheld (THEORY-1252).
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-1478 — Mobahi, Farajtabar & Bartlett — Self-distillation amplifies regularization
Cited as “source” byTHEORY-1252 — In ℓ2-regularized regression held to a fixed training-error tolerance, refitting to one's own predictions on the same inputs shrinks each eigencomponent of the labels by a factor that falls with its Gram eigenvalue, while the regularization weight needed to meet the tolerance rises as the labels shrink, so repeated rounds amplify regularization and reach the zero function after finitely many; the loss comes from the regularizer acting again, not from information the student lacks

Graph View

Mobahi, H., Farajtabar, M., & Bartlett, P. L. (2020). Self-distillation amplifies regularization in Hilbert space. arXiv:2002.05715.

Read in NOTE-1478.

Why this is here

A16’s example that iterated self-distillation changes what is preserved round by round: inheritance by repeated projection is not neutral copying. Bears on CLAIM-090 (serial transmission is reconstructive) and QUESTION-361. The reading sharpens one word. The round operator here is a shrinkage, not a projection, and nothing is withheld from the student: same inputs, the teacher’s exact outputs. The loss is the inheritor’s own inductive bias applied again. That is the “regularization” item on A16’s list of what makes iterated inheritance degrade, isolated from the others (THEORY-1252).

Backlinks