SOTA-420: When distilling a classifier, match the teacher's temperature-softened outputs at the same temperature and keep a down-weighted hard-label term

Source

Hinton, Vinyals and Dean (2015), LIT-680.

What to do

• Loss: T² · CE(softmax(v/T), softmax(z/T)) + λ · CE(y, softmax(z)), with teacher logits v, student logits z, and λ well below the soft term’s weight. The paper’s speech run used “a relative weight of 0.5”.
• Temperature: the same T in teacher and student during training, and T = 1 at inference. Search T. Do not import T = 20 from MNIST.
• T²: keep it. It holds the relative size of the two terms fixed as T changes, so a temperature sweep is not also a loss-weight sweep.

All three instructions come from LIT-680, which introduced distillation against a teacher’s temperature-softened outputs. It trains the student at the teacher’s T, adds the hard-label term at T = 1 because “the best results were generally obtained” with a considerably lower weight on it, and derives T² from the soft gradients’ 1/T² scaling. Its evidence is an acoustic model distilled from a 10× ensemble, reaching 60.8% test frame accuracy against the ensemble’s 61.1% and the baseline’s 58.9%, and an MNIST student that falls from 146 errors to 74 with soft targets at T = 20.

Why

The soft term’s gradients scale as 1/T². Without the factor, raising T silently shrinks the soft term relative to the hard one. That is the whole argument for T². It is a derivation about keeping hyperparameter searches comparable, not a measured gain in accuracy.

Low T ignores the teacher’s very negative logits, and high T approaches logit regression. Which is better depends on the student. A student with capacity to spare (300 or more units on MNIST) was indifferent to anything above 8. A starved one (30 units) needed 2.5–4.

Conditions

• Classification-style outputs, from 2015. The evidence is a 2-layer MNIST net and one production acoustic model, one run each. Nothing here covers sequence-level generation. The record’s language-model distillation evidence (LIT-441) uses sequence-level distillation, a different method.
• None of the three instructions is ablated in the source. T² is derived, the down-weighting is “we found”, and the temperature evidence is one sweep on speech plus a sentence about MNIST. That is why this is Proposed despite its age.
• Compression size. Big-to-small was shown only on MNIST. The speech result compresses an ensemble into one member-sized model.

Known implementations

• None recorded. The recipe is a standard component of many libraries, which is adoption and is not listed as evidence.