Chen, Goodfellow and Shlens (2015), Google — ARXIV-1511.05641 (ICLR 2016)
Key takeaways
- Function-preserving initialization is the idea. The student is given parameters such that it computes exactly the teacher’s function, then is trained by ordinary means. The paper’s argument for this over earlier growth methods is that the larger network starts no worse than the smaller, so it never passes through a period of low performance, and no layer has to be frozen.
- Net2WiderNet. Widening layer
ifromntoqunits uses a random mapping that keeps the firstnunits and fills the rest by sampling existing units with replacement. Incoming weights are copied. Outgoing weights are divided by the number of copies of their unit. The mappings of consecutive layers must be kept consistent through batch norm and concatenation. The authors add that, unless training uses something like dropout to break symmetry, a little noise should be added to all copies but the first. That makes the student only approximately equal to the teacher. - Net2DeeperNet. A layer is followed by a new layer initialized to the identity. This works for ReLU, and not for activations such as the sigmoid, where no identity layer of the same type exists. With batch norm, the new layer’s scale and bias must be set from activation statistics measured on training data.
- Results, all on ImageNet Inception-BN, all read off curves. A teacher with channels cut by √0.3 (about 30% of the parameters in most layers) is widened to the standard Inception. Net2WiderNet converges to about its final validation accuracy after roughly 3×10⁶ minibatches, and random initialization reaches the same accuracy roughly 2×10⁶ later. A “random pad” baseline, which widens with random new units, is also slower. Deepening each Inception module with two more identity-initialized convolution pairs trains faster than the same deep network from random initialization. A deeper student was not trained from scratch, for resource reasons, but it reached 78.5% validation accuracy, reported as a new state of the art.
- What is not shown. There are no seeds, and no comparison at matched total compute: the teacher’s training is not charged. All experiments use one architecture family on one dataset. A FitNets-style alternative, which trains a randomly initialized student to predict the teacher’s hidden layers, “did not … offer any compelling speedup” and is reported only in prose. The text gives the width factor of the design-space experiment as √2, while Figure 6’s legend says 2.
Standing in the anthology
This is the origin of function-preserving width growth. Two papers in the
record carry it forward to language models. bert2BERT (LIT-749)
carries Net2WiderNet’s mapping-and-rescale construction to BERT and GPT,
and calls itself an extension of Net2Net. HyperCloning (LIT-765)
replaces Net2Net’s random replication with a symmetric tiling, so every
unit is copied exactly n times. Both papers state that they build on this
one.
It bears on SOTA-438 as the general idea that practice applies. It is not that practice’s origin. Net2Net recommends growing a network from a trained smaller one to speed up experimentation on image classifiers. The practice recommends a particular symmetric construction for decoder language models, which this paper does not contain. Net2Net shares one limit with that practice’s source and with LIT-227’s sparse upcycling behind SOTA-209: the smaller model’s training is treated as free, or as a sunk cost, in every comparison.
Unread — no NOTE.