Wager et al. (2013) — ARXIV-1307.1493
Key takeaways
-
The characterization is sharper than “dropout is like weight decay.” For generalized linear models the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. The penalty is therefore adaptive: it depends on the data through the curvature of the loss, and directions the model is already confident about are penalized differently from directions it is not. Plain L2 has no such dependence.
-
The scope is stated and is narrower than the title suggests. The equivalence is derived for generalized linear models and holds to first order. Dropout and other feature-noising schemes are framed as controlling overfitting by “artificially corrupting the training data”, and the result is about what that corruption amounts to in expectation for that model class.
-
There is a connection to AdaGrad: a close relative of AdaGrad turns out to operate by repeatedly solving linear dropout-regularized problems. The same inverse-curvature scaling appears in both, which is a reason to treat it as the substantive content of the account rather than an artifact of the derivation.
-
The account is put to work, which is the part that makes it more than an interpretation. Because the regularizer depends on the feature distribution and not on the labels, unlabeled data can be used to estimate a better one. Applied to document classification this consistently improves on dropout training, including on IMDB reviews.
Standing in the anthology
The derivable consequence is what earns this a place. An account of a regularizer that only reinterprets it is hard to test; this one predicts that a better estimate of the feature distribution should give a better regularizer, and the semi-supervised result is that prediction coming out the right way. That is the evidence the record leans on.
It reaches the same place as LIT-393’s §5 from a different direction and with a different model class, and the two are worth reading together: a weight-decay term scaled by input magnitude and dropout variance there, an L2 penalty under an inverse-Fisher metric here. Neither covers deep networks.