LIT-395: Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Srivastava et al. (2014) — JMLR 15:1929-1958, https://jmlr.org/papers/v15/srivastava14a.html

Key takeaways

• The framing is model combination made affordable. Dropout samples from an exponential number of thinned networks during training; at test time a single unthinned network with weights scaled by p approximates averaging their predictions. Section 7.5 checks the approximation empirically against Monte-Carlo averaging over k sampled subnetworks, and weight scaling sits where 50-plus samples land.

• There is a sweet spot in dataset size, and both edges of it are reported (§7.4, Figure 10). On MNIST at sizes 100, 500, 1K, 5K, 10K and 50K with one fixed architecture: at 100 and 500 dropout gives no improvement at all — “the model has enough parameters that it can overfit on the training data, even with all the noise coming from dropout” — the gain then rises with data size “up to a point and then declines”. The paper’s own reading: a sweet spot at “some amount of data that is large enough to not be memorized in spite of the noise but not so large that overfitting is not a problem anyways.”

• It costs 2-3x training time, and the paper says why. “A dropout network typically takes 2-3 times longer to train than a standard neural network of the same architecture”, because each case trains a different random architecture so the gradients are not gradients of the network that will be used. The paper states this as a trade-off against overfitting, not as an implementation detail.

• The hyperparameters are coupled, and that is the part citations drop (Appendix A). A dropout net needs at least n/p units where n was optimal without it; 10-100x the learning rate that was optimal without it, because dropout noise cancels gradients; momentum 0.95-0.99 rather than 0.9; and max-norm with c typically 3-4 to stop the weights that high learning rate and momentum produce from blowing up. Typical p is 0.5-0.8 for hidden units and 0.8 for real-valued inputs. Dropping p = 0.5 into an otherwise unchanged recipe is not what this paper reports.

• Gaussian multiplicative noise works as well or slightly better. Multiplying activations by N(1, σ²) with σ² = (1−p)/p matches Bernoulli dropout’s first two moments, needs no test-time weight scaling, and edges it on MNIST (0.95 vs 1.08) while tying on CIFAR-10 (12.5 vs 12.6).

• For linear regression the marginalized objective is exactly ridge with a data-dependent Γ (§9.1), scaling each weight’s cost by the standard deviation of that input dimension — and absorbing p into w makes the regularization constant (1−p)/p, so less retention means more regularization. The paper is explicit that no closed form is available for logistic regression or deep nets.

Standing in the anthology

This is the evidence document for the practice; the recommendation was first stated two years earlier in LIT-394.

What makes it worth its length here is §7.4 and Appendix A rather than the technique, which everybody already knows. The conditions are the anthology’s business: a paper that reports where its own method stops helping, in both directions, is unusual, and the sweet-spot result is what lets the practice be stated conditionally instead of as a default.

The comparison it could not run is LIT-705’s, because batch normalization came after it. That paper puts the inverted form of this dropout inside batch-normalized networks, and dropout 0.5 in every DenseNet-BC bottleneck takes CIFAR-100 accuracy from 77.42% to 68.55%, with the models then misclassifying their own training data in eval mode. Its diagnosis is a limit of this paper’s test-time rule: weight scaling preserves a unit’s mean, not its variance, and a BN layer downstream keeps the train-mode variance.