LIT-393: Understanding Dropout

Baldi and Sadowski (2013) — NIPS 26:2814-2822, https://papers.nips.cc/paper_files/paper/2013/hash/71f6278d140af599e06ad9bf1ba03cb0-Abstract.html

Key takeaways

• The averaging theorem is exact for a single logistic unit, and the logistic function is why. Writing G for the weighted geometric mean of the unit’s outputs over dropout configurations and G' for that of the complements, the normalized weighted geometric mean NWGM = G/(G+G') satisfies NWGM = σ(E[S]) exactly — so it is computed by one forward pass with weights scaled by their retention probabilities, σ(Σ wⱼpⱼIⱼ). The proof turns on the identity [1−σ(x)]/σ(x) = ce^{−λx}, and the paper notes that the only functions satisfying NWGM(f) = f(E) are the constants and the logistic functions. The test-time trick is a property of the nonlinearity, not a generic fact about averaging.

• In deep networks one approximation remains, and it is named. The averaging properties reduce to three recursive equations: E(Oᵢ) ≈ NWGM(Oᵢ), NWGM(Oᵢ) = σᵢ[E(Sᵢ)], and E(Sᵢ) = Σ wᵢⱼpⱼE(Oⱼ). The first is the only approximation. It is exact iff the unit’s activity is identical across all subnetworks — the paper’s notion of a consistent neuron — and degrades with that activity’s variance. Linear units need it at all, so regression with linear output layers “shaves off one layer of approximations.”

• The approximation is good to second order, and better than the plain geometric mean. The paper bounds E − NWGM and shows NWGM beats G, with exactness when activities are consistently low (0 < Oᵢ ≤ 0.5) or whenever NWGM ≤ E. The bounds are corroborated by simulation.

• The same formalism yields the regularization account. For a linear unit under squared error, the expected dropout gradient is exactly the gradient of E_ENS + ½ Σ wᵢ² Iᵢ² Var(δᵢ) — ensemble error plus a weight-decay term adaptively scaled by the input magnitudes and by the dropout variance, maximized at p = 0.5. For a sigmoidal unit under relative entropy the same holds approximately, via a Taylor expansion using the NWGM result.

Standing in the anthology

Two accounts of dropout come out of this one paper, and the record files them separately because they answer different questions: the averaging result says why one cheap forward pass is a legitimate stand-in for an ensemble, and the regularization result says what dropout does to the objective. That the second is derived here using the first is the connection worth keeping — they are not independent stories, and the theory documents say so.

The regularization half is the weaker of the two: it is derived for a single linear unit and extended to a sigmoidal one by approximation, and LIT-396 covers that ground with a sharper characterization.