Hinton et al. (2012) — ARXIV-1207.0580
Key takeaways
-
Randomly omitting half the feature detectors on each training case greatly reduces overfitting when a large feedforward network is trained on a small training set. That is the whole procedure, and it is stated here first.
-
The named mechanism is co-adaptation, not ensembling. The claim is that overfitting takes the specific form of “complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors”, and dropout works by destroying the context those co-adaptations depend on. Each unit is then pushed to detect something “generally helpful … given the combinatorially large variety of internal contexts in which it must operate.”
-
The regime is stated in the first sentence and is a condition, not an aside. A large feedforward neural network trained on a small training set. The paper does not claim dropout helps a network that is not overfitting, and the size relationship between model and data is part of the claim from the very first statement of it.
-
Results were reported as records on speech and object recognition benchmarks of the day.
Standing in the anthology
This is the origin document, and it is filed as one. It is not where the
evidence for the practice lives — that is LIT-395,
which has the dataset-size sweep, the hyperparameter guide and the averaging
experiments. The practice cites the two in the shape the scheme asks for —
introduced_by: here, source: there — which is the distinction
ADR-017 draws between the work that produced the
evidence and the work that first stated the recommendation.
The record went a year without it, which is DP-007 in its plainest form. Dropout is assumed by name in five practices and two notes already here — SOTA-088, SOTA-114 and SOTA-081 list it among the operations a fused kernel has to cover, SOTA-058 among the ones sequence parallelism shards, and LIT-119 reaches for it by rate — and none of them could cite it, because nobody had ever needed to argue for it.