SOTA-240: Apply dropout where the model can memorize what it is shown, and not where it cannot

Source

Srivastava et al. (2014), LIT-395 — JMLR 15:1929-1958. Introduced by Hinton et al. (2012), LIT-394 — ARXIV-1207.0580.

The condition is in the first sentence of the 2012 paper and has been part of the claim ever since: a large network trained on a small training set. Dropout is a treatment for a model with enough capacity to memorize what it is being shown. It is not a default to carry into a setting where that is not true, and it has never been presented as one.

LIT-394 is where the practice starts: Hinton et al. first stated the procedure, randomly omitting half the feature detectors on each training case, and reported that it greatly reduces overfitting in that regime. The reason they gave is co-adaptation, a detector that is only useful alongside several specific others, which dropout breaks by making that company unreliable. The dataset-size evidence for the condition came two years later, in LIT-395.

The sweet spot, and why both edges matter

LIT-395 §7.4 varies MNIST training set size — 100, 500, 1K, 5K, 10K, 50K — against one fixed architecture with p = 0.5 hidden and 0.8 input. At 100 and 500 examples dropout gives no improvement at all: the model can overfit through the noise. The gain then rises with data size “up to a point and then declines”, because past that point overfitting is no longer what limits the model. The paper’s own words for what it found: a sweet spot at “some amount of data that is large enough to not be memorized in spite of the noise but not so large that overfitting is not a problem anyways.”

Both edges are the useful part. A practice derived only from the left one would say more data, more dropout; a practice derived only from the right one would say the opposite. What the curve actually supports is a ratio claim, which is why this practice is stated as one.

Where that leaves large-scale pretraining was an inference; it now has one reported result behind it. If a corpus is large enough that a model sees most of it once and cannot memorize it, the right edge of §7.4 predicts little to gain — against a cost the same paper measures at 2-3x training time. LIT-668 reports the prediction coming true and gives the reason in the same terms: after 1M steps ALBERT-xxlarge “still do[es] not overfit to [its] training data”, so dropout was removed, and MLM accuracy rose along with every downstream task — 90.4 to 90.7 on average, with SQuAD 1.1, SQuAD 2.0, MNLI, SST-2 and RACE all improving. The authors claim priority (“to the best of our knowledge, we are the first to show that dropout can hurt performance in large Transformer-based models”) and immediately bound it, noting that ALBERT’s shared-layer structure is “a special case of the transformer”.

So the right edge holds on one language model, for the reason the 2014 curve gives. And 0.3 points of average on one configuration is a confirmation, not a large effect.

LIT-681 is the second, with an ordinary architecture. It is a BERT-base-shaped MLM trained for 24 hours on one GPU over a single epoch, where “overfitting is not possible”. Dropout is off in pretraining and back on at 0.1 for fine-tuning. The one row that turns it on in pretraining gives 80.95 MNLI-m against 81.79 (its Table 12, one pretraining run). The paper’s reason is a different one from the 2014 curve: dropout “effectively reduces the number of gradient updates seen by each parameter” at nearly the same step cost. That argument is not measured. The outcome is, and it falls where this practice predicts. What is still not surveyed is current recipes. This record has two encoders, not a statement about why the field sets dropout to zero today.

The converse case is live and recent. LIT-119 reports dropout 0.1 after the linear projections recovering HumanEval-FIM “under the heavy repetition of a small FIM corpus” — a 2026 model report reaching for a 2012 technique under the condition the 2014 paper predicts it works in. LIT-175 is the same regime seen whole: multi-epoch training on a fixed corpus overfits severely, and the objective augmentations it proposes — token-level noise among them — are dropout’s question asked again about the data rather than the units.

It is not a drop-in, and the paper says so

Appendix A of LIT-395 is a coupled recipe, and adding p = 0.5 to an otherwise unchanged configuration is not what it reports:

• At least n/p units where n was optimal without dropout. Only pn are present in expectation, so the layer has to be widened to compensate.
• 10-100x the learning rate that was optimal without it, because dropout noise makes gradients cancel. Momentum 0.95-0.99 rather than the usual 0.9.
• Max-norm, c typically 3-4 — needed because of the high learning rate and momentum above, to stop the weights growing. The paper reports dropout + max-norm as better than either alone (1.05 vs 1.35 and 1.25 on its comparison).
• p of 0.5-0.8 for hidden units, 0.8 for real-valued inputs. Smaller p needs bigger n, which slows training and can underfit; larger p may not regularize enough.

The cost is 2-3x training time for the same architecture, and the paper attributes it to the noise rather than to the masking arithmetic: each case trains a different random architecture, so the gradients are not gradients of the network that will be used.

Gaussian multiplicative noise is the cheaper variant on one axis. Multiplying activations by N(1, σ²) with σ² = (1−p)/p matches Bernoulli dropout’s first two moments, requires no test-time weight scaling at all, and is reported as as good or slightly better (MNIST 0.95 vs 1.08; CIFAR-10 12.5 vs 12.6).

Known implementations

• Falcon-H1-Tiny — dropout 0.1 after the linear projections, for a repeated FIM corpus (LIT-119).

The verdict is about the schedule, not only the regime

LIT-475 is the strongest single confirmation of the negative half above: standard dropout costs ViT-T six points of ImageNet-1K top-1 — 73.9 down to 67.9 — on a model far too small to memorize 1.2M images, which is this practice’s condition being paid in full.

The same paper shows that dropout applied only for the first stretch of training and then switched off lowers training loss on the same models. So what the condition rules out is dropout as a standing regularizer on a model that is not overfitting. It does not rule out the operator appearing briefly at the start, where it is doing something else entirely (SOTA-285, explained by THEORY-044).

The evidence for that boundary is vision only, which is why it is a pointer here rather than a change to this practice’s claim.