Source
Wettig, Gao, Zhong and Chen (2022), LIT-672 — ARXIV-2202.08005.
What to do
Treat the masking rate as a hyperparameter of the model, not a constant of the objective. The measured optima, averaged over GLUE and SQuAD under an efficient pre-training recipe:
| model | parameters | rate |
|---|---|---|
| medium | 51M | 15% |
| base | 124M | 20% |
| large | 354M | 40% |
At 354M the 40% run beats 15% on seven of nine tasks — SQuAD +1.8, RTE +2.0, MRPC +1.0 — and loses on SST-2 (−0.2) and CoLA (−1.1). The efficiency number is the one to plan around: on QNLI and QQP, 40% reaches the 15% model’s score in half the training steps.
The sweep and the recommendation are both Wettig et al.’s, LIT-672: three model sizes, rates from 15% to 50%, under one efficient recipe. They start from the observation that the rate had been held at 15% “regardless of model sizes or masking strategies” without ever being swept, and read the moving optimum as capacity — a larger model can handle predicting more tokens from less context.
If you use a structured masking strategy, expect a lower optimum than these. Span masking and PMI masking make the task harder at a given rate, so uniform masking admits the higher rate; the prior comparisons between strategies were all run at a fixed 15% and their conclusions do not transfer unexamined to a higher one.
Conditions
The advantage is largest where the compute is smallest. The sweep uses an efficient recipe with relatively few steps. Under a longer schedule and under RoBERTa’s more expensive recipe, the authors report 40% “achieving similar performance to the 15% masking rate”. So this buys a faster path to a given quality more reliably than it buys a better endpoint, and a reader on a long schedule should expect the gap to close.
Two tasks prefer the old number, at the end of training: SST-2 and CoLA. The optimum is task-dependent as well as size-dependent, and a single-task target deserves its own check.
Encoder MLM only, 51M to 354M. Nothing here is measured on a decoder or on next-token prediction, and nothing here is above 354M — so “larger models want higher rates” is a trend over three points, not a law with a slope you can extrapolate.
It is not a claim about redundancy. The rate moves with capacity here while the signal is held fixed, which is what corrects THEORY-088 — and at an 80% rate, where validation perplexity exceeds 1,000 and nothing can be reconstructed, 95% of fine-tuning performance survives. Whatever sets the useful rate, it is not whether the masked content is recoverable.
Known implementations
- None known at the time of filing. 15% remains the default in the MLM
codebases this record can name, which is a fact about inertia rather than
about evidence (
DP-005).