Source
He et al. (2021), LIT-601 — ARXIV-2111.06377.
The claim
A masked-prediction objective is only as good as the task it creates. If the signal is redundant enough that a hidden region can be recovered from its neighbours, the model learns interpolation and stops. The masking ratio is the knob that decides which of those two things gets learned, and its right value is a property of the signal.
BERT masks 15% of tokens. MAE masks 75% of image patches — five times as much — and the paper calls the optimum “surprisingly high”. The justification is not a sweep: images are natural signals with heavy spatial redundancy, so a high ratio “largely eliminates redundancy, thus creating a task that cannot be easily solved by extrapolation from neighboring patches”.
The recommendation starts in LIT-601. He et al. trace masked autoencoding’s slow start in vision partly to information density — language is “highly semantic and information-dense”, images have heavy spatial redundancy — and prescribe a high ratio as the remedy. Their ablation over the ratio, discussed under the measurement caveat below, puts the optimum at 75%.
The text end of that contrast is weaker than it looks, and the difference
matters to how this practice should be read. BERT’s own words are “in all of
our experiments, we mask 15% of all WordPiece tokens in each sequence at
random” — LIT-670. There is no sweep of the rate in the paper. Its one
appendix table on masking is headed “Masking Rates” and varies the 80/10/10
[MASK]/random/unchanged substitution mix at a fixed 15% selection, which is a
different knob. RoBERTa — LIT-671 — revisits BERT’s recipe in detail and
varies when the mask is drawn, never how much of the sequence it covers.
So “the ratio that worked on text” is not a result anybody reported. 15% is a choice from 2018 that nothing in this record shows to be right for text, which does not weaken the recommendation — it strengthens the half that says do not inherit the number. What it removes is the reading where 15% and 75% are two measured optima whose difference needs explaining. One of them is measured.
Swept, the text side comes out higher than 15% and depends on the model. LIT-672 ran it: on average, 40% is optimal at 354M parameters, 20% at 124M and 15% at 51M. At 354M, 40% beats 15% on seven of nine GLUE-plus-SQuAD tasks and reaches the 15% model’s QNLI and QQP scores in half the training time. So the five-fold gap this practice was explaining is nearer two-fold once both sides are measured — and the thing that moved was the number nobody had checked.
Which changes what the contrast can be used to argue. Two of the things that move the text optimum are not properties of text: model capacity, and the masking strategy (uniform masking admits a higher rate than span or PMI masking, because it is an easier task at a given rate). And at an 80% rate text validation perplexity exceeds 1,000 — nothing can be reconstructed — while 95% of fine-tuning performance and 90% of BLiMP probing accuracy survive. Redundancy governs reconstruction; reconstruction turns out not to govern representation quality.
The recommendation is unaffected and better founded: do not inherit the number is stronger advice once the inherited number is known to have been wrong in its own modality. What this document no longer claims is that the gap between 75% and 15% is fully explained by the signals. The account that tries to explain it is
THEORY-088, and it has lost a term.
How to set it for a signal nobody has done yet
The transferable procedure is the question, not the number: can a competent interpolator solve my masked task? If yes, the ratio is too low, whatever it worked out to elsewhere.
Practical proxies for redundancy — the autocorrelation of the signal, how well a cheap baseline (nearest-neighbour, linear interpolation, a small local model) reconstructs a masked region — are cheaper to measure than a full pretraining sweep and answer the same question. A modality where a trivial baseline fills the gap needs a ratio high enough to break it.
The measurement caveat that comes with it
Linear probing and fine-tuning disagree about the optimum, and the paper
shows it: linear-probe accuracy rises steadily with the masking ratio to a
sweet spot, fine-tuning is flatter. So the ratio you pick depends on the
protocol you evaluate with, and a paper reporting one has made a choice it
may not have disclosed — which is SOTA-196 exactly.
The decoder has the same property. Depth matters for linear probing; a single-block decoder already reaches 84.8% fine-tuned.
Conditions
- Two modalities, one paper. The 15%-versus-75% contrast is a comparison across two literatures, not a controlled experiment, and no third modality is tested here. THEORY-088 is where the record keeps what would settle it.
- Redundancy is not one number. Spatial redundancy in images, temporal redundancy in video and spectral redundancy in audio are different things, and a single scalar ratio may not be the right control for all of them.
- A high ratio compounds with SOTA-372 — most of that practice’s speedup is the masking ratio — so the two are usually chosen together and their benefits are not independent.
- It is about the ratio, not the pattern. MAE masks uniformly at random;
SOTA-250argues for large, semantically meaningful blocks. Those are separate choices and this document only covers the first.