Source
Wettig, Gao, Zhong and Chen (2022), LIT-672 — ARXIV-2202.08005, measuring against the rule LIT-670 introduced.
What to do
When a position is selected for prediction, replace it with [MASK]. Do not
keep 10% of them unchanged and do not substitute a random token for another 10%.
Why the rule existed, and why it does not hold
BERT’s reasoning was a distribution mismatch: [MASK] never appears during
fine-tuning, so a model trained only on [MASK] inputs is being asked to
generalise to sentences it has never seen. The 80-10-10 mix was meant to keep
some uncorrupted and some differently-corrupted positions in the training
distribution. Almost every MLM since copied it.
Two things undercut it.
The rule’s own evidence was thin from the start. The rule is BERT’s, LIT-670, which
adopted it to reduce the pretrain/fine-tune mismatch and fixed the selection
rate at 15% without sweeping it. BERT’s only ablation of it
is Appendix C.2, whose table is headed “Masking Rates” and holds the selection
rate at 15% while varying the mix. The 80/10/10 row scores 84.2 on MNLI and
the 100/0/0 row scores 84.3 — the all-[MASK] condition is already ahead,
by an amount nobody should read either way. The paper kept the mix on the
argument rather than on the number.
The mismatch does not materialise. Wettig et al. measure it directly and report that “the model can adapt to full, uncorrupted sentences, regardless of the use of alternative corruption strategies in pre-training”. Fine-tuning has enough signal to close the gap the rule was insuring against.
And the mix is worse. Against a 40%-masking all-[MASK] baseline,
80-10-10 loses on every task except SST-2, where the same-token predictions do
help. Read through the corruption/prediction decomposition the same paper
builds, the reason is legible: same-token predictions “neither count towards the
corruption nor to the prediction” — they neither remove context nor teach much —
and the loss on randomly-substituted tokens is slightly higher than on
[MASK], because the model must also work out whether a given input token is a
corruption at all.
The comparison and the recommendation to drop the mix both come from Wettig et al., LIT-672, who ran it at a 40% rate and explained it with their own decomposition. BERT had tested the mix only at its 15% rate, where the two conditions were 0.1 apart.
Conditions
Encoder MLM, one group, one architecture family. Everything above is measured on masked language models between 51M and 354M parameters, on GLUE and SQuAD. Denoising objectives with a different corruption vocabulary are not covered.
SST-2 is a real exception, not noise to wave away. Same-token predictions help there, in both this comparison and BERT’s own C.2 feature-based column. If a single sentence-classification target is what you are optimising, check it.
This is a recommendation against an inherited default, which is a weaker
position than it sounds. Nobody has published the comparison in the other
direction either. The promote_when asks for that comparison from a second
group and explicitly refuses to count another model merely shipping the rule, or
merely dropping it — that is how the rule spread in the first place.
Known implementations
- None to name for the recommendation. For the rule it drops: BERT, RoBERTa, SpanBERT and DeBERTa all use 80-10-10, which is why the practice is worth stating.