He et al. (2021) — ARXIV-2111.06377
Key takeaways
- The paper opens by asking why masked autoencoding worked in language and not in vision, and answers in three parts. (i) Architecture: convolutions cannot easily take mask tokens or positional embeddings — closed by ViT (LIT-587). (ii) Information density: language is “human-generated … highly semantic and information-dense”, while images are “natural signals with heavy spatial redundancy”, so “a missing patch can be recovered from neighboring patches with little high-level understanding”. (iii) The decoder’s role: in vision it emits pixels, “of a lower semantic level than common recognition tasks”, where BERT’s decoder can be a trivial MLP. Three diagnoses, three different fixes.
- The 75% masking ratio is found by a sweep and explained by (ii). Figure 5 sweeps the ratio under both protocols; the paper calls the optimal ratios “surprisingly high” and finds 75% “good for both linear probing and fine-tuning”, five times BERT’s 15%. The reason it gives is (ii): high masking “largely eliminates redundancy, thus creating a task that cannot be easily solved by extrapolation from visible neighboring patches”. The redundancy itself is argued, not measured.
- The asymmetric encoder-decoder is where the speed comes from. The encoder sees only the visible patches — mask tokens never enter it — and a lightweight decoder takes encoded visible patches plus a shared learned mask token and reconstructs pixels. “Shifting the mask tokens to the small decoder … results in a large reduction in computation”: 3× or more faster pretraining and less memory, with no specialised sparse kernels. The decoder is discarded afterwards.
- Decoder depth sets the semantic level of the representation, and the two
evaluation protocols disagree about it. A deep decoder matters for
linear probing; a single-block decoder already reaches 84.8%
fine-tuned. The masking-ratio sweep shows the same split in sensitivity, though
not in where the optimum sits: linear probing rises steadily to the sweet
spot (a gap of up to 54.6% against 73.5%), while fine-tuning works across
a wide range (40–80%), and 75% suits both. Reporting one
protocol would have produced a different paper, which is
SOTA-196’s claim arriving from a third direction. - Numbers. ViT-Huge 87.8%, the best among methods using only ImageNet-1K data; downstream transfer beats supervised pretraining. For scale: training a supervised ViT-L from scratch is “nontrivial” and needs strong regularisation to reach 82.5%.
Standing in the anthology
Unit E of #304, and filed because the record already argued against it
without holding it. SOTA-250 — “Pretrain by predicting representations of
masked regions, not their pixels” — names MAE as the competitor it beats on
compute, and LIT-216 is the paper that ran that comparison. The record held
the winner of a named comparison and not the loser.
From MAE’s side the comparison reads as follows. I-JEPA (LIT-216) reports a ViT-H/14 at 300 epochs reaching 79.3% ImageNet linear-probe accuracy against 77.2% for MAE’s ViT-H/14 at 1600 epochs, and calls its pretraining over 10× more efficient than MAE’s at that size, although computing targets in representation space costs it about 7% more per iteration. Under full fine-tuning MAE stays ahead: 87.8% for its ViT-H/14 at 448 against I-JEPA’s 87.1% for a ViT-H/16 at 448, trained for 5.3× fewer epochs. The two protocols disagree again, as they do over MAE’s decoder depth.
What this note deliberately does not do is relitigate that. SOTA-250’s
position stands: predicting representations beats predicting pixels on the
evidence the record holds. The two practices sourced here —
SOTA-372 and SOTA-373 — are the parts of MAE that are
orthogonal to the target choice. An encoder that skips mask tokens and a
masking ratio set by redundancy are both needed whether you reconstruct
pixels or predict representations, which is why they outlived the argument
they arrived in.
The information-density claim is filed separately as THEORY-088, since it is a statement about signals rather than a recommendation.