Source
Mikolov et al. (2013), LIT-603. Read as NOTE-326.
The practice
- Discard each occurrence of token w with probability 1 − √(t/f(w)), where f is its relative frequency. Tokens rarer than t are never discarded, and the frequency ranking is preserved
- Start at t ≈ 10⁻⁵ for corpora of billions of words
- Expect speed first and accuracy second. The run time fell by 2–3×. Accuracy gains are largest for rare items and phrases
The rule and these numbers are LIT-603’s (§2.3, Table 1), where Mikolov et al. introduced it for skip-gram. On word analogies with 300-dimensional vectors, subsampling took NEG-5 from 38 to 14 minutes and 59 to 60%, and hierarchical softmax from 41 to 21 minutes and 47 to 55%. On phrase analogies it took NEG-15 from 27 to 42% and hierarchical softmax from 19 to 47%.
Why speed and accuracy move together, which is the unusual part
Most throughput interventions trade quality away. The argument that this one need not: an update spent on a very frequent token is nearly redundant with the last thousand such updates, so dropping it costs almost no signal, and the budget it frees goes to tokens where each occurrence still carries information. The speed-up and the rare-word gain have one cause.
That is a claim about the data, not about the model, and it comes with a falsifier worth stating: on a signal whose units are near-uniform there is nothing redundant to discard, and subsampling becomes plain data loss. Run that test before assuming the practice transfers to a non-text stream.
Read the Conditions below against this. The mechanism argument predicts a
gain concentrated in rare items; LIT-607 finds the gain on similarity and
a loss on analogies. Those are compatible — the argument says where the
freed budget goes, not that every downstream task rewards it — but the
argument is not evidence for the analogy case, and was never tested against
it.
It is not vocabulary pruning. Nothing leaves the vocabulary; occurrences are dropped from the stream. A rare word’s representation improves precisely because its own occurrences survive while its neighbours’ do not.
Conditions
- Word2vec-style co-occurrence training. The rationale, that frequent tokens co-occur with everything, is a property of text (heavy-tailed word frequencies). It transfers to other embedding methods with the same structure, but that is not tested here
- Gains on word analogies were small or zero with enough negatives (NEG-15: 61 → 61%)
- An independent ablation finds it hurts analogies. LIT-607 (Table 8b), at matched tuning on 1.5B tokens: SGNS similarity +0.1 to +2.2, and Google and MSR analogies −4.4 and −5.4. For PPMI the analogy losses are 5.0 and 12.2. The best SGNS configuration used subsampling on 4 of 8 tasks. Use it for speed, and check the task before counting on an accuracy gain
- Baroni et al. (LIT-608) point the other way on their mix. Every one of their ten best CBOW configurations, ranked across 14 benchmarks mostly of relatedness and categorization, uses subsampling