Source
Mikolov et al. (2013), LIT-603. Read as NOTE-326.
The practice
- Noise distribution Pₙ(w) ∝ count(w)^{3/4}, for negative sampling or NCE
- Use k = 5–20 negatives for small data and 2–5 for large, in the authors’ experience
Evidence
The recommendation starts in LIT-603, where Mikolov et al. defined negative sampling for skip-gram and chose U(w)^{3/4}/Z as its noise distribution. As evidence that paper offers only the assertion in the first bullet.
- Mikolov et al. give none. “Significantly” better “on every task we tried”, with no table.
- Levy, Goldberg and Dagan measure it (LIT-607, Table 8d), as context-distribution smoothing against α = 1, over 8 similarity and analogy datasets. SGNS gains 0 to +1.4 points and never loses. PPMI gains up to +9.2 and SVD up to +2.2. The authors call it the one setting that can be applied blindly. THEORY-092 is their account of why.
Why still Proposed
promote_when asks for several exponents and the uniform distribution at
matched compute. The measurement compares two values, 0.75 and 1. It
establishes that smoothing helps, and helps SGNS only a little. It does not
establish that 3/4 is the right exponent.
Conditions
- The gain for skip-gram itself is small. Most of the measured benefit goes to count-based PMI methods
- Word-level skip-gram. Other tasks and item distributions may want a different exponent