anthology-of-the-sota

THEORY-092: Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores

2 min read

Properties6
StatusProposed
Version2
Tagssignal-structure, representation-and-encoding
SourceLIT-607 — Improving Distributional Similarity with Lessons Learned from Word Embeddings
ExplainsSOTA-375 — When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution
ExtendsTHEORY-093 — Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily

Graph View

Source

Levy, Goldberg and Dagan (2015), LIT-607, §2.1, §3.2 and §6.2, read as NOTE-329.

The account

PMI(w, c) = log P(w, c) / (P(w) P(c)). A context that is rare overall has a tiny P(c), so a single co-occurrence gives it a high score. In a Zipfian vocabulary, the top-scoring contexts of a word are then often rare words that appear with it by chance and not with its neighbours. This is PMI’s known weakness: it overweights infrequent events.

Smoothing replaces P(c) with count(c)^α / Σ count^α, for α < 1. Rare contexts gain probability relative to frequent ones, so their PMI falls. In SGNS the same exponent sets the distribution negatives are drawn from. Since SGNS implicitly factorizes shifted PMI (THEORY-093), drawing negatives from count^0.75 is the same smoothing applied inside the objective.

The rare-context argument is LIT-607’s (§3.2), and the measurement beside it is the paper’s ablation of context-distribution smoothing, α = 0.75 against 1 (Table 8d): PPMI gains between 0 and 9.2 points, SVD between −0.3 and +2.2, SGNS between 0 and 1.4.

What it explains

  • Why the 3/4 exponent helps word2vec, a result its source asserted without numbers (SOTA-375).
  • Why smoothing helps PPMI most (up to +9.2 on MSR analogies) and SGNS least (0 to +1.4). SGNS’s sigmoid and frequency-weighted loss already damp extreme and rare cells, so it has less left to fix.
  • Why wide windows hurt PPMI and SVD. More random co-occurrences with rare words enter the vector with high PMI (footnote 10).

Where it is weak

  • Argued, not measured. Nothing isolates rare contexts to show they are what smoothing fixes. The evidence is the pattern of gains across methods.
  • One alternative exponent. Only α = 1 and 0.75 are compared, so the account explains why smoothing helps, not why 3/4 in particular.
  • GloVe cannot be tested. Its learned biases absorb this kind of correction by construction.

Relation to other accounts

THEORY-089 says word vectors behave linearly because they fit log co-occurrence statistics. This account is about which of those statistics are noise: the rare-context cells that PMI inflates. The two are compatible and neither depends on the other.

Backlinks