Source
Baroni, Dinu and Kruszewski (2014), LIT-608, read as NOTE-330.
What it said
Train word vectors by predicting contexts (word2vec) rather than by counting co-occurrences and reweighting them. On one 2.8B-token corpus and 14 lexical-semantics benchmarks, the predict models won most tasks by a wide margin and degraded far less under bad settings.
The recommendation is Baroni, Dinu and Kruszewski’s in LIT-608, which compared 36 count models with 48 word2vec CBOW models and told readers to “go for the predict models”. With the best setting per task, predict beat count 84 to 74 on rg, 75 to 62 on ws, 91 to 76 on TOEFL and 68 to 49 on the Google analogies, and tied only on selectional preference. The worst count model fell to 11 on rg and 1 on analogies, where the worst predict model kept 74 and 27.
Why it is retired
The comparison was not like for like. word2vec ran with the settings its authors had tuned: negative sampling (which shifts PMI by log k) and a noise distribution raised to 0.75 (which smooths the context distribution). The count models got neither. Their SVD variants also kept the singular-value weighting, which LIT-607 found costs up to 15 points.
Equalized, the gap goes. LIT-607 ports those choices to PPMI and SVD and tunes all methods on one corpus. SVD matches or beats skip-gram on similarity at small windows, and no family wins consistently. Only MSR’s syntactic analogies show a large prediction advantage.
The paper foresaw the mechanism. The authors note that basing the comparison on Collobert and Weston’s vectors “would have reached opposite conclusions”. The conclusion was about the two configurations, not the two families.
What still holds
- Prediction-trained vectors are a good default. SGNS “does not significantly underperform in any scenario” and is the cheapest to train (LIT-607). That is a claim about one method, not about the family.
- Their robustness across the grid tested here is not in dispute.