Baroni, Dinu and Kruszewski, University of Trento — ACL 2014, 238–247, DOI 10.3115/v1/P14-1023.
Key takeaways
- The question (§1). Prediction-trained vectors were spreading with “triumphalist overtones” and almost no direct comparison against count vectors. The authors are count-vector researchers, and say their “secret wish” was to find it was all hype.
- The setup (§2). One corpus (ukWaC + Wikipedia + BNC, 2.8B tokens) and 300K words for both. Count: windows 2 and 5, PPMI or LMI, full or SVD/NMF at 200–500 dimensions, 36 models. Predict: word2vec CBOW only, windows 2 and 5, hierarchical softmax or 5/10 negatives, subsampling on or off, 200–500 dimensions, 48 models.
- The benchmarks (§3). Relatedness (rg, ws, wss, wsr, men), TOEFL synonyms, categorization (ap, esslli, battig), selectional preference (up, mcrae) and the Google analogies.
- Predict wins almost everywhere (Table 2). With the best setting per task: rg 84 vs 74, ws 75 vs 62, TOEFL 91 vs 76, analogies 68 vs 49. Selectional preference is the one tie.
- Predict is far more robust (Table 2, worst setup). The worst count model collapses (rg 11, analogies 1), while the worst predict model stays usable (rg 74, analogies 27).
- Within each family (Tables 3–4). Count: PPMI beats LMI, and uncompressed beats SVD beats NMF. Predict: negative sampling beats hierarchical softmax, and every top-10 model uses subsampling.
- The implementation matters as much as the family (§4). Collobert and Weston’s predict vectors do worse than the count models. “Had we … based our systematic comparison … on the cw model, we would have reached opposite conclusions.”
Standing in the anthology
It is the claim LIT-607 reversed, and is filed to record both. Levy,
Goldberg and Dagan (2015) trace this result to settings that were not
equalized. word2vec ran with its tuned smoothing and negative sampling, which
the count models were not given, and SVD used its worst eigenvalue weighting.
Tuned alike, no family wins consistently. LIT-607 is filed as corrects
this paper.
The recommendation is filed as retired. SOTA-380, “use
prediction-based vectors rather than count-based ones”, is Rejected, and
SOTA-379 (tune the baselines alike before crediting a method) is filed as
corrects it. The paper was cited for years as the evidence for embeddings
over counts. DP-005 applies: citation is adoption, not evidence.
What survives. The benchmarks and the within-family findings. Negative sampling beating hierarchical softmax and PPMI beating LMI agree with the record’s other sources. That every top predict model subsamples is independent support for SOTA-374 on this benchmark mix. LIT-607’s ablation finds the opposite for analogies, and the practice is contested for that reason.
The robustness result is not reversed, only explained. Predict was robust across its grid and count was not. Part of that is the grid: the count side’s worst cell combined LMI, NMF and a narrow window, and LIT-607’s point is that the count side lacked the settings that make it robust.
What it does not establish
- “Systematic” means within the grid the authors chose. The authors say this themselves: options they did not consider “would have improved count vector performance somewhat”. LIT-607 found those options.
- CBOW only. Skip-gram, the stronger word2vec model in later comparisons, is not tested.
- Uneven analogy search. Analogy accuracy depends on vocabulary size, which differs from the state-of-the-art systems it is compared with.
- No significance tests, and several benchmarks are tiny: rg has 65 pairs and esslli 44 concepts. The paper blames esslli’s brittleness on its size.