anthology-of-the-sota

LIT-607: Improving Distributional Similarity with Lessons Learned from Word Embeddings

4 min read

Properties10
StatusActive
Version3
Tagsanalysis-and-evaluation, representation-and-encoding, signal-structure
ExtendsLIT-612 — Neural Word Embedding as Implicit Matrix Factorization
Compared againstLIT-603 — Distributed Representations of Words and Phrases and their Compositionality
CorrectsLIT-602 — GloVe: Global Vectors for Word Representation, LIT-608 — Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
Cited as “contested by” bySOTA-374 — When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵
Cited as “introduced by” bySOTA-379 — Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike
Cited as “paper” byNOTE-329 — Levy, Goldberg & Dagan: lessons from word embeddings
Cited as “source” bySOTA-375 — When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution, SOTA-379 — Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike, THEORY-089 — Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition, THEORY-092 — Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores

Graph View

Levy, Goldberg and Dagan, Bar-Ilan University — TACL 3 (2015), 211–225, DOI 10.1162/tacl_a_00134. Code: bitbucket.org/omerlevy/hyperwords.

Key takeaways

  • The question (§1). Baroni et al. (2014) found prediction-based embeddings beat count-based methods by a clear margin. But SGNS implicitly factorizes a shifted PMI matrix (Levy and Goldberg 2014, LIT-612), the same statistic count methods use. So where does the gap come from?
  • The answer is the unstated design choices (§3). word2vec and GloVe ship with choices that are “reported in passing” or “not even mentioned”, and each can be ported to PPMI and SVD. Pre-processing: dynamic context windows and subsampling. Association: shifted PMI (log k) and context-distribution smoothing (counts^0.75). Post-processing: adding context vectors (w + c), eigenvalue weighting and normalization.
  • Design matters more than method (Tables 2–4). Tuning adds up to 15.7 points over the vanilla setting, and more than 6 on average. Changing one hyperparameter often gains more than switching methods. Cross-validated tuning (Table 5) comes within about 1 point of the oracle.
  • No method wins consistently (§5.3, Table 5). On similarity, SVD beats SGNS on average at windows 2 and 5, and SGNS never leads SVD by more than 1.7 points. On Google analogies SGNS and GloVe lead PPMI by 3.7 points. Only MSR’s syntactic analogies show a large gap. Baroni et al.’s result (LIT-608) came from pre-tuned word2vec against vanilla PPMI and SVD, with SVD at its worst setting (eig = 1). It is filed as corrects that paper.
  • SGNS beats GloVe on every task (Table 5), on the same corpus with w + c allowed for both, and still at 10.5B words. The only exception is 3CosAdd, where GloVe leads by 0.8.
  • Context-distribution smoothing is the one safe default (§6.2, Table 8d). Raising context counts to 0.75 in PMI never hurts materially. It helps PPMI by up to 9.2 points (MSR), and SGNS by 0–1.4.
  • Other settings cut both ways (Table 8). Subsampling raises SGNS similarity by up to 2.2 and lowers its analogies by 4.4–5.4, and PPMI’s by 5–12. Shifted PMI helps SGNS and wrecks SVD. w + c swings from +4.7 to −8.9.
  • SVD “correctly” is SVD badly (§6.1, Table 6). Weighting word vectors by the singular values (eig = 1) drops average similarity from .616 to .534 at window 5. The symmetric variants (eig = 0.5 or 0) do much better.
  • More data does not always beat tuning (§5.2). At 10.5B words, 3 of 6 similarity tasks gain more from a wider hyperparameter search than from 7× the corpus. Analogies gain from data.

Standing in the anthology

It is the controlled comparison the word-vector papers did not run. LIT-604’s headline table compares vectors trained on different corpora and sizes. LIT-602 ran word2vec at its defaults against a tuned GloVe, which LIT-602 already flagged as a caveat. This paper tunes all methods on one corpus, and GloVe’s advantage reverses. It is filed as corrects LIT-602 on that claim. GloVe’s formulation and its ratio argument stand.

It supplies the numbers LIT-603 did not. The unigram^(3/4) exponent was asserted “significantly” better with no table. Here it is measured as context-distribution smoothing, against α = 1 only. It helps everywhere and helps SGNS least. SOTA-375 carries the measurement, and THEORY-092 carries the paper’s reason: PMI overweights rare contexts.

It qualifies SOTA-374. Subsampling costs 4–12 points on analogies for SGNS and PPMI while helping similarity. The practice now records that it is contested on this evidence.

It files one practice of its own, SOTA-379: before crediting an embedding method over count-based ones, give the baselines the same design choices and tune them alike. The same lesson appears for GNNs in SOTA-349 and SOTA-350, from other fields and other groups.

It supports THEORY-089. If the methods fit the same co-occurrence statistics, they should converge once their design choices are aligned. That is what Table 5 shows.

Eigenvalue weighting and w + c are not filed as practices. They are settings of particular methods (ADR-041), and w + c’s effect changes sign from task to task.

What it does not establish

  • English Wikipedia, word-level, similarity and analogy only. Footnote 8 says so: other tasks, data and hyperparameters “may yield other conclusions”. Downstream tasks are not evaluated.
  • No significance tests. “Insignificant differences” in the abstract means small, not tested.
  • The search space is the authors’ choice. GloVe keeps its default weighting function and cannot use shifted PMI or smoothing, so “tuned alike” means the same space where applicable. That is fair, and it is not identical treatment.
  • Smoothing is compared at α = 1 and 0.75 only. It shows 0.75 beats no smoothing. It does not show 0.75 is the best exponent.
  • The large-corpus comparison is SGNS against GloVe only. The count methods did not fit in memory.

Backlinks