anthology-of-the-sota

SOTA-379: Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike

2 min read

Properties7
StatusActive
Version2
Consensusunreplicated
Tagsanalysis-and-evaluation, representation-and-encoding
SourceLIT-607 — Improving Distributional Similarity with Lessons Learned from Word Embeddings
Introduced byLIT-607 — Improving Distributional Similarity with Lessons Learned from Word Embeddings
CorrectsSOTA-380 — Use prediction-based word vectors rather than count-based distributional vectors

Graph View

Source

Levy, Goldberg and Dagan (2015), LIT-607, read as NOTE-329.

Do this

  • List the choices bundled with the method under test, including those that appear only in its code: window weighting, subsampling, the negative shift log k, the 0.75 smoothing exponent, w + c, eigenvalue weighting and normalization.
  • Give each baseline every choice it can express. PPMI takes shifted and smoothed PMI. SVD takes those and symmetric eigenvalue weighting. GloVe takes w + c.
  • Tune all methods over the same space, on held-out data. In the source, 2-fold cross-validation came within about 1 point of test-set tuning.
  • Report per task. The winner changes from task to task.

Why

Baroni et al. (2014, LIT-608) found prediction-based embeddings well ahead, and recommended them (SOTA-380, now Rejected; this practice corrects it). They had compared word2vec with its recommended settings against vanilla PPMI and SVD, with SVD at its worst setting. Equalizing the settings removes the consistent gap. GloVe’s win over word2vec reverses the same way. Tuning is worth up to 15.7 points over vanilla, more than the gap between methods.

The recommendation, and the evidence for it, are LIT-607’s. Levy, Goldberg and Dagan named the choices bundled with word2vec and GloVe, ported them to PPMI and SVD, and tuned all four methods alike across 672 representations on 8 similarity and analogy datasets. On similarity SVD matched or beat SGNS on average at windows 2 and 5, and only MSR’s syntactic analogies kept a large gap. Their conclusion, to give baselines the same design choices and tuning before crediting a method, is what this practice files.

Conditions

  • Similarity and analogy benchmarks on English Wikipedia. Downstream tasks were not tested.
  • “The same space” means where applicable. GloVe cannot express shifted PMI or smoothing, so identical treatment is not always possible. Say which choices each method was given.
  • Differences are reported without significance tests. Treat gaps of a point or two as ties unless tested.

Known implementations

  • hyperwords (the authors’ code)

Backlinks