anthology-of-the-sota

LIT-602: GloVe: Global Vectors for Word Representation

3 min read

Properties7
StatusActive
Version2
Tagsrepresentation-and-encoding, signal-structure, concept-geometry
Compared againstLIT-603 — Distributed Representations of Words and Phrases and their Compositionality, LIT-613 — A Latent Variable Model Approach to PMI-based Word Embeddings
Corrected byLIT-607 — Improving Distributional Similarity with Lessons Learned from Word Embeddings
Cited as “paper” byNOTE-327 — GloVe
Cited as “source” byTHEORY-089 — Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition

Graph View

Pennington, Socher and Manning, Stanford (EMNLP 2014) — DOI 10.3115/v1/D14-1162

Key takeaways

  • The argument from ratios (§3, Table 1): on 6B tokens, P(solid|ice) / P(solid|steam) = 8.9 and P(gas|ice) / P(gas|steam) = 0.085. For words related to both or neither (“water”, “fashion”) the ratio is about 1. Raw probabilities are dominated by frequent, non-discriminative words, while ratios cancel them
  • From that to a model: require vector differences to encode log ratios. Symmetry and a bias term give wᵢᵀw̃ⱼ + bᵢ + b̃ⱼ = log Xᵢⱼ. It is fit by weighted least squares over nonzero co-occurrences, with f(x) = (x/x_max)^α below x_max and 1 above. x_max = 100, and α = 3/4 gives “a modest improvement” over α = 1. The final vector is W + W̃
  • Relation to skip-gram (§3.1): skip-gram’s objective, summed over the corpus, is a weighted cross-entropy on the same co-occurrence statistics. GloVe is the “global” least-squares variant
  • Results (Tables 2–3): word analogy 75.0% (300d, 42B tokens). On 6B tokens: GloVe 71.7%, skip-gram 69.1%, CBOW 65.7%, SVD on log counts 60.1%. Word similarity, 42B: WS353 75.9 and RW 47.8. Context: 10 words either side, with counts weighted 1/distance
  • Against word2vec at equal time (§4.7, Figure 4): same corpus, vocabulary, window and dimension. Time is GloVe iterations against word2vec negative samples, because the word2vec tool runs one epoch. GloVe is ahead throughout

Standing in the anthology

Filed on request, with the two word2vec papers (LIT-604, LIT-603). It is compared_against the second. It is a source for THEORY-089.

signal-structure, because its founding argument is about language. Which statistic carries meaning is a claim about text, and the ice/steam table is a measurement of it.

The comparison with word2vec is on word2vec’s defaults. The authors say so: they “set any unspecified parameters to their default values, assuming that they are close to optimal”, and acknowledge this “should be relaxed”. Training time for word2vec is varied through the number of negative samples, because its tool could not run more epochs, and its accuracy falls past about 10 samples. The margin is 2.6 points at 6B tokens.

The re-tuned comparison reverses it. LIT-607 (Levy, Goldberg and Dagan 2015) trains both methods on one corpus, allows w + c for both and tunes both. SGNS beats GloVe on every task, including at 10.5B tokens. It is filed as corrects this paper on that claim. The formulation and the ratio argument are not in question.

A later generative model recovers GloVe’s form rather than only competing with it. Arora et al. (LIT-613) derive, from a random-walk model of text, a weighted least-squares objective with GloVe’s shape, in which the bias terms GloVe arrived at by trial come out as squared vector norms. On the Google analogies their objective scores 0.71 against GloVe’s 0.73 (their Table 1), and their check that the partition function is nearly constant holds for GloVe’s vectors as well as their own.

No practice. Its weighting function and constants are GloVe’s own. Under ADR-041 they stay in the reading.

Read — NOTE-327.

Backlinks