anthology-of-the-sota

THEORY-089: Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition

2 min read

Properties6
StatusProposed
Version3
Tagsrepresentation-and-encoding, signal-structure, concept-geometry
SourceLIT-602 — GloVe: Global Vectors for Word Representation, LIT-603 — Distributed Representations of Words and Phrases and their Compositionality, LIT-607 — Improving Distributional Similarity with Lessons Learned from Word Embeddings
ExtendsTHEORY-093 — Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily
Extended byTHEORY-094 — Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors

Graph View

Source

Pennington et al. (2014), LIT-602, read as NOTE-327, for the ratio account. Mikolov et al. (2013), LIT-603, read as NOTE-326, for additive composition.

The account

Co-occurrence probabilities are dominated by frequent words that go with everything. Their ratios cancel that out and keep what discriminates. GloVe’s Table 1: P(solid|ice)/P(solid|steam) = 8.9 and P(gas|ice)/P(gas|steam) = 0.085, while water and fashion give about 1. If wᵢ·w̃ₖ ≈ log P(k|i), then (wᵢ − wⱼ)·w̃ₖ ≈ log P(k|i)/P(k|j). A relation shared by many word pairs, such as male to female or country to capital, shifts the same ratios in the same way, so it becomes a common offset, and analogies can be solved by arithmetic. Adding vectors adds log context probabilities, which multiplies the distributions, so the sum picks out contexts both words share (Russian + river → Volga River).

What it explains

  • Why analogy arithmetic works in both count-based (GloVe) and prediction-based (skip-gram) vectors: both fit the same statistics
  • Why SVD on raw counts does poorly and on log counts much better (7.3% against 60.1% on analogies at 6B)
  • Why element-wise addition composes meaning

What it rests on

For skip-gram, the premise that dot products fit a log co-occurrence statistic is THEORY-093: SGNS’s optimum is w·c = PMI(w, c) − log k. This account extends that one. It adds that ratios of those statistics become vector offsets.

Support from a controlled comparison

LIT-607 adds the test the two sources lacked. If count-based and prediction-based vectors fit the same statistics, they should converge once their design choices are aligned. Tuned alike, PPMI, SVD, SGNS and GloVe show no consistent winner, and the gaps change sign from task to task. That is indirect evidence: it shows convergence in benchmark scores, not that the vectors encode log ratios.

Where it is weak

  • A derivation from each model’s objective, not a measurement. Neither paper checks that vector differences match log ratios
  • Shown by analogy accuracy and hand-picked examples
  • Word-level, static embeddings. Whether anything like it holds for contextual representations is outside both sources

Backlinks