Contribution
A generative account of why PMI is low-rank. Earlier results said what word2vec fits at an unconstrained optimum (LIT-612) or assumed the linear structure (LIT-602). This paper derives the relationship in low dimensions from a model of how text is produced. It also shows why the linear structure of analogies survives even though every fitted inner product is noisy.
Key insight
Isotropy does two jobs. It makes the partition function nearly constant, which lets the hidden discourse be integrated out in closed form, giving the PMI relationship. It also makes the matrix of word vectors behave like a random design matrix, so solving for a relation’s offset averages away the per-word error. Low dimension is what makes isotropy possible (d ≪ n), so it is part of the explanation, not only a compression.
Assumptions
- Text generated by a slow random walk of a discourse vector, with log-linear emission
- Word vectors isotropic in bulk: Z_c nearly constant over directions, and random-like singular values
- Window size small relative to how fast the discourse moves
- For analogies: the relation shifts log co-occurrence ratios by a common factor, up to noise (GloVe’s premise)
Main results
- Lemma 2.1: Z_c ∈ (1 ± Õ(1/√n))Z for almost all c
- Theorem 2.2: log p(w,w’) = ‖v_w + v_w’‖²/2d − 2 log Z ± ε, so PMI = ⟨v_w, v_w’⟩/d ± O(ε)
- Corollary 2.3: for window q, add log(q(q−1)/2)
- Lemma 3.1: CBOW’s context average is the MAP discourse estimate
- Theorem 4.1: offset noise shrinks by a factor of √(d/n)
Key results
- Z_c concentrates mostly within [0.9, 1.1] of its mean, for SN, GloVe and CBOW (Fig 1)
- Squared norm against log frequency: r = 0.75 (Fig 2)
- Analogies (Table 1): SN 0.71 overall on Google against GloVe 0.73, CBOW 0.74, skip-gram 0.70
- Relation offsets: mean cosine 0.51 with the first singular direction and 0.035 with the second
- Exploiting repeated relations adds about 10 points, and a non-cheating variant about 3
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | under the model, PMI ≈ ⟨v, v’⟩/d in low dimensions | strong | Theorem 2.2, given its assumptions |
| C2 | the model’s assumptions hold for real embeddings | weak | loose checks on one corpus: concentration histograms, singular-value ratios |
| C3 | isotropy is why analogies survive fitting error | moderate | Theorem 4.1, plus relation offsets loading on one direction |
| C4 | GloVe’s biases are squared norms | moderate | derivation, plus r = 0.75 for SN’s own vectors (weaker for others) |
| C5 | analogy benchmarks reward exploiting repeated relations | strong | +10 points from the cheating solver |
Limitations
- The PMI-level error is comparable to PMI itself
- Word order is ignored
- The most frequent words are down-weighted because they do not fit
- The checks of isotropy use generous thresholds
Recommendations
None filed. The analogy-benchmark observation (repeated relations can be exploited) is recorded here, not as a practice.