Source
Arora, Li, Liang, Ma and Risteski (2016), LIT-613.
The account
The model. A discourse vector drifts slowly over the unit sphere. At each step a word is emitted with probability ∝ exp(⟨c, v_w⟩). Word vectors point in all directions about equally (isotropy), and their lengths vary, which produces a power law of word frequencies.
Why PMI is low-rank. With isotropic vectors, the normalizer Σ_w exp(⟨c, v_w⟩) is nearly the same for every discourse direction. So the discourse can be integrated out in closed form: two words close together share almost the same c, and averaging exp(⟨c, v_w + v_w’⟩) over the sphere gives exp(‖v_w + v_w’‖²/2d). Taking logs, PMI(w, w’) ≈ ⟨v_w, v_w’⟩/d, up to a constant set by the window size. This holds at practical dimensions, which THEORY-093’s argument could not reach.
Why relations are lines despite error. Each fitted inner product is noisy, with about 17% termwise error for PMI. That noise is larger than the margin between the right and wrong answers to an analogy. But if a relation shifts log co-occurrence ratios by a common factor, then v_a − v_b solves a linear regression whose design matrix is the whole set of word vectors. Isotropy makes that matrix well conditioned, so the regression averages the noise down by √(d/n). The offset comes out clean even though no individual inner product is.
Both steps are proved in LIT-613 from the random-walk model: the nearly constant partition function (Lemma 2.1), the PMI relation (Theorem 2.2), and the √(d/n) shrinkage of noise in a relation’s offset (Theorem 4.1). Its check that real embeddings fit the model is the partition function concentrating mostly within [0.9, 1.1] of its mean for its own vectors, GloVe and CBOW.
What it explains
- Self-normalization. Log-linear language models work when the partition function is ignored, because under isotropy it barely varies.
- GloVe’s bias terms, as squared norms, and its weighted least-squares form, as an approximate likelihood.
- Why low dimensions help analogies. Isotropy needs d ≪ n. “Smaller models generalize better” is not the reason: the objective never mentions analogies.
- Why relation offsets share one direction. Offsets load on the first singular direction of their relation (mean cosine 0.51) and not the second (0.035).
Where it is weak
- Isotropy is the load-bearing assumption and is checked loosely. Singular-value ratios of 1.4 and 10.1 are both called small constants. Later work on contextual and word embeddings reports strong anisotropy (a few dominant directions), and the record has not filed it.
- The PMI-level approximation error is not small, as the authors state.
- The generative model is a caricature. It has no word order and no syntax, and the most frequent words are down-weighted because they do not fit.
- Relations must satisfy the ratio premise of THEORY-089, which is itself assumed.
Relation to other accounts
It extends THEORY-093: the same target (shifted PMI), derived from a
generative model instead of an optimum, and valid at low rank. It extends
THEORY-089: the same offsets-from-ratios mechanism, with the missing
denoising step. LIT-606’s account of linear representations (THEORY-090)
takes directions as given and asks what they mean. This account says why
they exist in word vectors at all.