nucleation

LIT-855: Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models

3 min read

Properties5
StatusActive — read 2026-10-09 (NOTE-660); worth reading as the first closed-form account of what a contrastive word-embedding algorithm learns under a rank constraint, and in what order. Replacing word2vec's logistic loss by its quartic Taylor approximation, and choosing symmetric reweightings whose aggregate weight is constant, turns the objective into unweighted symmetric matrix factorisation of M*, the matrix of relative deviations of word co-occurrence from independence (Theorem 1, proved). From vanishing random initialisation, gradient flow then learns the eigenvectors of M* one at a time, each by a sigmoidal step in time (ln(λ_k/σ²))/λ_k (Result 3, a derivation that drops terms, not a proof). The resulting embeddings track word2vec's features and benchmark scores closely, and much more closely than truncated factorisations of PMI or positive PMI. The analogy part is empirical: difference vectors of semantic pairs look like a spiked random matrix, and the spike's strength tracks analogy accuracy.
Tagsrepresentation-learning, learning-theory, anthology-candidate
ExtendsLIT-242 — On the Stepwise Nature of Self-Supervised Learning
Cited as “paper” byNOTE-660 — Closed-Form Training Dynamics in Word2Vec-like Models
Cited as “source” byTHEORY-182 — A rank-limited contrastive word embedding trained from small initialisation learns the top eigenvectors of the co-occurrence matrix's relative deviation from independence, one at a time in order of eigenvalue, not the best low-rank approximation of its unconstrained optimum, THEORY-183 — When the co-occurrence of words on a latent continuum depends only on their separation, a spectral word embedding places them on the Fourier modes of that continuum, so the circles and rippled curves of months, years and places come from a symmetry of the corpus statistics, THEORY-739 — For the quartic proxy of word2vec with tied weights, symmetric reweighting and constant weights, the rank-d minimisers are the top d eigenvectors of the relative deviation-from-independence matrix M*, scaled by the square roots of their eigenvalues, up to rotation; and from an initialisation aligned with those eigenvectors, gradient flow acquires each one in an independent logistic rise, in order of eigenvalue

Graph View

Dhruva Karkada, James B. Simon, Yasaman Bahri and Michael R. DeWeese (2025), Advances in Neural Information Processing Systems 38 (NeurIPS 2025), pp. 4507–4540 — ARXIV-2502.09863 (first posted 14 February 2025 as Solvable Dynamics of Self-Supervised Word Embeddings and the Emergence of Analogical Reasoning)

Key takeaways

  • The target is a deviation from independence. With tied weights and the quartic Maclaurin approximation of the word2vec loss, the objective is (1/4)Σ G_ij (W Wᵀ − M*)²_ij + const, where M_ij = (Ψ⁺P_ij − Ψ⁻P_iP_j) / (½(Ψ⁺P_ij + Ψ⁻P_iP_j)) and G_ij = Ψ⁺P_ij + Ψ⁻P_iP_j (Proposition 2). M agrees with PMI to third order in its argument (Appendix D.1) but is bounded, |M*_ij| ≤ 2 when the two reweightings are equal, so least squares no longer over-weights rarely co-occurring pairs.
  • Which low-rank solution is learned. When G is constant (Setting 3.1: symmetric Ψ⁺, Ψ⁻ with Ψ⁺P_ij + Ψ⁻P_iP_j = g), the problem is unweighted factorisation, and the global minima are the top-d eigenvectors of M* scaled by the square roots of their eigenvalues, up to a right orthogonal transformation (Theorem 1, by completing the square and Eckart–Young). For G of rank one the minimiser is also known (Proposition 2); for general G it is weighted low-rank approximation, which is NP-hard, and the paper does not treat it.
  • The order in which it is learned. From aligned initialisation each singular value follows Saxe et al.’s sigmoid, s_k²(t) = s_k²(0)λ_k e^{λ_k t} / (λ_k + s_k²(0)(e^{λ_k t} − 1)), realised at τ_k = (1/λ_k) ln(λ_k/s_k²(0)) (Lemma 3.1, proved). From vanishing random initialisation, the embeddings first align silently with the eigenbasis and then follow the same steps (Result 3). Result 3 is derived by discarding off-diagonal couplings and checking that they stay small; the authors call it a conjecture to be made rigorous. Early stopping and small d therefore both limit the number of eigen-features, not their character.
  • It is a good proxy for word2vec. On 2 billion Wikipedia tokens (V = 10,000, d = 200): Google analogies 68.0% for word2vec, 65.1% for the trained quadratic model, 66.3% for explicit factorisation of M*, 50.6% for truncated PPMI and 8.4% for truncated PMI. The leading principal directions of word2vec and the quadratic model are the same interpretable topics. An ablation shows the clean stepwise separation comes from the reweighting, not from the quartic approximation.
  • Analogy structure, empirically. Difference vectors within a semantic class (male − female, verb − past tense) concentrate on a few eigen-features, and the spectrum of their Gram matrix looks like a Marchenko–Pastur bulk plus one spike, the mean difference vector. The spike’s signal-to-noise ratio first rises and then falls as rank is added, and its maximum tracks analogy accuracy across classes. This part is observation with a manual fit, not theorem.

Standing in the record

Filed on 2026-10-09 at the owner’s request, with no stated context; it came in a batch with arXiv 2509.24914, 2504.12916, 2205.10343 and 2410.17770. It was read on its own merits (NOTE-660), and the reading produces THEORY-182.

It carries anthology-candidate: it is a theory of the training dynamics of an ML algorithm, published at NeurIPS, of the kind the anthology holds alongside word2vec (ANTH-LIT-604) and Levy and Goldberg (ANTH-LIT-612). It stays here for now beside the record’s spectral self-supervised line (LIT-242, THEORY-001, THEORY-009), which it extends to language data.

Backlinks