Source
Karkada, Simon, Bahri and DeWeese (2025), LIT-855, Theorem 1 and Proposition 2, with Lemma 3.1, as read in NOTE-660, whose reader followed the proofs step by step. Re-derived here as a check.
What was actually shown
The quartic expansion of the skip-gram negative-sampling loss, with the two embedding matrices tied, symmetric reweightings and constant weights G_ij = c, is L(W) = (c/4)‖WWᵀ − M*‖²_F + const, where M*_ij = (Ψ⁺P_ij − Ψ⁻P_iP_j) / (½(Ψ⁺P_ij + Ψ⁻P_iP_j)).
- The minimisers. L depends on W only through X = WWᵀ, which is positive semi-definite of rank at most d. Write M* = Σ λ_i v_i v_iᵀ with λ₁ ≥ λ₂ ≥ …, and let X have eigenvalues μ₁ ≥ … ≥ μ_d ≥ 0. Then ‖X − M*‖² = Σ μ_i² − 2 tr(XM*) + ‖M*‖². Von Neumann’s trace inequality gives tr(XM*) ≤ Σ_{i≤d} μ_i λ_i, with equality when X shares M*‘s top eigenvectors. So ‖X − M*‖² ≥ ‖M*‖² + Σ_{i≤d} (μ_i² − 2μ_iλ_i), and each term is least, over μ_i ≥ 0, at μ_i = max(λ_i, 0). The minimum is X = Σ_{i≤d} max(λ_i, 0) v_i v_iᵀ, unique when λ_d > λ_{d+1}. When the top d eigenvalues are positive, W = V_d Λ_d^{1/2} R for any orthogonal R.
- The aligned dynamics. Gradient flow is Ẇ = −c(WWᵀ − M*)W. If W(0) = V_d S(0) with S(0) diagonal, then W(t) = V_d S(t) solves it, with ṡ_k = c(λ_k − s_k²)s_k. For u_k = s_k² this is u̇_k = 2c u_k(λ_k − u_k), so u_k(t) = λ_k / (1 + (λ_k/u_k(0) − 1) e^{−2cλ_k t}). Mode k reaches half its final strength at t_k = (1/(2cλ_k)) ln(λ_k/u_k(0) − 1) and rises over a time of order 1/(cλ_k). As the initial scale shrinks, the modes arrive one at a time, in order of eigenvalue.
What could have failed: a loss that sees the two embedding matrices separately, or weights G_ij that vary, would not reduce to a function of WWᵀ alone, and the trace argument would not apply.
What this does not say
- Not from random initialisation. That a vanishing random start first aligns with the eigenbasis and then follows these steps is LIT-855’s Result 3, derived with couplings dropped and not proved. It stays with THEORY-182.
- Not that word2vec factorises M*. Word2vec unties the matrices and uses an asymmetric negative sampling. That it behaves like the proxy is an empirical match, also THEORY-182’s.
- Not unique embedding directions. The loss sees only WWᵀ, so any rotation R is equally optimal.