nucleation

LIT-860: Symmetry in language statistics shapes the geometry of model representations

4 min read

Properties5
StatusActive — read 2026-10-09 (NOTE-664); worth reading as a derivation of the circles, rippled curves and curved sheets found in language-model representations of months, years and places from one property of the data: when the co-occurrence of two words on a latent continuum depends only on their separation, the matrix the embedding factorises is circulant or Toeplitz, so its eigenvectors are sinusoids of the latent coordinate. For a full-rank spectral word embedding the paper proves the principal components are Fourier modes with amplitudes set by the kernel's Fourier transform, solves the open-boundary exponential kernel exactly (a Sturm–Liouville problem), and bounds linear decoding of the coordinate by the missing Fourier tail (ε² ≲ r^(−1/D)). It measures the symmetry in Wikipedia statistics, shows the month circle survives zeroing every month–month co-occurrence, and explains that by a seasonal latent variable that couples many words. The "word embeddings" are explicit factorisations of the co-occurrence matrix, not trained networks; the language-model evidence is PCA pictures of Gemma 2 2B and EmbeddingGemma; the robustness argument is a perturbation sketch.
Tagsrepresentation-learning, mathematics, anthology-candidate
ExtendsLIT-863 — On the Emergence of Linear Analogies in Word Embeddings, LIT-862 — A mathematical theory of semantic development in deep neural networks
Cited as “paper” byNOTE-664 — Symmetry in Language Statistics and Representation Geometry
Cited as “source” byTHEORY-022 — The reverse-engineered grokking network computes modular addition by multiplying characters of ℤ/p at a few frequencies, the same real two-dimensional irreducibles later found as circles in language models; in both the group comes from the task, not the network, THEORY-183 — When the co-occurrence of words on a latent continuum depends only on their separation, a spectral word embedding places them on the Fourier modes of that continuum, so the circles and rippled curves of months, years and places come from a symmetry of the corpus statistics, THEORY-185 — When each binary attribute of a word affects its co-occurrence independently and multiplicatively, the PMI is affine in the attributes with rank at most d + 1, so a spectral PMI embedding is a linear image of the attribute hypercube and parallelogram analogies hold exactly; over the full hypercube of words, each attribute's two values equally frequent, the raw co-occurrence ratio mixes in products of attributes, and keeps exact analogies only when the strengths are alike enough that every attribute mode outranks every product mode, THEORY-730 — For an embedding whose Gram matrix is the absolute co-occurrence deviation matrix at full rank, words on a periodic lattice whose positive and negative co-occurrence parts each depend only on separation have Fourier modes of the lattice as principal components, in degenerate cos/sin pairs with amplitude the square root of the kernel's Fourier transform, and the exponential kernel gives a closed loop with integer frequencies on a periodic lattice and an open curve with non-integer frequencies on an interval

Graph View

Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart and Yasaman Bahri (2026), Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:56137–56172 — ARXIV-2602.15029

Key takeaways

  • The object. For any word set S on a latent continuum (the 12 months, the years 1700–2020, the 48 contiguous US states), the paper studies the submatrix of M_ij = (P_ij − P_iP_j)/(½(P_ij + P_iP_j)), which agrees with PMI to third order (Appendix B.4). Word embeddings are the factorisation of M taken from LIT-855; the paper extends it to untied word and context matrices under a special initialisation, so that the Gram matrix W Wᵀ is |M*|, the matrix absolute value (Appendix B.2).
  • The assumption. Translation symmetry: P_ij = P_iP_j C̃(dist(x_i, x_j)) for i, j in S, so M_S depends only on the separation of the latent coordinates (Assumption 3.1, with M positive semidefinite; Assumption B.1 asks the same of M⁺ and M⁻ separately).
  • The theorems. With d ≥ rank M* (no compression), the PCA coordinates of the embeddings are sinusoids of the latent coordinate, in degenerate cos/sin pairs, with amplitude the square root of the kernel’s Fourier transform at that wavevector (Proposition 1, any lattice dimension, periodic boundary; the proof is the diagonalisation of circulant matrices). For the exponential kernel e^(−|Δx|/σ) the amplitudes are √(2σ/(1 + σ²k²)): periodic boundary gives integer frequencies and a closed loop (Corollary 2); open boundary gives non-integer wavenumbers fixed by tan k = −σk (odd modes) and a centred-cosine condition (even modes), solved exactly as a Sturm–Liouville problem (Proposition 3). Projections onto any two components are Lissajous curves; the “ripples” and the “calendar Pringle” are the higher harmonics.
  • Linear decoding. If the eigenvalues fall with |k|, the best rank-r linear probe reconstructs the coordinate’s truncated Fourier series, and the relative error is at most (6/π²)(L²/(L² − 1))((r/Vol_D)^(1/D) − √D/2)⁻¹, so ε² ∼ r^(−1/D) (Proposition 4, periodic lattice). Measured on years 1900–2020 with a double-descent peak at the interpolation threshold.
  • The measurement. Month and year submatrices of Wikipedia’s M* are close to circulant and Toeplitz and fit a (periodised) exponential kernel off the diagonal (Figures 5–6). M* itself is not positive semidefinite: its positive and negative parts have comparable rank (Figure 7), so the fitted kernel is not the one factorised, and absolute amplitudes are mispredicted while relative amplitudes and wavenumbers fit.
  • Collective robustness. Zeroing every month–month entry of M* still yields the month circle at d = 1000 ≪ V = 25,000, and ten seasonal words suffice to recover the order, while seventeen number words do not (Figures 4, 15–17). A model in which a seasonal latent variable modulates many words makes the PMI circulant with eigenvalues growing with the number of words N, so a fixed perturbation leaves the top eigenvectors in place “using the Weyl or Davis–Kahan theorem” (§4.1, a sketch). Appendix D adds binary attributes (Korchinski et al.’s analogy model) and shows the seasonal and attribute subspaces are orthogonal (Theorem 5).
  • Language models. PCA of Gemma 2 2B residual activations (last token of a template prompt) and EmbeddingGemma vectors shows the same circles, curves and slowly varying geographic modes (Figures 1, 3, 12, 14). The comparison is visual; nothing ties the models’ geometry causally to the co-occurrence statistics.

Standing in the record

Filed on 2026-10-09 at the owner’s request, with no stated context. It came together with arXiv 2510.02670, and directly after the batch of five papers on the theory of trained networks (LIT-855 to LIT-859; see the curation entry of that day). It was read on its own merits (NOTE-664), and the reading produces THEORY-183.

It is the sequel of LIT-855 (two of its authors are shared): it takes that paper’s result that a word embedding is a spectral factorisation of M* (THEORY-182) and asks what the eigenvectors look like for words that lie on a continuum. It supplies a data-side account of the circles that LIT-322 found in language models and that THEORY-022 reads as real irreducibles of a group, and its degenerate cos/sin pairs are an instance of THEORY-019.

It carries anthology-candidate: it is about how concepts are laid out in model representations and about the statistics of language that produce them, which the anthology’s concept-geometry and signal-structure topics cover. It stays here for now beside LIT-855 and the record’s symmetry-and-spectrum line.

Backlinks