Source
- Korchinski, Karkada, Bahri and Wyart (2025), LIT-863, the Theorem of §5 with Appendix A1, Eq. 12 and Results 1–4 of §6, §§7–9, Appendices A4–A5 and Figures 1–3, 6–10, as read in NOTE-665.
- The combination with continuous attributes: Karkada, Korchinski, Nava, Wyart and Bahri (2026), LIT-860, Appendix D (Theorem 5), as read in NOTE-664.
What was actually shown
The theorem. Let word i be α_i ∈ {−1, +1}^d, and let P(i, j) = P(i)P(j) ∏_k P^(k)(α_i^k, α_j^k), with each 2 × 2 factor fixed by normalisation up to a strength s_k. The ratio M = P(i, j)/P(i)P(j) is then the Kronecker product of the factors. Its eigenvectors are products: the constant, d attribute modes v_k(i) ∝ α_i^k with eigenvalue ∝ s_k, then modes ∝ s_k s_k′ carrying products of attributes, and so on. Its logarithm, the PMI, is exactly δ11ᵀ + ADAᵀ + Aη1ᵀ + 1ηᵀAᵀ, with A the words-by-attributes matrix. So the PMI has rank at most d + 1, every eigenvector is affine in the attributes, and a spectral embedding satisfies W_A − W_B + W_C = W_D whenever α_D = α_A − α_B + α_C, for any strengths. Over the full, balanced hypercube, an embedding of M does so only if K ≤ d + 1 and every attribute mode outranks every product mode. The paper reaches that with small, narrowly spread s_k, and the exact condition is in the note of 2026-10-19 below. Otherwise the product modes interleave with the attribute modes, and accuracy rises and then falls with K. With pairwise-correlated attributes the PMI keeps rank ≤ d + 1 and affine eigenvectors, which then mix attributes (A5).
Robustness. I.i.d. noise of scale σ in the PMI has norm about 2σ2^(d/2), against attribute eigenvalues of order 2^d. Analogies survive until about 2^(d/2)/σ noise modes are admitted, and the rescaled curves collapse. A random 15% of the vocabulary keeps the spectrum up to scale. Zeroing every pair that differs only in one attribute leaves that attribute’s direction in place.
Wikipedia. Spectral embeddings of M and log(M + 10⁻²) over 10,000 words reproduce the pattern on Mikolov et al.’s analogies: the PMI beats M and saturates with K. Zeroing a family’s own pairs costs little. The PMI’s spectrum is broad and near log-normal, as in the model with broadly spread strengths. That comparison could have come out otherwise: the PMI could have been near-isotropic, as Arora et al.’s model implies, or deleting a family’s own pairs could have destroyed it.
What this does not say
- Not that language is made of independent binary attributes. The model posits them. No attribute, strength or dimension d is recovered from the data, and the ratio condition that earlier accounts postulated holds exactly in the model because independence builds it in. The account explains analogies by an assumption about the statistics; the assumption itself is untested.
- Not that trained embeddings do this. Every embedding is a truncated eigendecomposition. That word2vec learns the top eigenvectors of M*, which agrees with the PMI to third order, is THEORY-182, shown only for a tied quartic proxy.
- Not proved robust. The perturbation steps bound eigenvalues by Weyl’s inequality and assert that eigenvectors follow; no eigenvector bound is given. The deletion experiments zero entries of the matrix, which is not deleting sentences from a corpus.
- Not that analogies need K ≥ d. Equation 11 can hold with K < d + 1 for analogies within the kept attributes, but the nearest-neighbour score may then be degenerate, as the paper’s d = 2 example with K = 2 shows.
- Not an account of language models. The Discussion’s pointer to linear subspaces in LLMs is a motivation, and nothing is tested there.
- Not a symmetry account by itself. In the balanced case the eigenvectors are the Walsh characters of (ℤ/2)^d acting by flipping attribute values, a case of THEORY-019, the binary counterpart of the Fourier modes of THEORY-183. But affineness in the PMI comes from the logarithm turning a product into a sum, and it holds with the symmetry broken (q_k ≠ 1) too.
- Not an instruction. It says what such an embedding contains, not how to build one.
Connections
The Lattice Representation Hypothesis (LIT-267, read in NOTE-240) assumes that binary attributes have linear directions in a model’s representations, thresholds them into an object–attribute incidence, and reads its concept lattice as the model’s. This result derives that premise from co-occurrence: under independent attributes each one is a linear direction of the PMI embedding, and thresholding recovers the incidence exactly. In this model the hypothesis’s first step is a theorem.
The same independence empties its second step. Every combination of attributes is possible, so the incidence is the full hypercube and its concept lattice is Boolean, with no implications among attributes. The structure a concept lattice is for, attributes that imply or exclude each other as in LIT-267’s WordNet hierarchies or CLAIM-119’s categories that share attributes and differ in a few constraints, is what this model excludes. Whether the linear directions survive it is QUESTION-025.
An additive rival. THEORY-271 (Torii, Maeda and Hidaka, LIT-977, read in NOTE-781) gets an exact parallelepiped in a toy subject–verb–object corpus from a different premise. There aspects combine as a mixture over sentences, so counts add across aspects, and the shape is in the raw co-occurrence rows; the logarithm would break it. Here attributes multiply, and the shape is in the PMI. The two agree that parallelograms are in the data and disagree about which transform shows them, because they assume different ways for attributes to combine. It is a rival account for a different generative structure, not a second derivation of this one.
A supervised counterpart of Appendix A5. With pairwise-correlated attributes, the affine eigenvectors here mix attributes (A5). THEORY-292 (Erogullari et al., LIT-937, read in NOTE-816) finds the same in CNN image classifiers for concept directions fitted one at a time, and adds that a joint fit finds orthogonal directions that still separate each concept. A direction found from one statistic at a time mixes correlated attributes; a joint fit can separate them again. The setting is labelled image attributes, not co-occurrence.
Note of 2026-10-19: the theorem, re-derived
Any function on {±1}² is a combination of 1, a, b and ab, and a symmetric one has equal coefficients on a and b. So log P^(k)(a, b) = δ_k + η_k(a + b) + γ_k ab, and summing over attributes, log M = δ11ᵀ + Aη1ᵀ + 1ηᵀAᵀ + ADAᵀ entrywise, for any set of words. Every term’s columns lie in the span of 1 and the attribute columns, so the rank is at most d + 1. Since the PMI is symmetric, every eigenvector with a non-zero eigenvalue lies in that span and is affine in the attributes. A spectral embedding that keeps such modes is W_i = c + Bα_i. So α_D = α_A − α_B + α_C gives W_A − W_B + W_C = W_D, whatever the strengths.
For the raw ratio, with all 2^d words, M = ⊗k P^(k). In the balanced case each factor has eigenvalues 2 and 2s_k on the constant and on α^k. The mode that takes α^k from the attributes in a set S has eigenfunction ∏{k∈S} α^k and eigenvalue 2^d ∏{k∈S} s_k. With K ≤ d + 1 modes, an embedding of M is affine, and so gives parallelograms exactly, iff no product mode (|S| ≥ 2) is kept. That holds iff min_k s_k > s(1)s_(2), the product of the two largest strengths. Equal strengths satisfy it at any strength below 1. So what matters is that the strengths be alike relative to their size, and the source’s “small and narrowly spread” is a sufficient first-order regime. Without balance, each factor’s top eigenvector is not constant, and even the top mode of M is a product. Whether real co-occurrence is close to the premise is QUESTION-178.