nucleation

LIT-977: Distributional hypothesis as isomorphism between word-word co-occurrence and analogical parallelograms

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-781) in full; worth reading for Theorem 1, an exact condition in a toy corpus: eight subjects carrying every combination of three binary aspects have raw co-occurrence rows forming a parallelepiped if and only if, for each object, its four sentences are equally probable. That is a data-side account in which aspects combine as a mixture, so raw counts and not PMI are linear. Read it knowing that the theorem covers one 24-sentence design and exact shapes, that its aspects are independent (a Boolean lattice), that the real-text analogy results need log counts and an SVD, and that the restricted word2vec baseline is trained on a dummy corpus. Registered for QUESTION-025; THEORY-271 is filed from it.
Version2
Tagsrepresentation-learning, linguistics, anthology-candidate
Cited as “paper” byNOTE-781 — Torii, Maeda and Hidaka — Distributional hypothesis as isomorphism
Cited as “source” byTHEORY-271 — In a toy subject–verb–object corpus whose eight subjects carry every combination of three binary aspects, the subjects' raw co-occurrence rows form a parallelepiped if and only if, for each object, its four sentences are equally probable; any other sentence distribution breaks it

Graph View

Takuma Torii, Akihiro Maeda and Shohei Hidaka (2024), PLOS ONE 19(10): e0312151.

Key takeaways

From the reading, NOTE-781.

  • Count models do most of word2vec’s work on analogies. The data are the English Wikipedia of 2017, with a window of 5 and d = 300, scored on the full Google set.

    model1,487 words10,072 words
    logfreq-SVDabove 61%50%
    ppmi-SVD55%51%
    word2vec, negative sampling, restricted vocabulary38%40%
    word2vec, full corpus75%69%
    raw rowsunder 6%under 5%

  • Negative sampling is not needed. word2vec trained by full-softmax cross-entropy on the co-occurrence probabilities reaches about 56–60%. The restricted negative-sampling baseline is trained on a dummy corpus whose context words are independent given the centre, so the comparison with it is confounded.

  • Theorem 1. In a corpus of 24 subject–verb–object sentences, the eight subjects carry every combination of Sex, Royalty and Number, and each verb expresses one aspect through its object. The subjects’ raw co-occurrence rows form a parallelepiped if and only if each object’s four sentences are equally probable (Eq. 5). A random non-uniform distribution gives no parallelogram at all.

  • Why it holds. Each sentence carries one aspect, so a subject’s row is a sum of per-aspect contributions. Under Eq. 5 each contribution depends only on the aspect’s value, and the row is affine in the ±1 aspect vector.

  • Natural text. One “parallelepiped-like” set of Family-category words appears in a 2D PCA, which “merely demonstrates the ‘existence’” of one.

Standing in the record

Filed on 2026-10-12 for QUESTION-025, and read on 2026-10-14 in NOTE-781. It was registered as “a derivation of why linear analogy structure follows from co-occurrence statistics” and “a second account of the premise that THEORY-185 derives”. The reading corrects both. The derivation covers one toy corpus, raw counts and exact shapes. Its aspects combine additively, as a mixture over sentences, where THEORY-185’s multiply. So it is a rival account for a different generative structure: there the PMI is linear in the attributes, here the raw counts are. Theorem 1 is filed as THEORY-271. For QUESTION-025 its aspects are independent and its lattice Boolean, the case the question says does not settle it. Its one bearing is that, with the incidence held Boolean, any interaction in sentence probabilities breaks the exact shape. It also connects to THEORY-183’s reading of geometry as a symmetry of the statistics. The analogy benchmarks are ML practice on word embeddings, hence the anthology-candidate flag.

Backlinks