Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Signal structure — what the data itself is like, independent of any model — the statistics and structure of language, images and other signals that methods exploit or must respect, such as frequency spectra, heavy-tailed and Zipfian distributions, and what counts as a unit.
12 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-022 v2 | A phrase placed in a word slot is read as a lemma, and the reading comes from the construction rather than the phrase’s frequency | Goldberg and Shirtz (2025), LIT-410 — English has a productive construction in which a unit with the internal syntax of a phrase, up to a whole sentence, occupies a slot reserved for a word, and comprehenders give it the kind of meaning a word has: it evokes a semantic frame presumed to be shared. Four preregistered surveys measure the reading, and the fourth removes the obvious deflation — the effect is the same size on high-frequency phrases, and estimated corpus frequency predicts nothing. So the lexical unit is delimited by construction, not by how often the string has been seen, which is the assumption a frequency-merge tokenizer makes. | Active |
| THEORY-032 v3 | A spectral update wins where the incoming activations are low stable rank and the gradient spectrum is spread out | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by exactly the ratio of the gradient’s nuclear rank to the incoming activations’ stable rank, and transformers sit where that ratio is large. A comparison of bounds, not of realized training. | Proposed |
| THEORY-056 v2 | Distinct state machines agree at chance on long random inputs, so a learner that reads only statistics loses the signal as the sequences get longer | Giapitzakis, Fountoulakis, Nichani and Lee (2025), LIT-495 — reading a uniformly random word drives a random walk on S_N × S_N, and the probability two distinct semiautomata land in the same state is 1/N + error with |error| ≤ (1 − 1/2N)^T. The quantity that identifies which machine you are looking at decays exponentially in sequence length, so more data of this kind is less evidence, not more. | Proposed |
| THEORY-057 v2 | A corpus a model generates is narrower than the distribution it imitates, and the narrowing shows up as n-gram over-concentration | Zhu et al. (2024), LIT-205, and Finke et al. (2025), LIT-485 — two groups, different corpora, no citation between them, both locating the damage in the same statistic. Generated text concentrates on the generator’s high-probability continuations, so the corpus is narrower than what it imitates; 59.38% of TinyStories contains “once upon a time”, and the proportion of synthetic pretraining data correlates negatively with downstream performance. | Proposed |
| THEORY-080 v2 | A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes | Hsueh et al. (2025), LIT-555 — blur removes high frequencies deterministically, so many images share a blurry one and the deblurring target is their mean. Stepping toward it leaves the manifold unless noise has already made those images indistinguishable. Natural spectra fall as 1/f² against flat noise, so blur is safe in a band only once noise dominates it. That puts the useful ratio near 0.5 on CIFAR-10. The FID sweep fits the account, and the manifold departure itself is not measured. | Proposed |
| THEORY-086 v2 | A text document is better modelled as a mixture of several topics, each a distribution over words, than as a draw from a single topic | Blei, Ng and Jordan (2003), LIT-592 — exchangeability of a document’s words implies, by de Finetti, a mixture over a latent parameter. LDA takes that parameter to be a per-document distribution over topics. It has lower held-out perplexity than a mixture of unigrams, which forces one topic per document, on AP newswire and C. elegans abstracts, at every number of topics. The multi-topic structure and LDA’s Dirichlet smoothing are not separated, and perplexity is the only measure. | Proposed |
| THEORY-088 v3 | Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal’s information density | Proposed | |
| THEORY-089 v3 | Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition | Pennington et al. (2014), LIT-602, and Mikolov et al. (2013), LIT-603 — what distinguishes words is the ratio of how often they co-occur with probe words (ice and steam with solid versus gas). A model whose dot products fit log co-occurrence turns ratios into vector differences, so relations become offsets. The same log-linearity makes a sum of vectors act like a product of context distributions, an AND of their contexts. Derived from the objectives and illustrated. The mechanism is not measured. | Proposed |
| THEORY-092 v2 | Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores | Levy, Goldberg and Dagan (2015), LIT-607 — PMI divides by the context’s probability, so a rare context seen once with a word gets a high score and crowds out the informative ones. Raising context counts to α = 0.75 before normalizing raises rare contexts’ probability and lowers their PMI. That is why word2vec’s smoothed negative distribution helps, and why the same smoothing helps PPMI most (up to 9.2 points) and SGNS least (at most 1.4). | Proposed |
| THEORY-093 | Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily | Levy and Goldberg (2014), LIT-612 — setting the derivative of SGNS’s per-pair objective to zero gives w·c = PMI(w, c) − log k, with k the number of negatives. So word2vec’s skip-gram is a factorization of the matrix count-based methods use, shifted by a constant, with each pair’s error weighted by how often it occurs. NCE gives log P(w|c) − log k the same way. Exact at the optimum with unconstrained dimensions. At practical dimensions it describes the target, not the result. | Active |
| THEORY-094 | Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors | Arora et al. (2016), LIT-613 — if each word is emitted with probability ∝ exp(⟨c, v_w⟩) as a discourse vector c drifts, and word vectors are isotropic, the partition function is nearly constant and PMI(w,w’) ≈ ⟨v_w, v_w’⟩/d holds in low dimensions. That carries THEORY-093 from its unconstrained optimum to practical sizes. Isotropy also makes solving for a relation’s offset a well-conditioned regression that averages away per-word error, which is the step THEORY-089 skipped. The assumptions are checked only loosely. | Proposed |
| THEORY-099 | Muon’s update matches the outer-product structure of associative memories, which is why it learns tail classes that Adam under-trains | Wang et al. (2025), LIT-654. A linear associative memory is a sum of outer products; orthogonalising the update treats every outer-product direction alike. So Muon produces more isotropic weight spectra than Adam, and on heavy-tailed data that shows up as tail classes being learned where Adam under-trains them. The ablation localises the effect: Muon on value-output weights and the FFN nearly recovers full Muon, and Muon on query-key contributes little. | Proposed |