Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Representation and encoding — how the signal is encoded before the expensive network sees it — tokenizers and learned latents, positional encoding, and the frequency or basis choices that go with them.
20 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-021 | A model builds a latent vocabulary in its early layers whose units are not the tokenizer’s | Kaplan et al. (2024), LIT-412, with LIT-409 and LIT-417 — sub-word sequences are recombined into whole-unit representations at a unit’s last token, in early and middle layers, and the units include things the tokenizer has no entry for: arbitrary splits, typos, out-of-vocabulary words, named entities and non-compositional multi-word expressions. Three groups, three methods — a probe with a co-occurrence control, an erasure signature, and layer ablation — reaching the same place. The tokenizer’s vocabulary is the model’s input format, not its inventory of units. | Active |
| THEORY-022 v2 | A phrase placed in a word slot is read as a lemma, and the reading comes from the construction rather than the phrase’s frequency | Goldberg and Shirtz (2025), LIT-410 — English has a productive construction in which a unit with the internal syntax of a phrase, up to a whole sentence, occupies a slot reserved for a word, and comprehenders give it the kind of meaning a word has: it evokes a semantic frame presumed to be shared. Four preregistered surveys measure the reading, and the fourth removes the obvious deflation — the effect is the same size on high-frequency phrases, and estimated corpus frequency predicts nothing. So the lexical unit is delimited by construction, not by how often the string has been seen, which is the assumption a frequency-merge tokenizer makes. | Active |
| THEORY-034 v2 | Concepts sit in embedding space as intersections of half-spaces, so inclusion, intersection and union are geometric operations | Xiong (2026), LIT-460 — a linear attribute direction plus a separating threshold is a half-space, a concept is the intersection of its attributes’ half-spaces, and the resulting regions form a complete lattice. Conceptual refinement is geometric meet, generalisation is join. Evidence is WordNet, which is a hand-built hierarchy and so the friendliest test there is. | Proposed |
| THEORY-036 | Representations converge across architectures, objectives and modalities, and the endpoint is a model of what generated the data | Huh et al. (2024), LIT-458 — vision and language models measure distance between datapoints increasingly alike as they scale, across architectures and objectives. The conjecture is that they are converging on a representation of the joint distribution that generated the observations. The convergence is measured; the endpoint is proved only for a world of bijective observations. | Proposed |
| THEORY-052 | Synthesis requires equivariant representations and discriminative pretraining rewards invariant ones, which is why a generative prior groups objects it was never shown | Khangaonkar and Pirsiavash (2025), LIT-488 — self-distillation and contrastive objectives explicitly reward representations that do not change under augmentation; instance segmentation needs representations that do. The paper’s DINO control activates on objects and cannot separate their instances, which is the shape this account predicts. | Proposed |
| THEORY-058 | Apparent compression in the information plane is saturating activations collapsing into extreme bins, not information being discarded | Saxe et al. (2018), LIT-509 — a tanh unit must grow its weights to compute anything nonlinear, and as it does its activity piles into the saturation regions. Under a fixed binning that is a distribution collapsing into two bins — about 1 bit — which the information plane draws as a compression phase. Proposed, because LIT-507 shows compression in some non-saturating networks once the bins are placed adaptively. | Proposed |
| THEORY-063 | Contextual concept directions sit in the low-variance tail of the unembedding spectrum while vocabulary contrasts sit at the top | Acharya, Rimal and Dhakal (2026), LIT-526 — concept directions read out of the residual stream anti-concentrate in the low-eigenvalue directions of the unembedding second moment, in 17 of 17 models and by three independent extraction methods. Static unembedding-row contrasts do the opposite. So the same model carries semantic content at both ends of one spectrum, depending on whether the representation is contextual. | Proposed |
| THEORY-077 | A coordinate MLP learns high frequencies slowly because its tangent kernel’s spectrum falls off fast, and a sinusoidal input mapping works by making that kernel stationary with a bandwidth you choose | Tancik et al. (2020), LIT-550 — in the NTK regime, error along each kernel eigenvector decays at a rate set by its eigenvalue, and a coordinate MLP’s eigenvalues fall off fast with frequency. Sinusoids of the input give a stationary composed kernel whose bandwidth the frequencies set. The linear model predicts trained networks’ loss curves. It is a kernel-regime account of an input encoding, not of feature learning. | Active |
| THEORY-085 | InfoNCE maximizes a lower bound on mutual information that cannot exceed log N | Active | |
| THEORY-087 v2 | Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other | Deferred | |
| THEORY-088 v3 | Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal’s information density | Proposed | |
| THEORY-089 v3 | Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition | Pennington et al. (2014), LIT-602, and Mikolov et al. (2013), LIT-603 — what distinguishes words is the ratio of how often they co-occur with probe words (ice and steam with solid versus gas). A model whose dot products fit log co-occurrence turns ratios into vector differences, so relations become offsets. The same log-linearity makes a sum of vectors act like a product of context distributions, an AND of their contexts. Derived from the objectives and illustrated. The mechanism is not measured. | Proposed |
| THEORY-092 v2 | Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores | Levy, Goldberg and Dagan (2015), LIT-607 — PMI divides by the context’s probability, so a rare context seen once with a word gets a high score and crowds out the informative ones. Raising context counts to α = 0.75 before normalizing raises rare contexts’ probability and lowers their PMI. That is why word2vec’s smoothed negative distribution helps, and why the same smoothing helps PPMI most (up to 9.2 points) and SGNS least (at most 1.4). | Proposed |
| THEORY-093 | Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily | Levy and Goldberg (2014), LIT-612 — setting the derivative of SGNS’s per-pair objective to zero gives w·c = PMI(w, c) − log k, with k the number of negatives. So word2vec’s skip-gram is a factorization of the matrix count-based methods use, shifted by a constant, with each pair’s error weighted by how often it occurs. NCE gives log P(w|c) − log k the same way. Exact at the optimum with unconstrained dimensions. At practical dimensions it describes the target, not the result. | Active |
| THEORY-094 | Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors | Arora et al. (2016), LIT-613 — if each word is emitted with probability ∝ exp(⟨c, v_w⟩) as a discourse vector c drifts, and word vectors are isotropic, the partition function is nearly constant and PMI(w,w’) ≈ ⟨v_w, v_w’⟩/d holds in low dimensions. That carries THEORY-093 from its unconstrained optimum to practical sizes. Isotropy also makes solving for a relation’s offset a well-conditioned regression that averages away per-word error, which is the step THEORY-089 skipped. The assumptions are checked only loosely. | Proposed |
| THEORY-101 | A large trained ViT repurposes redundant patch tokens as scratch space for global information, which is what its feature-map artifacts are | Darcet et al. (2023), LIT-662. The high-norm tokens that spoil ViT feature maps are not damage: they sit on patches redundant with their neighbours, they have discarded their own local content, and they carry global information instead. The model found spare capacity and used it. The account’s own prediction — give the model dedicated slots and the artifacts should move there — was tested and holds. | Active |
| THEORY-102 | A causal mask implicitly encodes absolute position, because a token can count how many predecessors it may attend to | Haviv et al. (2022), LIT-665. A decoder-only transformer given no positional encoding recovers absolute position within four layers, and the reason is the mask: a token that can count its attendable predecessors knows its index. The account’s prediction is that removing the mask removes the mechanism, and a bidirectional MLM without an encoding does not merely degrade — it fails, at 147.18 perplexity against 4.00. | Active |
| THEORY-113 v4 | A gradient attribution survives weight randomization because of its input multiplier, not because the attribution is insensitive to the weights | Kokhlikyan et al. (2021), LIT-724. Integrated Gradients and its relatives multiply a path integral by (x − x₀), which does not depend on the model. That factor carries the input’s structure into the map, so the map keeps looking the same when the weights are destroyed. LIT-713 suspected this and said it did not measure it; this does, with the arm that discriminates — drop the multiplier and the map becomes parameter-sensitive. A second, harder arm comes free: on text, where summing over embedding dimensions destroys the multiplier’s structure, the global variant becomes parameter-sensitive too. | Proposed |
| THEORY-114 v2 | A modified-backpropagation rule that keeps only non-negative relevance produces a matrix chain that converges to rank 1, so the later layers can only flip the map’s sign | Sixt, Granz and Landgraf (2019), LIT-729, Theorem 1. The z⁺-rule backpropagates relevance through a product of non-negative matrices; such a product converges to a rank-1 matrix C = c γᵀ, and then C v = λ c for every v. So the relevance vector set at the output — the thing that carries which class is being explained — survives only as a scalar, and can at most flip the map’s sign. Class-insensitivity and independence of the later layers’ weights are the same fact. Active because it is a theorem with a discriminating arm that was published (DeepLIFT does not converge) and a second that was built to order (the DeepLIFT ablation does). | Active |
| THEORY-115 | Guided backpropagation and DeconvNet do partial image recovery, not attribution: the backward ReLU plus a CNN’s local connections reconstruct the input regardless of the class or the weights | Nie, Zhang and Patel (2018), LIT-730, Theorems 1 and 2. In a random three-layer CNN, s_k^GBP(x) ≈ x — guided backpropagation recovers the input, with untrained weights and regardless of the class — while the saliency map and DeconvNet are N(0, I). The causes are the backward ReLU and the small filters that local connectivity implies. So these maps are “unrelated to the decision-making of neural networks”, which is why they survive weight randomization, why they look the same for every class, and why an untrained edge detector matches them. | Active |