Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Concept geometry — how concepts are laid out in a model’s representation space — linear directions and the regions, lattices and spectra built from them, superposition and the dictionaries that pull it apart, reading concepts out with probes and writing them in with steering vectors, and whether different models arrive at the same geometry.
14 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-458 v2 | The Platonic Representation Hypothesis | Huh et al. (2024), ARXIV-2405.07987. Representations are converging: across architectures, objectives and modalities, and more so as models get larger. The conjecture is that they are converging on a statistical model of whatever generated the data. The mathematical argument holds only for bijective observations, which the authors say plainly and test at the boundary. | Active |
| LIT-460 v2 | The Lattice Representation Hypothesis of Large Language Models | Xiong (2026), ARXIV-2603.01227. If concepts are linear directions with thresholds, each concept is a half-space, and intersecting half-spaces gives a concept lattice — so conceptual inclusion, intersection and union become geometric meet and join. Tested on WordNet sub-hierarchies. The Linear Representation Hypothesis gives binary concepts; this is the set-theoretic structure over them. | Active |
| LIT-526 | Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations | Acharya, Rimal and Dhakal (2026), ARXIV-2605.01609 — contextual concept directions sit in the low-variance tail of the unembedding covariance in 17 of 17 models, while static vocabulary contrasts sit at the top. The paper also reports a robust null against the hypothesis it set out to test, and labels each of its own claims robust, partial or suggestive. Read as NOTE-270. | Active |
| LIT-544 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei and Murfet (2023), ARXIV-2310.06301 — the paper that separates two things everyone calls a phase transition. In the Toy Model of Superposition at two hidden dimensions, regular k-gons are critical points with derivable local learning coefficients; the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α] then predicts a Bayesian 5 → 6 transition at n_cr ≈ 600, observed at 600 ≤ n ≤ 700. The same critical points govern SGD plateaus — and the 5 → 6 transition is not one of the dynamical ones. Read as NOTE-288. | Active |
| LIT-568 | Are Sparse Autoencoders Useful? A Case Study in Sparse Probing | Kantamneni, Engels et al. (2025), ARXIV-2502.16681. On 113 probing datasets over Gemma-2-9B, and replicated on Llama-3.1-8B, logistic-regression probes are compared with probes on SAE latents. Selection is done the way a practitioner would, by validation AUC over a “quiver” of methods. Adding SAE probes does not help in standard conditions (−0.003 AUC) or under data scarcity, class imbalance, label noise or covariate shift. Earlier positive results for SAE probes disappear against stronger baselines, including some of the authors’ own. | Active |
| LIT-571 | Scaling and evaluating sparse autoencoders | Gao et al. (2024), ARXIV-2406.04093. TopK sparse autoencoders keep the k largest latents, which sets L0 directly and replaces the L1 penalty. They beat ReLU SAEs on the sparsity-reconstruction frontier, and the gap widens with scale. Encoder-as-decoder-transpose initialization plus an auxiliary dead-latent loss leave 7% dead latents in a 16M-latent SAE on GPT-4, against up to 90% without them. Clean scaling laws in latents and k. That SAE scores 98.2% “loss recovered”, yet substituted into GPT-4 it matches a model trained on 10% of GPT-4’s compute. | Active |
| LIT-602 v2 | GloVe: Global Vectors for Word Representation | Pennington, Socher and Manning (EMNLP 2014), DOI 10.3115/v1/D14-1162. Meaning shows in ratios of co-occurrence probabilities, not in the probabilities themselves, so word vectors should make differences encode log ratios. That leads to a weighted least-squares fit of wᵢ·w̃ⱼ + bᵢ + b̃ⱼ to log Xᵢⱼ over nonzero counts, with a weight capped at x_max = 100 and exponent 3/4. It scores 75.0% on the word analogy test with 42B tokens, and 71.7% against word2vec skip-gram’s 69.1% on the same 6B tokens. The word2vec side of that comparison ran at default settings. | Active |
| LIT-603 v3 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov, Sutskever, Chen, Corrado and Dean (NIPS 2013), ARXIV-1310.4546. Makes skip-gram fast and better. Negative sampling replaces the softmax with logistic regression against k noise words drawn from the unigram distribution raised to the 3/4 power. Frequent words are subsampled, with discard probability 1 − √(t/f) at t ≈ 10⁻⁵, which runs 2–10× faster and helps rare words. High-PMI bigrams become single tokens. Vector sums compose meanings, for example Russian + river ≈ Volga River. | Active |
| LIT-604 | Efficient Estimation of Word Representations in Vector Space | Mikolov, Chen, Corrado and Dean (2013), ARXIV-1301.3781. Two log-linear architectures drop the hidden layer of a neural language model. CBOW predicts a word from its averaged context, and skip-gram predicts the context from the word. The paper also introduces the Semantic-Syntactic Word Relationship test (8,869 semantic and 10,675 syntactic analogy questions), answered by vector arithmetic. On it, skip-gram at 300 dimensions reaches 53.3% on 783M words in about 3 days on one CPU, far ahead of earlier public embeddings. The comparisons are not controlled for data size or dimension. | Active |
| LIT-606 | The Linear Representation Hypothesis and the Geometry of Large Language Models | Park, Choe and Veitch (ICML 2024), ARXIV-2311.03658. “Concepts are directions” has three readings: a shared difference between word pairs, a linear probe, and a steering vector. The paper formalizes the first with counterfactual pairs in the unembedding and embedding spaces, and proves the first is a probe and the second a steering vector. Training identifies the representation only up to an invertible linear map, so Euclidean cosine has no special meaning. An inner product under which causally separable concepts are orthogonal unifies the two spaces, and one such product is the inverse covariance of the unembedding rows. Evidence is 27 concepts on LLaMA-2 7B, with a Gemma-2B comparison in the appendix. | Active |
| LIT-613 | A Latent Variable Model Approach to PMI-based Word Embeddings | Arora, Li, Liang, Ma and Risteski (TACL 2016). A generative model of text: a discourse vector drifts slowly on a random walk, and each word is emitted with probability ∝ exp(⟨c, v_w⟩). If word vectors are isotropic, the partition function is nearly constant, and integrating out the discourse gives log p(w,w’) ≈ ‖v_w + v_w’‖²/2d − 2 log Z, so PMI ≈ ⟨v_w, v_w’⟩/d in low dimensions. Isotropy also explains why analogies survive large fitting error: solving for v_a − v_b is a regression that denoises by √(d/n). The squared norms of the fitted vectors correlate 0.75 with log frequency, and the partition function concentrates within about ±10%. | Active |
| LIT-696 v2 | Rarely categorical, highly separable representations along the cortical hierarchy | Posani et al. In 14,000+ IBL Neuropixels units (4,617 passing a ΔR² ≥ 0.015 selectivity threshold), the clustering of 8-D encoding-model selectivity vectors beats a covariance-matched Gaussian null in only a handful of the ~20 testable cortical regions (VISp, AUDp, SSp-ul at Bonferroni p < 0.05; clustering z falls with hierarchy position, ρ = −0.62). Pooled modules and the whole cortex do cluster (whole-cortex z = 8.0). Among the 16 regions with enough trials, once the 16 task conditions are merged into pairwise-decodable “independent” ones (M_IC from 5 to 16, rising with hierarchy, ρ = 0.77), ≥ 95% of random balanced dichotomies are linearly decodable above a shuffle null in every region except gustatory cortex (≈0.82). | Active — read in full 2026-09-25 (NOTE-353); a careful, well-controlled, openly reviewed large-scale study worth reading for its method (Gaussian-null categoricality test, independent-conditions merging, dichotomy separability) and for its explicit warning that above-chance linear decodability of one variable is weak evidence in a high-dimensional code. The headline “all regions maximally separable” rests on a lenient above-chance criterion and on 16 regions, not 43. |
| LIT-698 | Topos and Stacks of Deep Neural Networks | Belfiore et al. (2021), ARXIV-2106.14587. The authors claim that every DNN architecture defines a Grothendieck topos of sheaves over a site built from its graph, that backpropagation is a flow of morphisms in that topos, and that layer invariances (CNN translations, LSTM structure) are stacks, and they conjecture that these stacks explain generalization and support a homological notion of “semantic information”. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-759 | From monoliths to modules: Decomposing transducers for efficient world modelling | Boyd et al. (2025), ARXIV-2512.02193 — a formal paper with no experiments. A world model is a transducer (a stochastic input-output machine with latent memory, generalizing POMDPs), and composing transducers is a Kronecker product of their operators. The paper inverts this. Two information measures, Intransducibility (with latents) and Acausality (observables only), are zero exactly when one block of variables can be a causal module driven by the rest. Peeling such modules off factors a monolithic model into a DAG of sub-transducers that can be learned and inferred separately. Minimal predictive models (ε-transducers) are closed under composition. | Active |