Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Representation learning — how learned systems come to represent their data — self-supervised and contrastive objectives, spectral embeddings, information bottlenecks, knowledge-graph embeddings, and whether representations converge.
208 of 1352 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-001 v4 | On a finite augmentation space, InfoNCE, logistic and spectral contrastive losses share one population optimum: the positive-pair density ratio | Johnson, El Hanchi & Maddison (2023), LIT-258 — over unconstrained functions on a finite augmentation space, the population minimizer of the spectral contrastive and NT-Logistic losses is K⁺ = p⁺(a₁,a₂)/(p(a₁)p(a₂)), and a minimizer of NT-Xent/InfoNCE over symmetric functions, with negatives i.i.d. from the marginal, is K⁺ up to a factor constant on communicating classes; that form is taken from van den Oord et al., not derived. It is a statement about the unconstrained optimum, not about what a network reaches. | Proposed |
| THEORY-002 v7 | Convergence of representations, in the Platonic hypothesis’s sense, is convergence of kernels, which fixes representations only up to the symmetry group of what is observed; whether closeness in distribution implies closeness of representation depends on the distance used | An inference joining Huh et al.’s Platonic hypothesis (ANTH-LIT-458) to Nielsen et al. (LIT-253) and Harvey, Larsen & Williams (LIT-250). It does not refute the hypothesis; it says what the hypothesis can and cannot mean. | Proposed |
| THEORY-003 v3 | Maximizing HSIC between representations and image identity maximizes the average squared MMD between the images’ view distributions, whose kernel mean embeddings exist by the Riesz representation theorem | Li, Pogodin, Sutherland & Gretton (2021), LIT-248, eq. 5 and App. B.2; Muandet et al. (2017), LIT-260, Lemma 3.1 — the one place in this family where Riesz is invoked explicitly. InfoNCE is not HSIC: Li et al.’s eq. 28 decomposes it exactly as −HSIC plus a log-mean-exp term, and Theorem B.1 bounds it with no variance assumption. | Active |
| THEORY-004 v8 | A representation is determined by its centred kernel up to a translation and an orthogonal transformation, and the rotation-aligned distance between two representations equals the Bures distance between their kernels | Harvey, Larsen & Williams (2023), LIT-250, Thm 1; Jorgensen & Tian, LIT-259, Remark 1.34 and Cor. 1.35, which name the kernel-to-Hilbert-space construction as GNS and state its minimal realizations unique up to a unitary (proof left as an exercise). This is the GNS connection the owner asked about, stated. | Active |
| THEORY-005 v4 | The positive-pair density ratio is the kernel of the conditional-expectation operator on L²(p), so spectral representations are that operator’s eigenfunctions, well defined when the positive-pair χ²-divergence is finite | An inference assembled on 2026-09-26 from Johnson, El Hanchi & Maddison (LIT-258), Tosh, Krishnamurthy & Hsu (LIT-251) and HaoChen et al. (LIT-249), with a two-line derivation of the record’s own. Johnson et al. write the operator formulation on a finite space (App. E.2); no source states it with its integrability condition. | Proposed |
| THEORY-006 v4 | InfoNCE is a lower bound on mutual information for every critic and can never exceed the log of the batch size | Poole et al. (2019), LIT-246, §2.3, Eqs. 8–10 — an exact proof (CPC’s own derivation, ANTH-LIT-589, passes through an approximation). Tschannen et al. (LIT-247) show that tightening the bound does not explain what makes the representations good. | Active |
| THEORY-007 v4 | Kernel PCA under the positive-pair density ratio recovers the eigenfunctions of the positive-pair Markov chain, and their top span is minimax-optimal for linear prediction of approximately view-invariant targets | Johnson, El Hanchi & Maddison (2023), LIT-258, Thm 3.2 and Thm 4.1 — proved for finite augmentation spaces at the population level, Thm 4.1 for d at least the number of communicating classes. HaoChen et al.’s spectral loss is the same factorization of the normalized positive-pair adjacency. Whether trained networks land on these eigenfunctions is not shown. | Active |
| THEORY-008 v5 | On a finite set of stimuli, what a regularized linear readout can decode from a representation is a function of its normalized kernel, and CKA, CCA and GULP are averages of readout agreement | Harvey, Lipshutz & Williams (2024), LIT-257, eqs. 2, 7–8, Prop. 2, Cor. 1 and Table 1 — proved identities. This is the finite-dimensional content of “a probe is a functional”: the probe’s output is fixed by the kernel, whatever orthonormal basis the representation is written in, for G = aC_X + bI; under any linear change of basis only for the CCA member, G = C_X. | Active |
| THEORY-009 v7 | In the positive-pair spectral self-supervised papers, self-adjointness comes from the symmetry of the positive-pair distribution, not from the Riesz representation theorem; elsewhere in the family it comes from a symmetrized loss, a precondition on the kernel, or composing the conditional-expectation operator with its adjoint | An inference from how the spectral SSL papers (LIT-258, LIT-249, LIT-252, LIT-242) obtain their eigenbases: each uses a symmetric pair matrix or kernel, and none invokes Riesz. In all but Simon et al., whose loss is symmetrized by hand, and Pfau et al., whose SpIN takes symmetry as a precondition on the kernel k, that symmetry is the symmetry of the positive-pair distribution. Answers the owner’s “do we even need Riesz?” for this family of results. | Active |
| THEORY-017 v6 | In a Hilbert-space model only unitarily invariant structure is intrinsic; a basis or tensor factorisation, and so any parts or features, is extra data supplied from outside the space, and the record’s Hilbert-space works differ in where they get it | Carroll (2021), LIT-123: a quantum theory is fixed up to unitary equivalence by the Hamiltonian’s spectrum, and subsystems come from a factorisation that the dynamics must select. Van Rijsbergen (LIT-262) computes only traces and inner products, and treats each observable’s eigenbasis as a “point of view”. DisCoCat’s ε/η composition (LIT-273) is orthogonally covariant, while its Frobenius sequel (LIT-272) stipulates a basis and composes by elementwise product in it. THEORY-004 is the same point for learned representations. This is a synthesis across the four, not a result any one of them states. It does not say that basis-dependent models are wrong; it says what they have assumed. | Active |
| THEORY-018 v4 | Softmax training identifies a language model’s final-layer representation only up to an invertible linear map, so no inner product on it is intrinsic, and Park, Choe and Veitch’s causal inner product is fixed by a stipulation | Park, Choe & Veitch (2023), LIT-304, prove the invariance γ ↦ Aγ + β, λ ↦ A⁻ᵀλ (eq. 3.1). They choose M = Cov(γ)⁻¹ by setting a free diagonal D = I, and admit they have no principle for choosing D. NOTE-302 derives, and the filer re-checked, that Cov(γ) alone is compatible with every positive-definite M. So the paper’s remark that most inner products are ruled out holds only once the true concept directions are known. Under superposition (LIT-323) no inner product can make all features orthogonal. | Active |
| THEORY-019 v4 | Symmetry forces spectral degeneracy but degeneracy does not identify a symmetry: an operator commuting with a group has eigenspaces built from its real irreducibles, so abelian rotation groups force pairs over ℝ, and the spectrum fixes neither the group nor a basis inside a multiplet | Peter & Weyl (1927), LIT-329 §3, build representations as eigenspaces of an invariant kernel, and Kondor & Trivedi, LIT-305 Lemma 6, show that intertwiners preserve isotypic components. Neither states the degeneracy corollary or its converse. This document puts that together from several readings, and that assembly, with its proof, is now the record’s own claim, CLAIM-385; what stays here is the reading. It adds the real-scalar correction in NOTE-282, found in the reading of LIT-362, and the counting in the reading of LIT-346. It does not support “degeneracy detects symmetry”: a spectrum is consistent with several groups and says nothing about the characters inside a multiplet. | Active |
| THEORY-021 v2 | In the exact configurations of uniform superposition (regular polygons, the tetrahedron, antipodal pairs, and tegum products of these that span the hidden space, with equal norms within each factor) the features form a rank-one POVM compressed from the n-feature basis: their dimensionality-weighted projectors sum to the identity | Elhage et al. (2022), LIT-323, found that features in uniform superposition settle into polytopes and tegum products, with feature dimensionalities clustering at 3/4, 2/3, 1/2, 2/5 and 3/8. The tight-frame and POVM reading is the reader’s inference in NOTE-278, checked numerically there. It was refined when filing: a tegum product is not globally tight, but the Dᵢ-weighted POVM still sums to the identity. None of this is in the paper. It does not make superposition contextual or quantum, and it says nothing yet about trained real models. | Active |
| THEORY-022 v4 | The reverse-engineered grokking network computes modular addition by multiplying characters of ℤ/p at a few frequencies, the same real two-dimensional irreducibles later found as circles in language models; in both the group comes from the task, not the network | Nanda et al. (2023), LIT-345, show that a one-layer transformer trained on addition mod 113 embeds its inputs at five key frequencies. It forms cos/sin(w_k(a+b)) and reads out Σ_k α_k cos(w_k(a+b−c)). The reading in terms of characters and irreducibles is NOTE-283’s, joined here to Engels et al.’s circles (LIT-322). It holds for one weight-decay model and its variants: dropout-trained generalisers are not Fourier-sparse. It is not evidence that a network discovered the group, which was given, or that the frequencies could be read off a spectrum. | Proposed |
| THEORY-035 v5 | Deep networks generalise because the diffusion phase of SGD compresses each layer’s information about the input | Shwartz-Ziv & Tishby (2017), LIT-324, report a fitting phase followed by a long compression phase in the information plane, and read compression by gradient diffusion as the mechanism of generalisation. Their §3.4 reports that with 5% of the data compression lowers I(T;Y). Saxe et al. (2018), LIT-372, rerun their code and find the compression phase only in double-saturating networks measured through an imposed binning, find it under full-batch GD as under SGD, and find compression and generalisation in all four combinations. The account is rejected. The IB functional of LIT-338, the tanh two-phase observation and the gradient-SNR transition are not. | Rejected — Saxe et al. show the compression phase appears only for double-saturating units under an imposed binning or noise model, occurs under full-batch GD as under SGD, and dissociates from generalisation in all four combinations; the paper’s own small-sample run already showed compression losing label information |
| THEORY-039 v2 | The later phases reported in neural-network training differ in kind: after the transition a network may simplify its weights, acquire item-specific information, lose generalisation, confine its tangent kernel or saturate its units, and none of these is a measured compression of I(X;T) | Nanda et al. (2023), LIT-345, Zucchet et al. (2025), LIT-371, Saxe et al. (2018), LIT-372, Prakash & Martin (2025), LIT-369, and Zhou et al. (2025), LIT-370, each report training as a sequence of phases. Their phases are defined by different quantities and fall at time scales from 10² to 10⁷ steps. What follows the transition differs in kind from work to work. Two of them, the two with a mechanistic intervention, show a circuit forming while the loss is flat. Only one measures I(X;T), and there the later fall belongs to the estimator. This is the record’s synthesis across the five, not a claim any of them makes. | Proposed |
| THEORY-078 v2 | The top C eigenvalues of a trained C-class classifier’s Fisher are carried by the second moment of its class-mean logit derivatives, so a cut at the bulk edge keeps the span of the class means, and they stand clear of the bulk to the extent between-class separation exceeds within-class variation | Papyan (2019), LIT-619: for softmax cross-entropy the Gauss–Newton matrix G, which is the model’s Fisher averaged over inputs, is an uncentred second moment of logit derivatives indexed by class and logit. It decomposes exactly into four terms (Eq. 28), and its top C eigenvalues match those of the class-mean term G₁. Papyan (2020), LIT-613, splits those C into one global-mean outlier and C − 1 between-class ones, finds a C(C − 1) cross-class mini-bulk beneath them, and proves the whole pattern for a Gaussian toy model. Papyan (2018), LIT-617, supplies the measurements and the attribution of the Hessian’s outliers to G. The decomposition is algebra. That G₁ carries the outliers is measured, by eye, under a training protocol narrower than practice. So this is Proposed. | Proposed |
| THEORY-083 v3 | In Papyan, Han and Donoho’s runs of VGG, ResNet and DenseNet trained with cross-entropy on seven balanced datasets without augmentation, training past zero error shrinks last-layer within-class variability against between-class variability, and moves the centred class means and the classifier toward a common simplex equiangular tight frame and the decision toward nearest class mean, nearly to completion on the four easier datasets and only partly on ImageNet | Papyan, Han & Donoho (2020), LIT-618, measured 480 trained classifiers on seven datasets with VGG, ResNet and DenseNet. Through the terminal phase, within-class variability of last-layer features falls (NC1), and the centred class means become equinorm and equiangular with cosine −1/(C − 1) (NC2). The classifier aligns with them (NC3), and the decision becomes nearest class mean (NC4). They prove that NC1 and NC2 imply NC3 and NC4 for the MSE-optimal and max-margin classifiers, and that the simplex ETF is the unique error-exponent-optimal codebook. Read from the plots, collapse is close to complete on the four easier datasets and partial on ImageNet at the last epoch. Papyan (2020), LIT-613, sees class means separate and grow more orthogonal with depth. The measurements cover one protocol, and the dynamics are unexplained outside free-features models. Within them, THEORY-278 (LIT-974) derives collapse for MSE under a re-whitened flow, and THEORY-286 (LIT-994) gives the cross-entropy endpoint under class imbalance. The paper’s further claim, that the Hessian’s class outliers are a symptom of collapse, is argued and not measured, and is left to THEORY-078. | Active |
| THEORY-086 v2 | In the kernel regime gradient descent fits the target eigenspace by eigenspace of the neural tangent kernel, at rates set by their eigenvalues; on uniform spherical data the eigenspaces are the harmonic degrees, and for fully connected ReLU networks the non-zero eigenvalues decay as k^(−d) at every depth | Three derived results that compose. Cao et al. (2019), LIT-624, Theorem 4.2: for a wide two-layer ReLU network, the training residual’s component in the top-r_k NTK eigenspace falls as (1 − λ_{r_k})^T, for any target. Basri et al. (2019), LIT-612: on uniform spherical data the Gram matrix is a convolution, so its eigenvectors are harmonics, with closed-form eigenvalues per frequency. The bias-free kernel is zero on odd frequencies k ≥ 3, and a bias restores them. Bietti & Bach (2021), LIT-608, Corollary 3: the L-layer ReLU NTK decays as k^(−d) on S^(d−1), the same exponent at every depth. Active for the kernel regime on uniform spherical data, where every step is proved. It says nothing about networks whose kernel moves. | Active |
| THEORY-104 v2 | Under distribution shift the best layer for a linear readout lies below the top: across the last layers of a trained network, in-distribution accuracy rises while out-of-distribution accuracy falls | Uselis & Oh (2025), LIT-678: linear probes on frozen intermediate layers of image classifiers beat last-layer retraining under shift, with zero-shot worst-group accuracy 79.4 → 87.1 (Waterbirds) and 56.0 → 82.0 (CelebA), layer 7 of 8 beating layer 8 on every subpopulation dataset, while in-distribution accuracy is best at the last layer. Chang, Deng & Chen (2025), LIT-657: in GPT-2 Small on a synthetic task, early-exit out-of-distribution accuracy peaks at layer 10 (56.90%) and falls to 50.00% at layer 12 while in-distribution accuracy climbs to 93.17%. The layer is chosen on out-of-distribution data throughout; ViT gains are small; the language-model evidence is one synthetic task; and Chang et al.’s information estimate is questioned in its NOTE. | Proposed |
| THEORY-111 v3 | In image classifiers, between two networks linearly connected by spawning from a shared checkpoint or by permutation alignment (MLPs, and VGG-16, ResNet-20 and ResNet-50 on CIFAR-10 and Tiny-ImageNet), nearly every layer’s features at three points on the line were proportional to the interpolation of the endpoints’ features, and pairs that were not linearly connected were far from it | Zhou et al. (2023), LIT-674: whenever two image classifiers are linearly mode connected, by spawning or by permutation, the features of nearly every layer of the interpolated network are proportional to the interpolation of the endpoints’ features. This is layerwise linear feature connectivity, and two conditions, weak additivity of ReLU and a commutativity of weights with features, imply it exactly. Lubana et al.’s Lemma 2, LIT-669, gives the exact one-hidden-layer case: linear connectivity between interpolating minima forces shared activation patterns, and with shared patterns the hidden features are linear along the path. Measured on pairs at three interpolation points, image classification only. The regularity outside image classification, and the step to many-model averages such as soups and SWA, are QUESTION-318. | Active |
| THEORY-113 v2 | Merging ResNets trained from different initialisations on disjoint halves of a label set by permuting one onto the other and averaging fell far short of the ensemble; their cross-model feature correlation fell with depth, merging only the early layers recovered most of the ensemble’s accuracy, and merging features within each model as well as across both helped at 4× width and above; Stoica et al. attribute the failure to features with no counterpart in the other model | Stoica et al. (2023), LIT-665, merge ResNets trained from different initialisations on disjoint label sets. Permute-then-average fails, which they explain by the permuted model lying in a basin for its own task only. Mean cross-model feature correlation falls with depth (0.50, 0.37, 0.27 at three depths). Merging only the early layers and keeping later layers as task heads recovers most of the ensemble’s accuracy. Merging features within one model as well as across both helps once the models are wide enough to hold redundant features. With full merging on ImageNet-1k, ZipIt! is no better than permutation (8.6%). No same-task control separates the effect of task difference from that of depth; that is QUESTION-316. | Active |
| THEORY-116 v2 | A linear barrier between two minima that survives the removal of symmetries marks a difference in the input attributes the two models rely on: they compute by different mechanisms | Lubana et al. (2022), LIT-669: models that rely on a synthetic cue and models that ignore it are separated by a linear barrier even after permutation matching. Fine-tuned models that keep a linear path to their parent keep its cue, and those that lose the path lose the cue. Their Conjecture 1 states this claim’s direction. What their Appendix F.3 proves, for a one-hidden-layer ReLU network with interpolating minima, is the converse: linear connectivity forces identical activation patterns, so models relying on attributes of different complexity cannot be linearly connected under any permutation. Zhou et al. (2023), LIT-674, support that converse in deep image classifiers, where linearly connected pairs have features that blend layer by layer. The stated direction rests on experiments with synthetic cues and on a one-layer sufficiency argument. | Proposed |
| THEORY-117 v3 | In Uselis, Dittadi and Oh’s two-concept grids, ResNet-50s trained from scratch, with checkpoints chosen on the unseen pairs, reach unseen combinations as the share of combinations seen grows and not as up to four times more data of the same combinations is added; and when embeddings are exactly linearly factored, seen pairs that connect every value of both concepts, as two per value arranged in one cycle do, determine the embedding of every unseen pair | Uselis, Dittadi & Oh (2025), LIT-667. In an n × n grid of two concepts with k combinations seen per value, ResNet-50s trained from scratch fit seen pairs and fail on unseen ones; four times the data of the same combinations leaves 60–80% drops, while more values and more combinations help. Features pass from spurious, to decodable but entangled, to linearly factored as coverage grows. Model selection was on the test set, by design, so the results bound what is possible. The record re-derived that exactly factored embeddings of every unseen pair are fixed by seen pairs connecting every value, as the paper’s k = 2 cycle does. Honest selection, other architectures and more concepts are QUESTION-207. | Active |
| THEORY-169 v2 | For unnormalized causal linear attention and for softmax attention, conditioning on a context is exactly training a learner on it and predicting: the first is one batch gradient step of a linear inner model from zero, the second kernel regression that stores the context, and one linear self-attention layer with constructed weights carries one gradient step of in-context linear regression; for these layers conditioning and per-sequence weight updates differ in what state is kept, not in kind | Sun et al. (2024), LIT-801, Theorems 1–2, with von Oswald et al. (2022), LIT-792, Proposition 1. Linear attention equals a layer whose state is a linear model’s weights, updated by batch gradient descent on the context from zero; softmax attention equals a Nadaraya–Watson learner whose “training” appends the context to a list; and, with constructed weights, one linear attention layer reproduces one gradient step of a linear regression on in-context pairs. These are identities of output for specific layers. They do not say that trained transformers do gradient descent (the anthology rejects that, ANTH-THEORY-068), nor anything about weight updates that persist across sequences. | Active |
| THEORY-175 v3 | Cosine similarities to matched anchor points are unchanged by exactly the maps that preserve each sample’s cosines to the anchors, among them every orthogonal map times a positive scalar and every per-sample positive rescaling but not translations; in Moschella et al.’s experiments independently trained encoders agreed in them only approximately, enough for a decoder reading them to transfer untrained between seeds, architectures and languages with loss, while their nearest neighbours mostly differed | Moschella et al. (2022; ICLR 2023), LIT-849, as read in NOTE-652. The invariance is by construction; its class (the maps preserving each sample’s cosines to the anchors, which include every orthogonal map times a positive scalar and every per-sample positive rescaling, but not translations) is the reader’s statement, re-derived in a dated note. The approximate agreement is the paper’s evidence: relative vectors match at cosine 0.86–0.97 across encoders, while ten-nearest-neighbour overlap is 0.10–0.39. It does not say the maps between trained spaces are isometries, which the source never measures (QUESTION-211), nor that the transfer needs no paired data: the anchors are paired data. | Active |
| THEORY-176 v2 | A task-loss stitching penalty compares representations in one decision problem, through a fixed top network and a restricted map class, so it is not Blackwell’s order: representations that are functions of each other can stitch differently, and a negative penalty shows the fitted map did not reproduce the layers it replaced | Bansal, Nakkiran & Barak (2021), LIT-850, define the stitching penalty L_ℓ(r; A) − L(A), with L_ℓ(r; A) the least task loss of A’s top layers fed r through a map from a simple family S, and use it to call one representation better than another. Read against Blackwell (1953), LIT-781, and THEORY-156, that ordering is one coordinate of the comparison of experiments with the decision rules cut down to A’s head composed with S. It is not invariant under Blackwell equivalence, which the paper intends, and it meets Blackwell’s order only where the fitted map reproduces the replaced layers exactly. The relations are this record’s derivations from the paper’s definitions, not the paper’s. | Active |
| THEORY-181 v3 | In a model that decodes the sum of two learned embeddings, at zero training loss with an injective decoder, a training set fixes the generalising representation of addition when its same-answer pairs leave only translation and scale free; for non-modular addition with p = 10 and one-dimensional embeddings, generalisation sets in near the training fraction, about 0.4, at which that becomes likely | Liu et al. (2022), LIT-858, in a toy built for it: same-answer training pairs force equal embedding sums, equal sums carry answers to unseen pairs, and for addition with p = 10 the fraction at which the forced equations first pin the embedding to a + kb (about 0.4) matches the onset of generalisation and Power et al.’s critical fraction (LIT-341). It is shown for one size, 1D embeddings and an operation built into the architecture. It is not shown that larger networks generalise by this mechanism, and for S₃ the networks generalise beyond it. | Active |
| THEORY-182 v2 | A rank-limited contrastive word embedding trained from small initialisation learns the top eigenvectors of the co-occurrence matrix’s relative deviation from independence, one at a time in order of eigenvalue, not the best low-rank approximation of its unconstrained optimum | Karkada, Simon, Bahri and DeWeese (2025), LIT-855: for the quartic approximation of word2vec with tied weights and symmetric, constant-weight reweighting, the minimisers are proved and the stepwise order is derived, and the prediction matches trained word2vec far better than truncated PMI. It is shown for a proxy of word2vec, not for word2vec, and the dynamics are an approximate derivation. | Proposed |
| THEORY-183 v2 | When the co-occurrence of words on a latent continuum depends only on their separation, a spectral word embedding places them on the Fourier modes of that continuum, so the circles and rippled curves of months, years and places come from a symmetry of the corpus statistics | Karkada, Korchinski, Nava, Wyart and Bahri (2026), LIT-860: proved for an embedding whose Gram matrix equals the co-occurrence matrix (or its absolute value) at full rank, with the exponential kernel solved for both boundary conditions; the symmetry measured in Wikipedia; matching PCA pictures in Gemma 2. That trained networks get their geometry this way is inferred, not tested: nothing in the paper changes the statistics and watches the geometry follow. | Proposed |
| THEORY-185 v4 | When each binary attribute of a word affects its co-occurrence independently and multiplicatively, the PMI is affine in the attributes with rank at most d + 1, so a spectral PMI embedding is a linear image of the attribute hypercube and parallelogram analogies hold exactly; over the full hypercube of words, each attribute’s two values equally frequent, the raw co-occurrence ratio mixes in products of attributes, and keeps exact analogies only when the strengths are alike enough that every attribute mode outranks every product mode | Korchinski, Karkada, Bahri and Wyart (2025), LIT-863: proved for a generative model of co-occurrence in which words are bundles of independent binary attributes, with robustness to noise, pruning and deleted pairs argued from eigenvalue bounds and simulated. The embeddings are eigendecompositions, not trained models; that real co-occurrence is close to this form is inferred from spectral and accuracy-curve shapes, not measured. | Active |
| THEORY-186 v4 | A deep linear network trained from small weights learns the singular modes of its input–output correlations one at a time in order of strength, each in a sharp sigmoidal transition, while a shallow network learns them all together; the stages come from the product of layers and the data’s spectrum, not from nonlinearity | Saxe, McClelland and Ganguli (2019), LIT-862, restating their 2014 solution (LIT-864): exact for decoupled, balanced initial weights and white inputs, simulated for random small weights. With hierarchically generated data the spectrum falls with depth in the tree, so the schedule is coarse to fine. It is not a claim about nonlinear networks, about correlated inputs, or about learning that starts from prior knowledge. | Proposed |
| THEORY-189 v2 | Nested angular cones in the Poincaré ball, symmetric about the outward ray with an aperture depending only on the apex’s radius, are transitive only if the aperture is at most π/2 and r sin ψ(r)/(1 − r²) does not increase outward; so no such cone can be defined near the origin, the widest admissible family is sin ψ = K(1 − r²)/r, and every point a cone contains lies farther from the origin than its apex | Ganea, Bécigneul and Hofmann (2018), LIT-866: Lemma 2 and Theorem 3 prove the necessity half under four stated requirements on the cones; the transitivity of the closed-form family (Theorem 4), which the source asserts without proof, is proved in this record’s note of 2026-10-19; the radial monotonicity is this record’s derivation from the hyperbolic law of cosines. It says what a cone order on the ball can be, not which posets embed in it, and nothing about meets, joins or learned embeddings. | Active |
| THEORY-194 v5 | On Monte Carlo samples of the 2D Ising model and of the 2D dimer model with added noise spins, a block coarse-graining trained to maximize mutual information with the system beyond a buffer recovered the variables renormalization already uses (Kadanoff block spins; the dimers’ low-momentum electric fields, with no weight on the noise), and iterated on 128 × 128 Ising samples gave the flow away from the critical point, locating it to about 1% and giving ν ≈ 1.0 ± 0.15, while a restricted Boltzmann machine trained by contrastive divergence on the noisy dimer data coupled to the noise and to columnar patterns instead | Koch-Janusz and Ringel (2017), LIT-873: shown numerically on two 2D lattice models whose relevant variables were known (Ising block spins, dimer electric fields), with an iterated Ising flow giving T_c to about 1% and ν ≈ 1.0 ± 0.15, and analytically only for 1D and quasi-1D chains. The general equivalence of the objective with a short-ranged effective Hamiltonian is argued, not proved, and no system with unknown relevant variables was tried. | Active |
| THEORY-195 v3 | On the Random Hierarchy Model, in the experiments reported (s = 2, L ≤ 4, small v), deep CNNs matched to the tree and trained in a feature-learning regime reach a tenth of chance error from about n_c m^L examples, polynomial in the input dimension, at the training-set size at which their hidden layers become invariant to swapping synonymous tuples; two-layer networks and the same CNNs as fixed kernels need a number proportional to the whole input space, exponential in the input dimension, and deep CNNs stay near chance when every tuple is made equally likely in every class | Cagnetta, Petrini, Tomasini, Favero and Wyart (2023), LIT-877: measured for depth-(L + 1) CNNs and deep fully connected networks on the Random Hierarchy Model with s = 2 and L ≤ 4, with a large-v scaling argument for the correlation threshold and an exact one-step-gradient model of a two-layer network on one patch. The law n_c m^L is fitted, not proved for trained deep networks, and in this model the correlation threshold and the grammar’s size are the same expression, so the data cannot say which of the two sets it. | Active |
| THEORY-196 v2 | A finite tree embeds in the hyperbolic plane with worst-case distortion 1 + ε, but because a hyperbolic distance d takes about d bits to write, its coordinates need Θ((ℓ/ε) log deg_max) bits: logarithmic in the branching, linear in the longest path ℓ, so hyperbolic space is economical for shallow bushy hierarchies and not for deep ones | Sala, De Sa, Gu and Ré (2018), LIT-874. The upper bound is Sarkar’s construction with the paper’s bit count. The lower bound is proved for H² and one family of trees, with an approximation, and dimension’s help is shown only as an upper bound. The cost is a cost of coordinates in a fixed-point model, not of the information in the tree, and it says nothing about Euclidean or linear-direction representations of the same hierarchy. | Proposed |
| THEORY-197 v4 | Mehta and Schwab’s correspondence between Kadanoff’s variational renormalization group and restricted Boltzmann machines is an identity of parametrizations, under which an exact RG step is a perfect fit of the data distribution; away from that point it does not make training by distribution fitting a renormalization | Mehta and Schwab (2014), LIT-882, Eqs. 18–22: setting the RG kernel T = −E + H makes the RG coarse Hamiltonian equal the RBM’s hidden-marginal Hamiltonian for every parameter value, and Kadanoff’s exactness condition equal to zero KL divergence. Short of exactness the paper itself says the two use different variational schemes, and Koch-Janusz and Ringel (LIT-873) give a case where distribution fitting keeps the wrong variables. The free-energy part of the argument was derived independently here; Lin, Tegmark and Rolnick’s v2 appendix makes it in substance (LIT-887). | Active |
| THEORY-200 v2 | On sparse hierarchical data, deep networks become insensitive to swapping synonymous parts and to small displacements of informative features at the training-set size at which they learn the task | Tomasini and Wyart (2024), LIT-880: measured for locally connected, convolutional and fully connected networks on the Sparse Random Hierarchy Model with s ≤ 3 and L ≤ 3, with thresholds for the two sensitivities tuned per setting; the sample size is about (s0 + 1)^L n_c m^L times s^(L/2) without weight sharing, with a one-step argument for the (s0 + 1)^L, and about (s0 + 1)^2 n_c m^L with it, unexplained. Both invariances keep the parent’s label statistics by construction, so the model shows that trained networks do the grouping, not that the invariances could not separate, and the extension to images is an analogy. | Proposed |
| THEORY-201 v2 | In next-token prediction on hierarchically generated sequences, the correlation between two tokens falls by about a factor m per level of their common ancestor, so a training set of P sequences resolves correlations only out to a distance t*(P) at which they meet sampling noise; learners come to represent the hidden symbols up to that depth, giving loss steps at about v m^(2ℓ−1) examples, and on character-level text the context at which the loss saturates grows as P^(1/(2β)) with β the measured correlation decay | Cagnetta and Wyart (2024), LIT-883, extending LIT-877 from labels to masked tokens: the correlation plateaus and the noise floor are derived for the Random Hierarchy Model in the large-v, large-m limit; the loss steps and the representation probes are measured in small transformers and CNNs; the link between them is argued, not proved; and the real-text evidence, from two character-level corpora, tests the effective context window, not the hierarchy. | Proposed |
| THEORY-202 v3 | In variational renormalization over general Boltzmann machines, neither a matched partition function nor Kadanoff’s exact trace condition makes a coarse-graining a renormalization: the first holds for joint Hamiltonians whose marginal is not the data, the second for hidden variables that do not interact with the system, so the variables a renormalization keeps must be fixed by a further stated condition | Lin and Tegmark (2016), LIT-887, arXiv v1–v2 appendix, a part dropped from the journal text: a family of joint Hamiltonians with Z_tot = Z and the wrong marginal, and the remark that any non-interacting pair meets the trace condition. Under Mehta and Schwab’s identification (LIT-882) the trace condition is a perfect fit of the data, so even an exact variational RG step leaves the coarse variables’ meaning free. The restricted Boltzmann machine proper needs visible–visible terms for the second construction; that narrows the example, not the claim. Promoted on 2026-10-20 with both constructions re-derived. Whether the trace condition suffices inside Kadanoff’s kernel families is QUESTION-205. | Active |
| THEORY-212 | In a two-layer linear network trained by gradient flow, the relative scale of the two layers, which gradient flow conserves, sets the learning regime: balanced layers learn in sigmoidal stages and both build task-specific representations, while strongly unbalanced layers learn exponentially with a static kernel and leave the larger layer task-agnostic | Dominé et al. (2025), LIT-943, read in NOTE-715. With whitened inputs and a λ-balanced start, W₂ᵀW₂ − W₁W₁ᵀ = λI, the full gradient flow is solved exactly. λ is conserved and sets the regime. As λ → 0, singular values are learned in sigmoidal stages, as in Saxe et al., and both layers become task-specific. As |λ| → ∞, learning is exponential, the NTK is static, and the larger layer stays near the identity while the smaller carries the task at scale ∝ 1/|λ|. The paper analyses gradient flow only. Under SGD the conserved quantity drifts (THEORY-211), which QUESTION-081 takes up. | Proposed |
| THEORY-227 v2 | Mutual information between an input and its representation does not fix the representation’s quality: invertible encoders all preserve it exactly, yet under a linear probe some do better than the raw input and some worse | Tschannen, Djolonga, Rubenstein, Gelly & Lucic (2020), LIT-247, read in NOTE-749. Invertible RealNVP encoders on MNIST halves keep the true MI constant, yet trained with an MI estimator they gain linear-probe accuracy, and one trained adversarially against a linear probe ends “significantly worse than using raw pixels”. At matched bound values the encoder architecture decides quality. What makes MI-trained representations good must therefore come from somewhere other than MI. The paper’s alternative, that InfoNCE works as metric learning, is argued, not tested. | Active |
| THEORY-228 v2 | On a finite augmentation space, at the population optimum of the spectral contrastive loss with k ≥ 2r, linear-probe error is bounded, up to constant and logarithmic factors, by the error of recovering the label from an augmentation divided by the squared sparsest-partition conductance of the augmentation graph | HaoChen, Wei, Gaidon & Ma (2021), LIT-249, read in NOTE-740. On a finite augmentation space, with labels recoverable from augmentations up to error α and k ≥ 2r, every population minimizer of the spectral contrastive loss has linear-probe error Õ(α/ρ²_⌊k/2⌋), where ρ is the sparsest-partition conductance of the augmentation graph (Thm 3.8). The guarantee needs no conditional independence of the views given the label. It is a statement about the population optimum of one loss: it says nothing about what training reaches, nothing about InfoNCE, and its infinite-space extension is asserted, not proved. | Active |
| THEORY-229 v3 | For a symmetrized Barlow Twins loss on a linear or fixed-kernel model, gradient flow from a small initialization of common scale aligned with the top eigenvectors of the pairs’ symmetrized cross-correlation learns them one at a time, in order of eigenvalue, and ends at a top spectral solution, which is exactly a minimum-norm zero-loss solution: kernel PCA on K_Γ | Simon et al. (2023), LIT-242, read in NOTE-741. For L = ‖C − I‖²_F with λ = 1, no batch norm and C symmetrized by hand, a linear model from aligned initialization learns mode j at τ_j ≈ −log(s_j(0)²γ_j)/(8γ_j) (Prop. 4.1, proved), the top spectral solutions are exactly the minimum-norm zero-loss ones (Prop. 4.3), and the kernel version is kernel PCA on K_Γ (Prop. 5.1). Generic small initialization is covered by an informal derivation only. The operator is built from the model’s kernel and the sample pairs, not from the positive-pair density ratio, and the stepwise growth seen in ResNets under SimCLR and VICReg is an experiment, not a consequence. | Active |
| THEORY-230 v2 | The Fisher–Rao geometry of a language model’s next-token laws over a set of contexts is a function of those laws alone, so models with identical laws share it, while activation geometry changes under invertible reparameterizations of the hidden layer that leave outputs unchanged; in Picozzi’s ten transformer, state-space and recurrent models it agreed at a mean rank correlation of 0.88 against 0.62 for activation geometry, and about 9% of the top eigenvector overlap exceeded a null matched to the shared laws | Picozzi (2026), LIT-936, read in NOTE-755, a single-author v1 preprint with code on request. The matrix of Fisher–Rao distances between a model’s next-token laws on a context battery is invariant to every output-preserving reparameterization, while activation geometry has a GL(d) gauge (Prop. 1); two models with identical outputs can have hidden geometries that rank-correlate at −4/17 (Prop. 3). Over ten transformer, state-space and recurrent models the mean rank agreement is 0.88 against 0.62 for mid-layer activations. Convergence is proved only as a consequence of each model’s excess risk to a common law, and that bound is vacuous on real models. Against a law-matched null, about 9% of the top-six eigenvector overlap is shared beyond the laws. Promoted on 2026-10-20. Replication and a test that varies the law are QUESTION-216. | Active |
| THEORY-233 v2 | CKA and normalized Bures similarity are the Euclidean and the Bures comparison of normalized kernels, and each bounds the other only within a factor set by the kernels’ ranks, so agreement in one does not carry to the other with a dimension-free constant | Harvey, Larsen & Williams (2023), LIT-250, §5 and App. A.3, read in NOTE-753. CKA = 1 − ½‖K_X/‖K_X‖_F − K_Y/‖K_Y‖_F‖²_F, while NBS = 1 − ½B²(K_X/Tr K_X, K_Y/Tr K_Y), with B the Bures distance. They induce the same topology, but CKA/√(r(K_X)r(K_Y)) ≤ NBS² ≤ min[r(K_X), r(K_Y)]·CKA, saturated at rank 1. NBS bounds CKA of the square-root kernels, with equality for commuting kernels. Linear kernels only, and the discrepancy is illustrated on Wishart samples, not trained networks. | Active |
| THEORY-234 v2 | Two softmax next-token predictors induce the same distribution exactly when their embeddings and unembeddings are related by an invertible linear map on the subspaces that enter the logits, with arbitrary distortion off them, so a linear relational property is shared by all such models or none when it lies in those subspaces, and can fail to be shared otherwise | Marconato et al. (2024), LIT-950, read in NOTE-746. For p(y|s) ∝ exp(f(s)ᵀg(y)) on a shared vocabulary, p_{f,g} = p_{f̃,g̃} iff the models are extended-linearly equivalent: invertible linear maps between the subspaces M and N that enter the logits, at any dimensions and without the diversity condition (Thm 5). The dimension of M is then a property of the distribution. Relational linearity, linear subspaces and linear probing hold in all equivalent models or none when Γ ⊆ N and Γ_q ⊆ M (Thm 14), and parallelism is shared only within N (Thm 15). Exact equality of distributions only, and no experiments. | Active |
| THEORY-236 v2 | A representational similarity index that is invariant to every invertible linear map takes the same value on all representations whose rank equals the number of examples | Kornblith, Norouzi, Lee & Hinton (2019), LIT-254, Theorem 1, read in NOTE-743. If s(X, Z) = s(XA, Z) for every invertible A, and rank(X) = rank(Y) = n, then s(X, Z) = s(Y, Z): any two full-row-rank representations of the same n examples are related by an invertible linear map. So when width reaches the number of examples, an index with that invariance, such as CCA, says nothing. The theorem fixes no group; the paper’s case for orthogonal invariance plus isotropic scaling is an argument from training dynamics, not a consequence of it. | Active |
| THEORY-237 v3 | The critic-based variational lower bounds on mutual information, Donsker–Varadhan, NWJ, MINE’s and InfoNCE, are relaxations of one unnormalized Barber–Agakov bound, and, when the joint and the product of the marginals are mutually absolutely continuous, each attains its supremum over critics only at a critic equal to the log density ratio up to a term its normalization leaves free, where every bound but InfoNCE equals the mutual information | Poole, Ozair, van den Oord, Alemi & Tucker (2019), LIT-246, read in NOTE-739. Writing q(x|y) = p(x)e^f/Z(y) in Barber and Agakov’s bound gives I_UBA, tight at f = log p(y|x) + c(y). Jensen gives Donsker–Varadhan, a tangent bound on the log gives TUBA and NWJ (unique optimal critic 1 + log p(x|y)/p(x)), and NWJ applied to a K-sample problem with the batch mean as normalizer gives InfoNCE. MINE’s bias-corrected gradient is the exact gradient of TUBA. A Monte-Carlo DV estimate is neither an upper nor a lower bound. The InfoNCE optimum is cited, not derived, in the source (the record derives it, note of 2026-10-19), and the bias–variance comparisons are on toys. | Active |
| THEORY-238 v3 | The expected generalization error of an encoder–decoder learner is bounded by the encoder alone, through the KL divergence of its train-and-ghost latents to a prior symmetric under label-preserving permutations, for any decoder; and under lossy compression the bound can stay finite for deterministic continuous encoders, whose lossless bound is infinite | Sefidgaran, Zaidi & Krasnowski (2023), LIT-245, read in NOTE-742. Theorem 4: E[gen] ≤ inf 2√((2·MDL + K + 2)/n) + ε, where MDL is the expected KL of the joint law of the latents of the training set and an independent ghost set to any prior invariant under label-preserving permutations, and ε is the distortion allowed by lossy compression. It holds for any decoder. With a product prior it reduces to sample-wise mutual information. The bound is never evaluated, the prior may depend on unseen ghost data, and a 2√((K+2)/n) floor remains. It does not prove that I(X;U) cannot bound generalization. | Active |
| THEORY-239 v3 | For any joint law of an input and a context, symmetric or not, the top-d left singular functions of the conditional-expectation operator attain the minimum, over d-dimensional spans, of worst-case linear-probe approximation error over tasks of compatibility at least 1 − ε, namely (s₁² − (1 − ε)²)/(s₁² − s²_{d+1}) when s_{d+1} < 1 − ε ≤ s₁, though other spans can tie | Zhai (2025), LIT-975, read in NOTE-770: Lemma 1.4, Theorem 3.2 and Theorem 3.4 with App. B. The operator g ↦ E[g(A)|X] runs from L²(P_A) to L²(P_X) and is never assumed self-adjoint; its singular value decomposition comes from composing it with its adjoint. The guarantee is for linear probes and approximation error at the population level only. The thesis never works a non-reversible kernel on one space; the note of 2026-10-19 works a three-state one. It says nothing of what a trained network reaches. | Active |
| THEORY-241 | Fitting the same function in the same architecture does not fix the internal decomposition: SGD reproduces three Picbreeder images from random weights without the reuse and factoring of the evolved networks, and perturbing its weights breaks the regularities that perturbing the evolved networks’ weights preserves | Kumar, Clune, Lehman & Stanley (2025), LIT-944, read in NOTE-771. For the skull, butterfly and apple CPPNs, SGD in the layerized evolved architecture fits the image, the network’s whole function on the grid. It shows no reuse of the symmetric parts, and single-weight and random-direction sweeps distort the output, where the evolved networks’ sweeps vary it meaningfully. Qualitative, curated and three images. The paper’s account of why, the order of discovery in open-ended search, and its extension to large models are conjecture. | Proposed |
| THEORY-243 v3 | In the three-unit identity network x ↦ x(1, 0, 1) ↦ (0, 2, 1)·h, aligning the input with the third unit and aligning it with the direction (h₁ + h₂)/√2, two alignments related by an orthogonal change of basis, pass every interchange intervention on the input equally, so interchange interventions on that variable alone do not fix which internal direction mediates it | Wu et al. (2024), LIT-961, Appendix A, and Makelov et al. (2023), LIT-968, Appendix A.3 and §8, read in NOTE-809 and NOTE-782. Both rotate the same three-unit identity network. The input aligns with h₃ in one basis and with (h₁ + h₂)/√2 in the other, equally well under interchange interventions on the input (re-derived 2026-10-20). The reply says intervening on further variables separates them; that was not checked. Makelov et al. rest their verdict on component boundaries; the reply rests its on the interventions performed. Shown exactly for one constructed network only; whether trained networks have such ties is open (QUESTION-234). | Active |
| THEORY-246 v2 | Among ResNet-50s trained on ImageNet with nine objectives and stopped early, seven of which beat softmax cross-entropy’s top-1 by up to 0.9 points, fixed penultimate features with more class separation transfer worse by linear probe to datasets whose classes are mostly not ImageNet’s (Spearman ρ = −0.93 across the nine; mean accuracy 75.9% for softmax to 48.3% for squared error), and within cosine softmax a higher temperature raises separation and lowers that accuracy; the objectives’ representations differ mainly in the last third of the network, and full fine-tuning brings mean transfer to within 1.3 points across them | Kornblith, Chen, Lee & Norouzi (2021), LIT-1044, read in NOTE-762. Across nine objectives for ResNet-50 on ImageNet, penultimate class separation R² runs from 0.349 (softmax) to 0.845 (squared error), Papyan’s Tr(Σ_W Σ_B†)/C from 21.2 to 0.33, and mean linear transfer to eight new-class datasets falls with R² (Spearman ρ = −0.93), from 75.9% to 48.3%. Cosine-softmax temperature raises R² from 0.24 to 0.77 and lowers transfer from 76.0% to 58.9%. CKA puts the differences in the last third of the network; fine-tuning erases them. The cost is for new classes: relearning the training classes, more separation helps. Networks early-stopped near 92% training accuracy; one dataset; correlational across losses. Promoted 2026-10-20 on this one study (ADR-047 v3), narrowed to what it measured. | Active |
| THEORY-251 | In few-shot classification by a base language model, failures from the choice of examples or from relabelled categories leave the class representations separable and fail at the vocabulary readout | Kirsanov et al. (2025), LIT-946, read in NOTE-789: at five demonstrations, accuracy across draws of examples runs from below 0.1 to about 0.6 while final-layer manifold capacity barely moves, and consistently swapped labels fail while capacity matches gold labels. The readout explanation is inferred from that dissociation, not measured; base Llama-3.1-8B and Gemma-2-2B, mostly synthetic data. | Proposed |
| THEORY-256 v3 | When a sequence generator’s latent dynamics factor as a tensor product, its predictive state embeds losslessly and linearly in orthogonal per-factor subspaces of total dimension Σ(d_n − 1); 4-layer GPT-2-style transformers trained by next-token prediction on five Mess3 and Bloch Walk factors settle near that dimension, in per-factor subspaces that are linearly independent and approximately orthogonal in their leading components, and under the token noise tested reach it first even when noise makes it lossy | Shai et al. (2026), LIT-986, read in NOTE-758. The lossless embedding is an elementary theorem (Prop. 2.2, Thm 2.3, App. A). The rest is experiment on five synthetic factors: about 10 effective dimensions against 242 for the joint state, approximate orthogonality in the first two components, and under token noise a collapse to the factored form before expansion. It does not show that the preference is architectural rather than a feature of training, a question the paper leaves open, and it varies neither the objective nor the data domain. | Active |
| THEORY-264 | A task given by demonstrations and the same task given by an instruction are carried in a language model by mostly different attention heads, so function vectors find no common task representation across the two presentations | Davidson et al. (NeurIPS 2025), LIT-942, read in NOTE-803: 4 to 7 of the top 20 function-vector heads are shared between the two forms in twelve Llama and OLMo models, the two vectors add, and swapping heads and activations between them costs accuracy. Kirsanov et al. (2025), LIT-946, read in NOTE-789, find the same split in representational geometry. Neither shows that no common representation exists, only that these methods find none; neither varies a pragmatic frame. | Proposed |
| THEORY-269 v2 | In OLMo-1B, OLMo-7B and GPT-J, how well an affine map from subject to object representation reproduces a factual relation (its LRE causality) tracks how often the relation’s subjects and objects co-occur in pretraining, with mean causality above 0.9 beyond about 4.4k, 2.0k and 1.1k co-occurrences respectively, and, across OLMo-7B’s eight checkpoints, at whatever step that count is reached | Merullo, Smith, Wiegreffe and Elazar (2025), LIT-972, read in full in NOTE-786. 24 factual relations, three models, eight OLMo-7B checkpoints; r = 0.82 between LRE causality and mean subject–object co-occurrence, above either term’s frequency alone. Correlational by the authors’ own statement. The linear objects are per-relation affine maps between hidden states, not attribute directions. | Active |
| THEORY-271 v2 | In a toy subject–verb–object corpus whose eight subjects carry every combination of three binary aspects, the subjects’ raw co-occurrence rows form a parallelepiped if and only if, for each object, its four sentences are equally probable; any other sentence distribution breaks it | Torii, Maeda and Hidaka (2024), LIT-977, Theorem 1 (§5.5), read in full in NOTE-781. Necessity is sound and short. The case analysis over permutations, and sufficiency, are loosely written. Exact shapes, raw counts, one design, and independent aspects that make the concept lattice Boolean. It says nothing about the logarithm or PMI, about noisy or approximate shapes, or about attributes that imply one another. | Active |
| THEORY-274 | Sequence models trained only on a graph’s edge bigrams store it as a global embedding geometry, not as a lookup, although the lookup is expressible, found faster and no less succinct; in a wide two-layer weight-tied model the geometry is the top Laplacian eigenspace, reached by cross-entropy training without a rank bottleneck, regularization or multi-hop supervision | Noroozizadeh, Nagarajan, Rosenfeld & Kumar (2025), LIT-995, read in NOTE-800. Empirical throughout: path-star graphs up to 5×10⁴ nodes, four tiny graphs, frozen-embedding controls; the two-layer spectral analysis is informal and rests on an unproved observation (App. F). It does not explain why deeper models prefer the geometry, which the paper poses as open, and its main transformer runs used weight decay. | Proposed |
| THEORY-278 v3 | Under MSE loss, in Han, Papyan and Donoho’s VGG, ResNet and DenseNet runs on five balanced ten-class datasets, the trained classifier stays near the least-squares classifier for its own features from an early epoch; and on that central path, for unconstrained features continually renormalized to identity within-class covariance and without weight decay, gradient flow drives the class-separation singular values to infinity at equal rates, which in that model is neural collapse | Han, Papyan & Donoho (2021), LIT-974, read in NOTE-783. MSE loss splits exactly into a least-squares term and a non-negative deviation term, and the deviation becomes negligible early in training in 15 dataset–network cells. On that central path, with λ = 0, the loss is ½ Σ 1/(ω_j² + C) in the singular values of Σ_W^(−1/2)M̃. Under a flow that re-whitens the features after every step, each ω_j grows as (4t/N)^(1/4), and their ratios tend to 1. That is NC1 and NC2, and NC3–NC4 follow. Not shown: that ordinary training is this flow, or anything for cross-entropy or with weight decay. | Active |
| THEORY-279 v2 | Early in training, image classifiers and language models act on the low-order statistics of their inputs, first-order before second-order, and later lose that sensitivity as they come to use higher-order structure | Belrose et al. (2024, ICML), LIT-952, read in NOTE-777. Over training, ConvNeXt V2, Swin V2 and RegNet-Y on five small image datasets are probed with images whose class means or covariances are grafted from another class, and with maximum-entropy samples that keep only those statistics. Accuracy on these rises and then falls, often below chance. Pythia fits unigram-sampled text before bigram-sampled text. The ordering is empirical and covers orders one and two. The Taylor argument that motivates it is a heuristic, and the paper does not measure an ordered sequence beyond second order. | Proposed |
| THEORY-281 v2 | Perfect concept erasure caps what a representation can keep at the information not about the concept: for X of finite support, if Z is independent of A then I(Z;X) is at most H(X|A), and a deterministic erasure reaches that cap only when the concept groups’ distributions are permutations of one another | Chowdhury et al. (2025), LIT-963, read in full in NOTE-820. The cap (Lemma 3.3) is a two-line chain-rule argument that needs only the Markov condition A → X → Z. The permutation condition is proved necessary and sufficient for deterministic maps (Appendices A.3 and A.4), with finite supports and disjoint groups. The paper’s further statement, that the cap cannot be reached when the groups are not permutations, is false for stochastic maps, which reach it for any finite distributions with disjoint supports. | Active |
| THEORY-282 v2 | A subspace intervention can reproduce the output effect of changing a variable without localizing it, by pairing a direction that varies with the variable but cannot reach the output with one that can reach the output but does not vary on the data | Makelov, Lange and Nanda (2023), LIT-968, read in NOTE-782. Proved for the construction (§3.2, Lemma A.1); shown in GPT-2 Small’s MLP8 on IOI (46.7% FLDD, falling to 13.5% without the null-space component, 4.2% interchange accuracy) and in GPT2-XL MLPs on factual recall, where a residual-stream direction passes the same test. It does not say that subspace patching is unreliable in general, nor that a found direction with a null-space component is spurious: in a one-dimensional readout the test passes both kinds. The reply (LIT-961) contests the label, not the measurements. | Active |
| THEORY-283 v2 | For a hierarchical multi-index target with isotropic Gaussian inputs and even, separable links, direction k becomes weakly recoverable by the Bayes-optimal estimator at a sample count proportional to its squared weight to the power minus one, so features emerge one at a time through sharp spectral transitions in the number of samples; a target-agnostic spectral estimator attains the thresholds and the optimal weighted error, and a two-layer network whose first layer is that estimator, with a ridge readout, attains them in excess risk | Defilippis, Krzakala, Loureiro and Maillard (2026), LIT-940, read in NOTE-805. Take x ~ N(0, I/d), n = αd, and f⋆ = Σ a_k g_k(⟨w_k, x⟩) with even links of generative exponent two and a_k ~ k^(−γ). Direction k is weakly recoverable at α_k = Θ(a_k^(−2)) (Thm. 3.2). The optimal weighted error is Θ(α^(−1+1/(2γ))) until α ≍ m⋆^(2γ). The spectral estimator on Σ T(y_i)x_ix_iᵀ attains both (Thm. 3.6). The cascade is in the number of samples, not in training time. The network result is for a spectral first layer plus a ridge readout, not gradient descent. | Active |
| THEORY-284 v2 | If a feature’s values form a compact metric space carried continuously and injectively onto the unit sphere, and cosine similarity is locally a C² decreasing function of squared feature distance, then every path on the representation manifold is √(−2g′(0)) times as long as the feature path it images, so on-manifold shortest paths recover the feature’s intrinsic length metric up to one scale; a homeomorphism alone does not fix it | Modell, Rubin-Delanchy & Whiteley (2025), LIT-935, Theorem 1, read in NOTE-802. Under Hypothesis 1 (continuous injective ϕ_f from a compact feature space to the sphere) and Hypothesis 2 (CosSim = g(d²) locally, g ∈ C², g′(0) < 0), path lengths scale by √(−2g′(0)). So geodesics on the representation measure the feature, without knowing g. Both hypotheses are assumed, not derived from training. GPT-2’s years are homeomorphic to both a linear and a logarithmic scale, and only the metric test told them apart. | Active |
| THEORY-286 v2 | Under class imbalance the last-layer geometry that unregularized cross-entropy prefers is not the simplex ETF but simplex-encoded-label interpolation: the logits equal the simplex-encoded labels, the classifier and embedding Gram matrices come from that matrix’s SVD, any finite weight decay changes the geometry, and training approaches it more slowly as imbalance grows | Thrampoulidis et al. (2022), LIT-994, read in NOTE-764. For the unconstrained-features max-margin problem under STEP imbalance, every solution has logits ŴᵀĤ = Ẑ (one-hot minus 1/k), ĤᵀĤ = UΛUᵀ and ŴᵀŴ = VΛVᵀ from Ẑ’s SVD (Theorem 1). This is the ETF only if the data are balanced or binary. No finite ridge gives it under imbalance (Proposition 1), though the ridge path tends to it (Proposition 2). ResNet-18 and VGG-13 runs sit closer to SELI than to the ETF, and approach it more slowly at larger R. Not shown: that gradient descent reaches it, and anything with more than one seed. | Proposed |
| THEORY-287 | Internal variables located by interchange intervention as mediating a language model’s in-distribution behaviour predict whether its outputs stay correct under shifts that keep the task’s mechanism better than output confidence or probes of the rest of the representation | Huang, Tao, Icard, Yang and Potts (2025), LIT-987, read in NOTE-765. In Llama-3-8B-Instruct over five tasks and ten shifts, counterfactual simulation is best on 8 of 10; a probe of the located variable beats a probe of its orthogonal complement on 7 of 10. Interchange accuracy and predictive power rise together only until accuracy nears 1. It does not say that the located variables are constitutive, that causal features always win, or anything about shifts that change the mechanism. | Proposed |
| THEORY-288 v2 | In small CIFAR and MNIST autoencoders whose only basis-privileging primitive is the activation function, elementwise tanh and Leaky-ReLU concentrate latent representations into narrow beams arranged by the activation’s discrete symmetry group, even in layers no activation precedes, while their orthogonally equivariant radial counterparts leave them spread, so axis-aligned discreteness there comes from the primitive, not the task | Bird (2025, v4 2026), LIT-976, read in NOTE-759. Autoencoders of width 12–32 and up to three hidden blocks on CIFAR and MNIST, with rotation-equivariant affine maps, initialisation and optimiser. Elementwise tanh (B_n) and Leaky-ReLU (S_n) give beams along or against the basis, or on the ±1 diagonal, and radial tanh and Leaky-ReLU (O(n)) do not. The evidence is density plots, five seeds a condition, with no statistic. The downstream claims, that this causes grandmother neurons, superposition and neural collapse, are untested. | Proposed |
| THEORY-292 | In CNN image classifiers, concept directions fitted one concept at a time align for attributes that co-occur in the labels, yet a joint fit finds mutually orthogonal directions that still separate each concept with little loss of AUROC, so the alignment is a property of the separate fits and not forced by the representation | Erogullari, Lapuschkin, Samek and Pahde (2025), LIT-937, read in full in NOTE-816. Shown at the last convolutional layer of VGG16 and ResNet18 on CelebA (natural correlations) and FunnyBirds (an injected 70% beak–tail correlation). AUROC is “largely preserved” by the paper’s account, with no values in the text and no error bars. It does not say the model’s own axes are orthogonal, and it says nothing about attributes that imply one another. | Proposed |
| THEORY-294 v3 | For a wide two-layer ReLU network with zero bias on one-dimensional data whose two classes lie on opposite sides of the origin, the tangent kernel is a constant multiple of the initialization kernel, so lazy training leaves the within-class variability of its features, NC1, where initialization put it | Kothapalli & Tirer (2024), LIT-985, read in NOTE-774. NC1 = tr Σ_W / tr Σ_B of kernel-defined features is a ratio of linear functionals of the kernel matrix (Theorem 4.1). For one-dimensional data, two classes on opposite sides of the origin and zero bias, the ReLU tangent kernel of a wide two-layer net is a constant multiple of the NNGP kernel, so NC1 is the same (Theorem 5.1, re-derived). In higher dimensions it is measured no lower, and for ReLU higher; whether lazy training leaves NC1 in place there is QUESTION-192. The inference that collapse requires feature learning is the reading’s, not proved. | Active |
| THEORY-299 v2 | Structured and translational knowledge-graph embeddings are 0-cochains of a cellular sheaf of vector spaces pulled back from the schema, each triple scored by a norm of the coboundary on its edge and, in the squared-Euclidean form, the whole graph by the sheaf Laplacian; since the zero cochain is always a global section, what they learn is a contrastive margin, not the existence of a section | Gebhart, Hansen and Schrater (2021; AISTATS 2023), LIT-232, §4 and Appendices A and C, read in NOTE-843. A direct identification, covering Structured Embedding, the Unstructured Model, TransE, TransR and RotatE and not bilinear models. That existence of a section is trivial here, and that the only gluing question is whether fixed data extend, is the paper’s own remark taken further by the record; it is what separates this use of sheaves from the contextuality line. | Active |
| THEORY-301 v2 | The Procrustes distance between regularization-normalized representations and their average linear-decoding distance bound each other only through the participation ratio of the kernel difference: a small Procrustes distance forces a small decoding distance, but not conversely | Harvey, Lipshutz & Williams (2024), LIT-257, Proposition 3 and App. A.1, read in NOTE-834. With K_X = XG(X)⁻¹Xᵀ/M, α = Tr K_X, β = Tr K_Y and R_Δ the participation ratio of K_X − K_Y, P⁴/R_Δ ≤ E‖Xw* − Yv*‖² ≤ (2(α + β)P² − P⁴)/R_Δ for targets with E[zzᵀ] = I. The Procrustes–Bures equivalence is imported from LIT-250. Kernels with orthogonal supports show the converse fails: Procrustes distance maximal, decoding distance as small as 2/r. Finite samples, linear readouts only. | Active |
| THEORY-305 v2 | SimCLR’s InfoNCE loss, over all n objects with a Gaussian kernel on the unit sphere, equals a graph-Laplacian quadratic form on the augmentation matrix plus the log of InfoNCE’s normalizers, which is spectral clustering only in a sense that admits any regularizer and does not make the minimizer the Laplacian’s eigenvectors | Tan, Zhang, Yang & Yuan (2023, ICLR 2024), LIT-227, read in NOTE-846: Theorem 3.1 and its proof (App. A), Lemma 2.5 and Definition 2.7. The identity between objectives is exact on a finite object set with symmetric π. The paper’s “spectral clustering” is any Laplacian form plus any regularizer; it names no minimizer and no population optimum, so it supports neither an eigenvector reading of InfoNCE nor the density-ratio optimum of THEORY-001. | Active |
| THEORY-306 v3 | MSN’s objective penalizes unequal cluster sizes explicitly and SwAV’s enforces equal sizes, and in one study with mini-batches of two to eight ImageNet classes, SimCLR, MSN and VICReg lost 11 to 18 points of ImageNet linear accuracy while MAE and data2vec lost under one | Assran et al. (2022), LIT-231, §§3–5 and Appendix C, read in NOTE-850. The prior is explicit in MSN’s loss (a KL to uniform over mean cluster assignments) and in SwAV’s equipartition constraint. For VICReg it rests on an informal reduction of a simplified loss to K-means and on K-means’s cited tendency to equal clusters, and for SimCLR on a cited equivalence, so the title claims the prior for MSN and SwAV only. The effect is shown with batches of two to eight ImageNet classes, one run per cell, not with natural long tails; the open half is QUESTION-171. | Active |
| THEORY-312 v2 | The large-batch InfoNCE loss is the negative HSIC between representations and image identity plus a nonnegative log-mean-exp of the centred critic, so it bounds a kernel dependence loss from above | Li, Pogodin, Sutherland & Gretton (2021), LIT-248, eq. 28 and Theorem B.1, read in NOTE-830. On a finite dataset of N images with one-hot identity labels, the large-batch InfoNCE loss equals −HSIC(Z,Y) + E_Z log E_Z′ exp(k − μ₁) exactly. The second term is nonnegative by Jensen (the record’s step), and for a bounded kernel it is at least γHSIC(Z,Z) less a second-order remainder (Theorem B.1, printed with the remainder’s sign reversed). It does not say InfoNCE is HSIC, and it says nothing of the finite-batch estimator. | Active |
| THEORY-319 v2 | When two views are redundant for a label, the predictor that infers one view from the other is within εX + 2√(εXεZ) + εZ of the Bayes predictor in mean squared error, and it is linear in the contrastive density ratio, so for a bounded label and a square-integrable ratio a linear readout of m random evaluations of the exact ratio approaches it at rate 1/m, with no symmetry between the views required | Tosh, Krishnamurthy & Hsu (2021), LIT-251, read in NOTE-841: Lemma 1, §2.3–2.4, Lemma 2 and Theorem 3. The label enters only through how redundant the views are about it; the representation is the density ratio p(x,z)/(p(x)p(z)), learned without labels. Population level. The error-propagation results need a uniform bound on the ratio, and the low-dimensional direct embeddings are shown to exist, not to be reached. | Active |
| THEORY-320 v2 | In a spectral contrastive graph that joins labelled known classes to unlabelled data, a novel labelling’s linear-probe residual is exactly its part outside the top eigenspace that the labelled rows of the discarded eigenvectors cannot absorb, so which partition of the unlabelled data is discovered depends on the known classes: ones sharing the target’s attribute can remove the residual, and ones sharing a rival attribute can raise it | Sun, Shi, Liang & Li (2023), LIT-235, read in NOTE-833. On the population graph, at the exact rank-k optimum, R(U*, y) = ‖(I − P_{L♭})U♭ᵀy‖² (Theorem 5.5, stated as ≤, proved as = in C.1). In a one-sample-per-kind toy, a known class sharing colour makes a colour labelling recoverable, and one sharing shape makes it worse than none (Lemma 5.4). The general error bound (Theorem 5.6) is approximate and pools all known classes into one block, and is not part of this account. Nothing here is about trained networks. | Proposed |
| THEORY-420 | In trained transformers, zeroing a decile of a weight matrix’s singular values does damage that falls monotonically with their size in square matrices, whose random-matrix null reaches zero, but not in rectangular ones, whose null has a positive lower edge: there the smallest decile is never the least damaging and can be among the most | Staats, Thamm & Rosenow (2024), LIT-856, read in NOTE-659. Freshly initialised matrices of BERT, Pythia and Llama follow the Marchenko–Pastur law; trained ones depart from it above the bulk, and, in rectangular matrices only, below it. Zeroing Llama-3 8B’s smallest Down-Projection decile leaves 2.0% on GSM8K from 43.2%, worse than every decile but the largest, while the same decile of the square Attention-Output leaves 40.0%. Whether the below-bulk outliers are what carries the damage is not tested: the decile mixes them with the lower bulk, and overlap with the input’s principal directions and ablation importance disagree for several matrix types. | Active |
| THEORY-421 v2 | In HGCN embeddings of two synthetic 3-ary trees trained for link prediction or node classification, distance from the origin is only a partial reading of hierarchical level: the root is not innermost (distance 1.8–3.3 against a minimum of 1.2–2.4) and level order is recovered for 69–75% of node pairs; a loss built only on pairwise distances leaves the origin arbitrary | Yang, Zhou, Ying, Chen & King (2023), LIT-871, read in NOTE-669. On two synthetic 3-ary trees of eight levels, HGCN leaves the root at hyperbolic distance 1.8–3.3 from the origin where the minimum is 1.2–2.4, the distances are roughly normal rather than leaf-heavy, and radius orders true levels for 69–75% of 5,000 random pairs. Recentring on the centroid plus an outward push raises that to 75–84%. The diagnosis rests on one architecture, two synthetic trees and one table without variance, and the paper does not separate a real failure to order levels from the origin’s arbitrariness under losses that see only pairwise distances. | Active |
| THEORY-439 v2 | Given the same few demonstrations, temporarily updating a language model’s weights on them and then conditioning beats conditioning alone, most on tasks with structural rules or distribution shift and most when the update is trained in the in-context format: for models of 1–8B parameters, examples used as context and the same examples used as a training signal are not interchangeable | Akyürek et al. (2024; ICML 2025), LIT-796, read in NOTE-637. On BIG-Bench Hard with Llama 3.1 8B and 10 demonstrations over five seeds, zero-shot 40.9%, in-context 50.5%, per-task LoRA test-time training then in-context 57.8%; 12 of 27 tasks gain at least 2 points, and Boolean expressions falls. Training the update on leave-one-out in-context tasks beats training on bare pairs (57.8% against 51.5%). The update is discarded after each task, so it is as transient as a context. It is two benchmarks and one model family, and which task property predicts the gain is open. | Active |
| THEORY-445 | A network built from group correlations over a discrete split group of plane symmetries, pointwise nonlinearities and coset pooling is equivariant to that group by construction, through the regular representation alone: the group is fixed before training and built into the weight sharing, not found in the data | Cohen & Welling (2016), LIT-314, read in NOTE-288. G-correlation commutes with the left action of G by the change of variables h → uh (eq. 12), and so do pointwise nonlinearities and pooling over cosets, up to edge effects and for rotations that are symmetries of the grid. Matched-parameter p4 and p4m layers lowered rotated-MNIST error from 5.03% to 2.28% and plain-CIFAR10 error from 9.45% to 6.46%. The paper uses no irreducible representations, characters or Fourier analysis, learns nothing about which symmetry the data have, and does not separate the effect of equivariance from the changed channel structure. | Active |
| THEORY-478 v2 | If the diffusion densities’ Hellinger distances obey a Hölder bound with exponent θ in d_T / t^(1/β) and are bounded below once d_T exceeds a multiple of t^(1/β), then for 0 < α < θ/β summing scale-weighted hyperbolic distances between diffusion densities over dyadic times gives a distance two-sided equivalent, up to constants, to min{1, d_T^(αβ)} for pairs above the finest scale: it keeps the tree’s neighbourhoods and not its global distances; whether a tree’s heat kernel, or sampled data, meets the bounds, and with which θ, is not shown | Lin, Coifman, Mishne & Talmon (2023), LIT-870, read in NOTE-675. The hyperbolic diffusion distance is equivalent to min{1, d_T^(2α)} for 0 < α < 1/2 (Proposition C.4), under heat-kernel bounds on the tree that are cited rather than proved; one Hölder step holds only for d_T ≥ 1, and repaired, the reader finds the admissible range halves. Pairs beyond a radius are only known to be far, which matches the method’s best mean average precision and its higher distortion than hyperbolic MDS on five graph benchmarks. A Euclidean distance on the same multiscale features does far worse. | Active |
| THEORY-481 v4 | Under gradient flow averaged over tasks and data, a linear-attention layer trained on in-context regression learns as a two-layer linear network from an aligned start does: when tasks share the input covariance’s eigenbasis and the weights start and stay aligned with it, then in the long-context limit each input mode is learned on its own logistic curve whose timescale falls with the square of its variance, so learning is staged by the input spectrum, and the layer converges to the estimator preconditioned by the inverse input covariance; at finite context length the modes decouple exactly only when inputs are isotropic and the modes start equal, and otherwise couple at order 1/N | Mainali & Teixeira (2025), LIT-859, read in NOTE-658. The prediction of one linear-attention layer reduces to p₂ W Σ̂_x q₁ x_q, a product of two learned factors, so averaged gradient flow decouples into modes with τ_α dp_α/dt = q_α(1 − p_α q_α s^∞_α) and τ_α = (ηP s_α²)⁻¹, logistic from a balanced start. The fixed point is ŷ = W Σ̂_x Σ_x⁻¹ x_q as N → ∞. The paper’s finite-N form, the one Ahn et al. and Zhang, Frei & Bartlett found, holds at finite N only for isotropic inputs with equal modes. The trajectory is new. The decoupling into independent modes is exact as N → ∞. At finite N it holds only for isotropic inputs with equal modes: the paper commutes the non-diagonal empirical covariance at (A71) and (A73), and the correct Wishart moment couples the modes at order 1/N. Alignment of the weights with the data’s eigenbasis is assumed throughout, the null blocks vanish only in expectation over tasks, and the transfer to non-linear transformers is timing coincidences. | Active |
| THEORY-496 | In few-shot classification and multiple choice by language models up to 175B parameters, whether each demonstration’s label is correct contributes little to the gain over zero-shot, while the distribution of the inputs, the label space and the paired format carry most of it | Min et al. (2022), LIT-814, read in NOTE-607. With k = 16 demonstrations, labels drawn at random from the correct label set cost 0–5 points against gold labels (2.6 on classification, 1.7 on multiple choice on average) across twelve model–inference combinations and 26 datasets. Out-of-distribution inputs, random English words as labels, or breaking the input–label pairing each cost far more. The random labels always come from the right label set, gaps reach about 14 points on single datasets, and the authors’ reading that the models learn no new correspondence at test time is untested on tasks pretraining cannot have taught. | Active |
| THEORY-503 | The nearest rank-r matrix to a real matrix in the least-squares norm keeps its r largest singular values and zeroes the rest; it is unique when the r-th and (r+1)-th singular values differ, its squared error is the sum of the dropped squares, and the singular-vector factors that build it are not unique | Eckart & Young (1936), LIT-330, read in full in NOTE-301, which checked the error bound. The argument is first-order stationarity on the orthogonal group plus two cited theorems (the SVD and a simultaneous-SVD criterion), and assumes a minimiser exists. Only the Frobenius norm and real matrices are covered. Uniqueness is of the approximating matrix, under a spectral gap, never of the factors; the paper’s last section argues the canonic components cannot be read straightforwardly as psychological factors. | Active |
| THEORY-504 v2 | In the Gemma Scope sparse autoencoders on Gemma 2, the two fidelity measures disagree by site: residual-stream autoencoders cost more next-token loss than MLP or attention ones at comparable fraction of variance unexplained, and base-model autoencoders transfer to the instruction-tuned model by loss more than by variance explained | Lieberum et al. (2024), LIT-611, read in NOTE-482. Across every layer and sublayer of Gemma 2 2B and 9B, JumpReLU autoencoders on the residual stream raise the language model’s loss more, when spliced in, than MLP and attention autoencoders whose FVU is about the same; and autoencoders trained on the base model lose little loss on instruction-tuned rollouts while showing a larger FVU gap. The authors’ explanations (the residual stream is the bottleneck; fine-tuning re-weights old features) are conjectures, and the authors say no agreed metric of autoencoder quality exists. | Active |
| THEORY-565 v3 | From a symmetric, positivity-preserving, positive semi-definite affinity, the normalized Markov operator is conjugate to a symmetric one, so its leading eigenfunctions scaled by powers of their eigenvalues embed the data with Euclidean distance equal to diffusion distance, and a density-corrected normalization recovers the Laplace–Beltrami operator independently of how the data were sampled | Coifman et al. (2005), LIT-981, read in full in NOTE-1441. The finite construction and its isometry are THEORY-728; the paper’s own diffusion distance is the one ã^(m) induces, not the inverse-stationary-weighted one. Theorem 2.1 and the α-family are stated without proof; the companion (LIT-1246, NOTE-985) proves the α-family’s limit, Active without boundary as THEORY-744, and its heat-kernel step is QUESTION-231. Self-adjointness comes from the kernel’s symmetry, and “directed diffusion” is anisotropic, not a non-reversible chain. | Proposed |
| THEORY-566 v2 | In Ganea et al.’s hyperbolic neural networks, linear maps and nonlinearities are Euclidean ones read through the chart at the origin, so a stack of them without biases is a Euclidean network conjugated by that chart, and the curvature acts only through the bias translations and the distance-based output layer | Ganea, Bécigneul & Hofmann (2018), LIT-879, read in NOTE-686, which checked the closed forms numerically. Möbius layers are defined as exp₀ ∘ f ∘ log₀, so any composition of them is exp₀ ∘ (the Euclidean composition) ∘ log₀, as the paper notes; only the Möbius bias, which is parallel transport from the origin followed by the exponential map, and the hyperbolic logistic regression’s distances to geodesic hyperplanes are not absorbed by the chart. The structural fact is proved; that the networks’ gains, confined to a synthetic task and to separating WordNet subtrees in low-dimensional Poincaré embeddings, come from those two operations is this record’s inference. NOTE-686 filed no THEORY. | Active |
| THEORY-640 | Measured as mutual 10-nearest-neighbour overlap, competent vision models agree more with one another and better language models agree more with vision models, but the cross-modal score reaches only 0.16 of a possible 1, is the maximum over all layer pairs, and was adopted after a global kernel metric showed only a very weak trend | Huh, Cheung, Wang & Isola (2024), LIT-302, read in NOTE-286. Across 78 vision models, those solving more VTAB tasks agree more in mutual k-nearest-neighbour structure; language models with lower bits-per-byte agree more with vision models on 1,024 image–caption pairs. The cross-modal score “only reaches 0.16” of 1, is maximised over layer pairs, and the metric was chosen after CKA “revealed a very weak trend”. It is local neighbourhood agreement that rises with competence, not kernel equality and not convergence to a model of reality. | Active |
| THEORY-683 | Permuting a fixed set of few-shot examples moves a GPT-family model’s classification accuracy between near chance and near the supervised state of the art, at every model size tested; the good orders are specific to the model, the spread survives more examples and calibration, and bad orders mostly fail by predicting one label | Lu et al. (2021, ACL 2022), LIT-833, read in NOTE-626. All 24 orders of four balanced SST-2 examples give accuracies from about 50% to above 85%, in GPT-2 and GPT-3 models of eight sizes; Spearman correlations of order accuracy between sizes lie between −0.35 and 0.27 (175B against 2.7B: 0.05). The variance stays high up to 32 examples and after calibration. It is a measurement on causal GPT models and text classification, with the transfer analysis resting on one example set, and the paper finds no property of an order that predicts its effect. | Active |
| THEORY-698 | Thresholding linear attribute probes on a language model’s embeddings yields a complete concept lattice for any incidence whatever, so the lattice is guaranteed by construction; what the Lattice Representation Hypothesis measured is per-attribute agreement with GPT-4o annotations at F1 70–83, and no lattice law | Xiong (2026), LIT-267, read in NOTE-240. Theorem 1 is the basic theorem of formal concept analysis applied to the crisp relation got by thresholding Fisher-LDA probes, and it holds for every binary relation; the paper’s own proof plan says so. The empirical content is that the thresholded probes on three 7–8B models agree with a GPT-4o-annotated incidence at macro F1 69.7–83.2 and predict WordNet subsumption at F1 57.1–77.1. No closure, absorption or recovered-lattice comparison is tested, and the paper’s join is not a lattice join as written. It is not evidence that language models encode a lattice beyond the incidence. | Active |
| THEORY-719 | On a finite augmentation space, over unconstrained functions of the two views, the spectral contrastive and NT-Logistic losses are both minimized at the positive-pair density ratio | Johnson, El Hanchi & Maddison (2023), LIT-258, App. B.2–B.3, read in NOTE-751: on a finite augmentation space, with every function of the two views allowed, the spectral contrastive loss is minimized by K⁺ = p⁺/(p⊗p) exactly, and the NT-Logistic loss with equal positive and negative classes by the log-odds log K⁺, so its exponentiated score is K⁺. Population optima over unconstrained functions only; not what an encoder reaches, and not InfoNCE, which is THEORY-001’s open part. | Active |
| THEORY-723 | For the final embedding and unembedding of a softmax model, the output distribution fixes the representations only up to an invertible linear map, and whether closeness in distribution bounds representational dissimilarity depends on the distance: not for KL in the saturation regime, but yes for the log-likelihood variance distance | Nielsen et al. (2025), LIT-253, read in full in NOTE-738, with Harvey, Larsen & Williams (2023), LIT-250, for the kernel side. What is observed fixes a representation only up to that observation’s symmetry group: a centred linear kernel up to a translation and an orthogonal map (THEORY-004), and a softmax output distribution up to an invertible linear map of the embedding with the inverse transpose on the unembedding (Thm 2.2). KL can go to zero while the embeddings stay non-linearly related (Thm 3.1, Cor. 3.2), but d_LLV ≤ ε gives d_f,g ≤ 2Mε (Thm 4.7). Final layer only. It does not say what the Platonic hypothesis means; that is THEORY-002. | Active |
| THEORY-728 v2 | On a finite data set, a symmetric positive affinity normalized to a Markov operator is conjugate to a symmetric one, and its eigenvectors scaled by powers of their eigenvalues embed the data so that Euclidean distance equals the diffusion distance weighted by the inverse stationary law | Coifman et al. (2005), LIT-981, read in full in NOTE-1441, with the derivation written out here; the 1/π-weighted distance is the companion’s (LIT-1246), not the paper’s own. For n points and a symmetric affinity K with positive entries, A = D⁻¹K is a Markov matrix with stationary law π ∝ d. It is conjugate to the symmetric D^(-1/2) K D^(-1/2), so its eigenvalues are real and in [−1, 1], and in [0, 1] when K is positive semi-definite. The map x ↦ (λ_j^m ψ_j(x))_{j≥1} turns the m-step diffusion distance, weighted by 1/π, into Euclidean distance exactly. Truncation gives a lower bound. It says nothing about the continuum limit, which is THEORY-565. | Active |
| THEORY-730 | For an embedding whose Gram matrix is the absolute co-occurrence deviation matrix at full rank, words on a periodic lattice whose positive and negative co-occurrence parts each depend only on separation have Fourier modes of the lattice as principal components, in degenerate cos/sin pairs with amplitude the square root of the kernel’s Fourier transform, and the exponential kernel gives a closed loop with integer frequencies on a periodic lattice and an open curve with non-integer frequencies on an interval | Karkada, Korchinski, Nava, Wyart & Bahri (2026), LIT-860, Propositions 1, 3 and 4 and Corollary 2, with the proofs in Appendix E followed in NOTE-664. The result is conditional throughout. It concerns an idealised embedding with W Wᵀ = |M*| and d ≥ rank M*, not a trained one. It needs the positive and negative parts of M* to be translation invariant separately (Assumption B.1), not M* alone. It says nothing about whether any corpus or model satisfies these conditions, which is THEORY-183. | Active |
| THEORY-732 | For ReLU networks interpolated linearly between two endpoints, weak additivity and commutativity at every layer make each layer’s features exactly the interpolation of the endpoints’ features, output-layer linearity bounds the path’s error by twice the endpoints’, and for one hidden layer with interpolating minima linear mode connectivity alone gives the exact case | Zhou et al. (2023), LIT-674, read in NOTE-512 with Appendix A’s proofs checked: for an L-layer ReLU MLP on a finite dataset, weak additivity of ReLU over the two networks’ pre-activations and the commutativity W_A H_B + W_B H_A = W_A H_A + W_B H_B at every layer make the interpolated network’s features at every layer exactly the interpolation of the endpoints’ features (Theorem 1). Output-layer linearity, at any positive scale, keeps the path’s error within twice the endpoints’ (Lemma 1). Lubana et al. (2022), LIT-669, Lemma 2: for one hidden layer and interpolating minima, linear mode connectivity forces shared activation patterns, and with shared patterns the hidden features are linear along the path. It says nothing about when the conditions hold. That trained pairs satisfy them is THEORY-111’s empirical claim. | Active |
| THEORY-734 | In the hyperbolic plane, a root carrying deg_max ≥ 15 chains of length m, embedded with worst-case distortion 1 + ε for ε ≤ 1, forces some node to hyperbolic distance at least (ℓ/(4ε))·ln(deg_max/(π√2)) from the origin, so fixed-point Poincaré coordinates need Ω((ℓ/ε) log deg_max) bits | Sala, De Sa, Gu and Ré (2018), LIT-874, Lemma D.1, read in NOTE-679, with the proof redone here without the paper’s small-angle approximation. Only the root’s children are used to fix the embedding’s scale c, and then the chains turn the scale into depth. Plane only, one family of trees, fixed-point coordinates in the Poincaré disk, deg_max ≥ 15 and ε ≤ 1. It says nothing about H^r for r > 2, or about whether any tree needs fewer bits under another number system. | Active |
| THEORY-737 | In a spectral contrastive graph that joins labelled known classes to unlabelled data, the least-squares residual of a novel labelling on the top-k eigenspace, with the labelled entries left free, equals exactly the part of the labelling outside that eigenspace that the labelled rows of the discarded eigenvectors cannot absorb | Sun, Shi, Liang & Li (2023), LIT-235, Theorem 5.5 with its proof in C.1, read in NOTE-833. The paper states the identity as ≤, and its proof is a chain of equalities. Re-derived here in three steps from orthogonality and least squares. With the paper’s Lemma 5.1 it bounds the novel labelling’s 0–1 linear-probe error by twice this residual. It is about the exact rank-k factorization of a population graph, not about trained networks. | Active |
| THEORY-738 | In the unconstrained-features max-margin problem with k classes under step imbalance and feature dimension at least k − 1, the unique optimal logits are the simplex-encoded labels, every margin equals one, and the classifier and embedding Gram matrices are the two square-root factors of that matrix’s singular value decomposition; the geometry is the simplex equiangular tight frame exactly when the classes are balanced or there are two of them | Thrampoulidis, Kini, Vakilian and Behnia (2022), LIT-994, Theorem 1 and Corollary 1.3, read in NOTE-764 and re-derived here. By THEORY-726 the problem is nuclear-norm minimisation of the logits under margin constraints. The polar factor of the simplex-encoded label matrix certifies it optimal, because under step imbalance its off-diagonal entries are negative. It does not say that gradient descent or trained networks reach this geometry, which is THEORY-286’s. | Active |
| THEORY-739 | For the quartic proxy of word2vec with tied weights, symmetric reweighting and constant weights, the rank-d minimisers are the top d eigenvectors of the relative deviation-from-independence matrix M*, scaled by the square roots of their eigenvalues, up to rotation; and from an initialisation aligned with those eigenvectors, gradient flow acquires each one in an independent logistic rise, in order of eigenvalue | Karkada, Simon, Bahri and DeWeese (2025), LIT-855, Theorem 1 and Proposition 2, read in NOTE-660 and re-derived here. The quartic proxy is a function of WWᵀ alone, so von Neumann’s trace inequality gives its rank-d minimisers. Gradient flow from an aligned start keeps the eigenbasis and reduces each mode to a logistic equation. It does not say that word2vec, or the proxy from random initialisation, behaves this way; that is THEORY-182’s. | Active |
| THEORY-740 | A two-layer linear network trained by gradient flow on white inputs from a decoupled, balanced start learns each singular mode of its input–output correlation in a logistic rise that waits a time of order its inverse strength times the log of the inverse initial scale, so the modes arrive one at a time in order of strength; a single-layer network on the same data learns every mode on one timescale with no plateau, so under these conditions the stages come from the product of layers and the data’s spectrum | Saxe, McClelland and Ganguli (2014, 2019), LIT-864 and LIT-862, read in NOTE-667 and NOTE-666, re-derived here. From a decoupled, balanced start the two-layer equations keep the SVD basis of Σyx, and each mode’s strength obeys τȧ = 2a(s − a), a logistic rise. One weight matrix on the same data relaxes every mode exponentially on the one timescale τ. It does not say that random starts follow the solution, which is THEORY-186’s. | Active |
| THEORY-745 | For the reversible walk p = k/d built from a symmetric non-negative kernel on a finite connected set, the t-step diffusion distance is exactly the Euclidean distance between the coordinates λ_l^t ψ_l, and keeping only the coordinates with |λ_l|^t > δ|λ₁|^t errs in D_t² by at most δ²|λ₁|^{2t}(1/π(x) + 1/π(y)), an absolute bound and not the relative accuracy δ that Coifman and Lafon state; the truncated maps are nested as t grows, so with δ fixed the steps t order a refinement chain by eigenvalue modulus, while untruncated they are all invertible images of one another | Coifman and Lafon (2006), LIT-1246, read in full in NOTE-985. The identity D_t² = Σ λ_l^{2t}(ψ_l(x) − ψ_l(y))² is exact. The paper’s Proposition 1 says the truncated map keeps it “up to relative accuracy δ”. The true bound is absolute and can be large relative to D_t for nearby points. In the refinement sense, the diffusion-map family coarsens with t only through the threshold. Finite sets, counting measure and reversible walks only. It says nothing about the manifold limit or about non-reversible chains. | Active |
| THEORY-746 | Over a fixed finite set of basis functions, slow feature analysis is a generalized eigenproblem: the outputs that minimize ⟨ẏ²⟩ under zero mean, unit variance and decorrelation are the bottom generalized eigenvectors of the derivative covariance against the covariance, in order of slowness, and a stack of such stages, each of whose function spaces contains its inputs, can only keep or lower each of its first m slowness values on the training signal, and may keep them | Wiskott and Sejnowski (2002), LIT-1249, read in full in NOTE-971. Within the span of a fixed basis, the slowest decorrelated unit-variance outputs are the bottom generalized eigenvectors of the pencil (⟨ż̃ż̃ᵀ⟩, cov z̃), and each output’s slowness is its eigenvalue. A consequence the paper does not draw: a stack of such stages whose function spaces contain their inputs is never faster at a higher stage, componentwise, on the training signal, so in this model “higher is slower” is imposed by the objective, weakly, and can stall. It is about the training signal and a fixed function space. It says nothing about test data, about a generative model, about invariance as opposed to slowness, or about learning dynamics. | Active |
| THEORY-759 | A six-layer, 256-unit ReLU network trained by full-batch Adam to regress a sum of sinusoids at 5 to 50 cycles on [0, 1] fits the components in order of frequency, and keeps that order when the amplitudes rise tenfold with frequency, so in this setting the order of acquisition is set by the learner and not by the strength of the target’s components | Rahaman et al. (2019), LIT-1261, Experiment 1, read in full in NOTE-983. With equal amplitudes, and with amplitudes rising from 0.1 at 5 cycles to 1 at 50, the normalised fit |f̃(k_i)|/A_i reaches 1 at low frequency first. This is the reverse of the order by strength that THEORY-186 gives for a deep linear network. One architecture, one optimiser, a 1-D input, ten runs averaged. The paper’s theory gives a bound on speed, not this order. | Proposed |
| THEORY-775 | With a Gaussian random-walk prior and an instantaneous unit-variance Gaussian encoder, the KL half of the evidence lower bound is the unordered slowness loss plus a constant; with a linear Gaussian encoder and decoder on whitened input, the bound is maximized on the span of the slowest SFA directions whose lag-one autocorrelation exceeds one half, with the basis inside that span unidentified, and on unwhitened input the maximizer is neither SFA’s subspace nor invariant to rescaling the input | Schüler and Wiskott (2025), LIT-1296, read in full in NOTE-1015. The ELBO of a dynamical VAE with prior N(z_{t−1}, I) and encoder N(g(x_t), I) is Σ E log p(x_t | z_t) − ½Σ‖g(x_{t−1}) − g(x_t)‖² − (T − 1)d/2, an exact identity. For linear Gaussian encoder and decoder on whitened input, the maximizer spans the bottom eigenvectors of the difference covariance with eigenvalue μ < 1, each scaled by (μ^{−1/2} − 1)^{1/2} and then rotated arbitrarily. So it recovers SFA’s slow span, cut at lag-one autocorrelation ½, not SFA’s features. On raw input it trades variance against slowness. It says nothing about nonlinear encoders, about predictive sufficiency, or about Turner and Sahani’s model. | Active |
| THEORY-782 | In networks that learn addition from summed embeddings, the critical training fraction of grokking is the fraction at which the training set’s same-answer pairs come to fix the generalising representation up to translation and scale, at every group size, as Liu, Kitouni, Nolte, Michaud, Tegmark and Williams propose | Liu, Kitouni, Nolte, Michaud, Tegmark and Williams (2022), LIT-858, read in NOTE-662, with Power et al.’s critical fraction (LIT-341). The paper presents the critical training fraction as where the training set’s equations fix the embedding. THEORY-181 holds the theorem for the ideal model and the match at the one size measured, p = 10 with 1D embeddings. Held to addition: for S₃ the networks generalise beyond this mechanism. The multi-size test is QUESTION-199. | Proposed |
| THEORY-786 | Deep networks learn hierarchically generated data from a number of examples set by when the correlations between input patches and the class rise above sampling noise, so detecting those correlations is how they learn the grammar, as Cagnetta and coauthors propose | Cagnetta, Petrini, Tomasini, Favero and Wyart (2023), LIT-877, propose that deep networks learn compositional data by detecting the correlations between input patches and the class, and that the sample complexity is where those correlations emerge from sampling noise. The record holds as shown, in THEORY-195, the measured law and the synonym-invariance it coincides with. This is the mechanism, filed under ADR-047 v4 when the narrowing dropped it. | Proposed |
| THEORY-788 | Training a deep classifier past zero training error generally collapses its last-layer features onto their class means, drives the centred means toward a simplex equiangular tight frame aligned with the classifier, and makes the decision nearest class mean, as Papyan, Han and Donoho propose from their runs | Papyan, Han and Donoho (2020), LIT-618, title their paper on the “prevalence” of neural collapse and present it as a pervasive outcome of the terminal phase of training. The record holds as shown, in THEORY-083, only what their runs measured: three architectures with cross-entropy on seven balanced datasets without augmentation. This is the general law, filed under ADR-047 v4 when the narrowing dropped it. | Proposed |
| THEORY-789 | The block coarse-graining that maximizes mutual information with the system beyond a buffer picks out the renormalization-group-relevant variables in general, so those variables can be found from samples by information alone, without being known beforehand, as Koch-Janusz and Ringel propose | Koch-Janusz and Ringel (2017), LIT-873, propose that their mutual-information criterion finds a system’s relevant degrees of freedom with no prior knowledge of them. What the record holds as shown, THEORY-194, is that it recovered variables renormalization already uses, on 2D models whose answer was known. This is the general thesis that result was offered for, filed under ADR-047 v4 when the narrowing dropped it. | Proposed |
| THEORY-793 | Gradient flow on a symmetrized Barlow Twins loss with a linear or fixed-kernel model, from generic small initialization and not only from initialization aligned with the eigenvectors, learns the top eigenvectors of the pairs’ symmetrized cross-correlation one at a time in order of eigenvalue and ends at a top spectral solution, as Simon and coauthors derive informally | Simon et al. (2023), LIT-242, read in NOTE-741. The stepwise dynamics, the minimum-norm end point and the kernel-PCA form are proved from aligned initialization, and that part is the Active THEORY-229. Generic small initialization rests on the paper’s Result 4.6, which its authors style a result and derive informally in App. C, with numerical checks at d = 3. This THEORY holds that generic case. | Proposed |
| THEORY-794 | In vision models, compositional generalization to unseen combinations of concepts is driven by the diversity of the combinations seen in training rather than by data volume, and arrives with representations that are linearly factored by concept, as Uselis, Dittadi and Oh propose | Uselis, Dittadi & Oh (2025), LIT-667, read in NOTE-525. Their title and abstract present diversity of combinations, not data volume, as what drives compositional generalization in vision models. Their evidence is ResNet-50s trained from scratch on two-concept grids, with checkpoints chosen on the unseen pairs, and that measured part, with the factorization theorem, is the Active THEORY-117. This THEORY holds the general causal claim. | Proposed |
| THEORY-797 | Deep classifiers trained with MSE loss stay near the least-squares classifier for their own features and move along the central path by the dynamics of continually renormalized gradient flow, whose equal-rate growth of the whitened class-separation singular values is how they reach neural collapse, as Han, Papyan and Donoho propose | Han, Papyan & Donoho (2021; ICLR 2022), LIT-974, read in NOTE-783. The paper’s proposed explanation of neural collapse under MSE in real networks, filed apart from its measured central-path proximity and its derivation in a renormalized model (THEORY-278) under ADR-047 v4. Renormalization is imposed, not measured, and every experiment used weight decay, which the derivation excludes. | Proposed |
| THEORY-798 | Independently trained neural encoders of the same data learn latent spaces that differ mainly by an angle-preserving map, a rotation or reflection times a rescaling, so their cosine similarities to shared anchor samples agree across seeds, architectures and languages, as Moschella et al. propose | Moschella, Maiorca, Fumero, Norelli, Locatello & Rodolà (2022; ICLR 2023), LIT-849, read in NOTE-652. The hypothesis behind relative representations, filed apart from what their experiments measured (THEORY-175) under ADR-047 v4. The source assumes it and tests it only through its consequences; it never fits a map between two spaces. | Proposed |
| THEORY-802 v2 | In trained networks as in constructed ones, interchange interventions on a fixed set of variables can be satisfied equally by alignments related by a change of basis, so which internal direction mediates a high-level variable is fixed only relative to a chosen decomposition into units, and complex models will generally have several distinct, fully correct abstractions | The general reading that THEORY-243 held before the promotion audit narrowed it to the three-unit network it proves (batch 062). Wu et al. (2024), LIT-961, conjecture that complex models have “multiple distinct fully correct abstractions” and that the causal-abstraction framework cannot settle structural equivalence; Makelov et al. (2023), LIT-968, §8, grant that crossing component boundaries makes “illusory” tenuous, and answer by privileging components. Filed under ADR-047 v4 so the wide form stays a thesis evidence can confirm or reject. Its only support is the constructed tie. | Proposed |
| THEORY-805 | Volume-maximizing joint-embedding objectives as a class, VICReg and SimCLR as well as MSN and SwAV, carry a prior toward equal-sized clusters, and that prior costs them semantic transfer when the pretraining data are class-imbalanced, including on naturally long-tailed data | The general reading that THEORY-306 held before the promotion audit narrowed it to the explicit priors of MSN and SwAV and to one experiment with two to eight classes per batch (batch 033). Assran et al. (2022), LIT-231, read in NOTE-850, argue the prior for the whole class, VICReg through an informal reduction to K-means and SimCLR through a cited equivalence, and read it as the cause of the degradation reported on real long-tailed data. Filed under ADR-047 v4 so the wide form stays a thesis evidence can confirm or reject. | Proposed |
| THEORY-808 | Transformers trained by next-token prediction on sequence processes whose latent dynamics factor as a tensor product learn to represent the belief state in approximately orthogonal per-factor subspaces of about the factored dimension rather than the joint one, and reach that factored representation first even when noise makes it lossy | The general reading that THEORY-256 held before the promotion audit narrowed it to the five Mess3 and Bloch Walk factors and the 4-layer GPT-2-style models tested (batch 019). Shai et al. (2026), LIT-986, read in NOTE-758, present next-token training (the paper is titled “Transformers learn factored representations”) as yielding factored, orthogonal belief representations; NOTE-758’s claims C2 to C4 are each shown on one family of synthetic processes. Filed under ADR-047 v4 so the wide form stays a thesis evidence can confirm or reject. It does not include the paper’s inference that the preference is architectural (C5), which App. D.4 leaves open, or the claim that the factored form persists at convergence under strong noise, which Fig. 12 contradicts. | Proposed |
| THEORY-813 | Training a wide network in the lazy, kernel regime does not make its features more collapsed than at initialization, so neural collapse, where it occurs, is something only feature learning produces | The general reading that THEORY-294 held before the promotion audit narrowed it to the one-dimensional, zero-bias, two-class case it proves. Kothapalli and Tirer (2024), LIT-985, read in NOTE-774, draw it as their conclusion: the lazy regime does not collapse anything, so the data-dependence of collapse has to come from a kernel that moves with the data. Beyond the solved case their support is simulation on two-class Gaussian data, where the NTK comes out less collapsed than the NNGP. Filed under ADR-047 v4 so the wide form stays a thesis evidence can confirm or reject. | Proposed |
| THEORY-825 | Hyperbolic graph embedding models trained on task losses that never mention the hierarchy do not place a tree’s root nearest the origin, and their distance from the origin orders hierarchical levels only partly | Yang, Zhou, Ying, Chen & King (2023), LIT-871, read in NOTE-669, assert for hyperbolic graph models in general what they measured for HGCN on two synthetic trees: the root is not innermost and radius reads level only partly. The general form was the lead clause of THEORY-421 until that account was narrowed to the measurement on 2026-10-20 (ADR-047 v4); it is filed here so that it stays a thesis evidence can confirm or reject. | Proposed |
| THEORY-865 v2 | In Gurnee, Sofroniew et al.’s experiments on Claude Sonnet 4.5, Haiku 4.5, Opus 4.5 and Opus 4.6, the residual-stream directions a corpus-averaged Jacobian lens marks as verbalizable (the J-space) carried most of the causal effect on verbal report and on unspoken two-hop intermediates while holding a small share of their variance (swaps along them succeeding on 59% and 61% of trials, against 5% and 28% for the remainder, which fell to 0% and 6% with the J-space clamped), held instructed concepts, and passed one swapped argument to several functions (76 of 192 swaps); in two small task sets (n = 8 and 11), report and flexible inference followed a swap while continuation, anomaly detection and line wrapping did not, a split the authors partly define by this result; and by the lens’s own statistics the J-space works in a middle band of layers, holds about 25 active directions and under 10% of activation variance, and is amplified and relayed by the weights more than other directions | Gurnee, Sofroniew et al. (2026), LIT-1347, read in NOTE-1069 (main text in full, appendices in part). It is the functional analogue of conscious access, in the access sense only, found in a transformer and tested by intervention. The sharp edge: one company’s models studied by that company, a lens whose own statistics draw the layer band, a flexible/automatic split partly defined by the result, and an explicit refusal to say anything about phenomenal consciousness. | Active |
| THEORY-869 v2 | In Claude Sonnet 4.5, Opus 4.5 and Opus 4.6, Gurnee, Sofroniew et al. found that projecting out the ten most active J-space directions in layers 38 to 54 of 100 lowered an experiential-language score, graded by Opus 4.6-based model graders on five or six prompts per condition, both in the models’ narrations of their own processing and in their descriptions of another person’s experience, while norm-matched perturbations along other directions left it near baseline and the text stayed fluent; on Haiku 4.5 coherence failed first | Gurnee, Sofroniew et al. (2026), LIT-1347, §3.5.3 and §A.23, read in NOTE-1069. The authors conclude that “the J-space supports the model’s propensity to provide rich experiential reports, whether about itself or another entity”. The sharp edge: the measure is a register of text graded by another model, on five or six prompts per condition, in one company’s models; and the effect holds for reports about others, so it is not evidence of anything the self-reports are about. | Active |
| THEORY-965 | Sparse-autoencoder features of a large language model’s residual stream are units the model computes with: they track abstract concepts across languages, modalities and concrete or abstract instances, and intervening on one changes the model’s behaviour in the way its label predicts, better than directions found from a few labelled examples | Templeton et al. (2024), LIT-1533, read in NOTE-1178, with Bricken et al. (2023), LIT-1545, for the one-layer case. The thesis of both papers: features are the model’s own units, shown by steering. The authors’ own hedges: steering needs features clamped beyond their observed range; the reconstruction error rides along; “we haven’t rigorously evaluated our features against other steering methods”. The record holds a result against it for known causal variables (THEORY-968). | Proposed |
| THEORY-966 | Greedy edge pruning of the ACDC kind, which removes each edge whose removal changes the output metric by less than a threshold given the edges already removed, keeps only one input of a redundant OR gate, and which input it keeps depends on the order in which edges are visited, so when a model has redundant sufficient paths the circuit it returns is one of several sufficient subgraphs | Conmy et al. (2023), LIT-1547, Appendix M, read in NOTE-1170. For out = 1 − ReLU(1 − x − y) with x = y = 1, removing either input alone leaves the output at 1, so the first one visited is pruned and the second is then kept; visiting y first keeps x instead. It does not say how common redundant paths are in trained models, nor that any other method does better: HISP keeps neither input. | Active |
| THEORY-967 | In a two-layer attention-only transformer, attention heads change function during training: heads in the first layer that predict simple n-grams early later become the previous-token heads of the induction circuit, a bracket-matching support head, or supporting members of a circuit with second-layer n-gram heads, giving up simple n-grams as they do, and particular n-grams pass from one head to another; so later circuits form partly by repurposing components that already had a role | Wang, Hoogland, van Wingerden, Furman and Murfet (2024), LIT-1535, read in NOTE-1188. “Heads 0:1, 0:4, 0:7 ultimately become the two previous-token heads and (layer 0) Dyck head respectively. However, these heads appear to start out as simple n-gram heads.” Layer-0 multigram heads lose information about simple n-grams from stage LM3, as path patching and composition scores show them coming to feed layer-1 bracket-matching heads. The “ago” n-gram moves from head 0:0 to head 1:0. Whether the late roles are built from the early ones, or merely follow them in time, is not tested. | Proposed |
| THEORY-968 | In the Mechanistic Interpretability Benchmark’s causal-variable localization baselines on Gemma-2 2B and Llama-3.1 8B, features of the public Gemma Scope and Llama Scope sparse autoencoders, selected by a learned binary mask, never gave a higher mean interchange-intervention accuracy than the same mask over the model’s own hidden dimensions, in all sixteen task–variable cells and by 1 to 14 points, while distributed alignment search, trained against the causal model it is scored on, gave the highest mean in fifteen of them | Mueller et al. (2025), LIT-1550, read in NOTE-1169. Unsupervised SAE features were worse units than neurons for known task variables across ARC, MCQA, two-digit addition and RAVEL. It does not say SAE features are never causal units: the comparison is on residual streams, under one selection method, with the base reconstruction error carried into every intervention, and the best method is supervised by the hypothesis it is scored against. | Active |
| THEORY-970 | On three language-model tasks whose circuits earlier manual work reported, indirect object identification and Greater-Than in GPT-2 Small and Python docstrings in a four-layer attention-only model, ACDC run with resampled corrupted activations against KL divergence ranked the reported edges with areas under the ROC curve of 0.85 to 0.98, above head-importance pruning and subnetwork probing on all three; on two compiled tracr programs it ranked them poorly with corrupted activations, 0.68 and 0.20, and exactly with zero activations | Conmy et al. (2023), LIT-1547, read in NOTE-1170. Edge AUCs with corrupted activations and KL: Docstring 0.982, Greater-Than 0.853, IOI 0.869 for ACDC, against 0.805, 0.693, 0.789 for HISP and 0.937, 0.806, 0.823 for SP. It does not say ACDC finds the true circuit: the reported circuits are incomplete, the results depend on metric, ablation and parent order, and negative components are missed at sparse thresholds. | Active |
| THEORY-971 | A unit that thresholds a pre-activation equal to a term depending only on one operand plus a term depending only on the other cannot be active on exactly the pairs with a given sum when that sum has two decompositions: if it is active on (x, y) and (z, w), it is active on (x, w) or on (z, y); so a sum computed from separately represented operands needs an earlier unit that combines them, as Ameisen et al. argue for addition in language models | Ameisen et al. (2025), LIT-1552, read in NOTE-1185, re-derived here. It is why their addition graphs contain “lookup table” features between operand features and sum features. Whether a given unit in a trained model meets the premise is an empirical matter: attention patterns that depend on both operands, or any earlier nonlinearity that mixes them, break it, and that mixing is exactly the intermediate step the result says is required. | Active |
| THEORY-973 v2 | In small attention-only transformer language models of two to six layers, in-context learning, measured as the loss at token 500 minus the loss at token 50, appears abruptly in one early window of training in which induction heads form, and ablating those heads at test time removes nearly all of it; one-layer models show neither, and letting a single layer attend by the previous token brings the jump to one-layer models and moves it earlier in two-layer ones | Olsson et al. (2022), LIT-1538, read in NOTE-1173: Arguments 1–3 for the small attention-only series, with the smeared-key models. The score jumps from under 0.15 to about 0.4 nats in the window where induction heads form, pattern-preserving ablation of the induction heads removes nearly all of it to the end of training, and an architecture that lets one layer express induction moves the jump. It says nothing causal about models with MLPs or large models, and nothing about few-shot task learning, which the score does not measure. | Active |
| THEORY-974 | When a transformer’s MLP sublayers are replaced by transcoders and the attention patterns and LayerNorm scales on a prompt are held fixed, the contribution of an earlier transcoder feature at the same token to a later one’s pre-activation is the earlier feature’s activation times the dot product of its decoder with the later feature’s encoder, scaled by one LayerNorm constant per token, so the weight factor is fixed by the trained weights and holds on every input | Dunefsky, Chlenski and Nanda (2024), LIT-1554, read in NOTE-1174, Eq. 7 and App. D.2–D.4, re-derived here. The weight factor is a weight of the replacement model, not of the original, and it is exact only for the replacement. It says nothing about why attention attends where it does, and “input-invariant” holds only up to a per-token LayerNorm scalar and only while the later feature is active. | Active |
| THEORY-975 | In a two-layer attention-only transformer trained on the Pile, the weight-refined local learning coefficients of individual attention heads start alike and diverge over training, and clustering the curves groups the heads by the functional types that ablation assigns them at the end of training, previous-token, current-token, induction and n-gram-memorizing; among layer-1 heads, those running simple algorithms end with the lowest values and those memorizing the most n-grams with the highest | Wang, Hoogland, van Wingerden, Furman and Murfet (2024), LIT-1535, read in NOTE-1188: §4.1, Figs. 1–2 and 11, appendices B, F.5 and G. Per-head curves of how degenerate the loss is in that head’s weights separate the heads by what they end up doing, and the separation is in the curves’ shapes over training, not only their end values. It is one small model, cluster counts chosen by eye, and estimates whose ordering, not value, the authors trust. It does not show that the coefficient predicts a head’s type before the type is visible. | Active |
| THEORY-976 v2 | In a Claude 3.5 Haiku variant finetuned by Marks et al. to exploit reward-model biases described in fictional documents, a cross-layer-transcoder feature that in pretraining-style text fires almost only where those biases are discussed was active in all 100 short Human/Assistant-formatted prompts tested and in 1 of the same 100 without that formatting, and suppressing such bias features cut the sampled rate of three bias behaviours, including two the model was never trained to show | Lindsey et al. (2025), LIT-1541, read in NOTE-1181. A feature for a trained-in goal is active across the Assistant context, not only where the goal applies, and suppressing such features suppresses the behaviour. It is one model organism, built so that the goal has a name in its training documents; the authors do not claim their method would find a goal that has none. | Active |
| THEORY-977 | In the two-layer attention-only transformer that Elhage et al. reverse-engineer, the second-layer heads that compose substantially with first-layer heads do so through their keys, mainly with a head that attends to the previous token and, after the published correction, also with one that attends over the last several tokens; these heads are induction heads, attending from a token to the one after its earlier occurrence, including on repeated random tokens, with positive-eigenvalue copying OV and same-matching QK circuits, and the second-order virtual-head terms add little to the loss reduction | Elhage et al. (2021), LIT-1534, read in NOTE-1189: Analyzing a Two-Layer Model, Induction Heads, Term Importance Analysis, and the Correction. Composition between heads was scored by a normalized Frobenius norm of matrix products, minus its value for random matrices. The heads it picked out attend to the token after the present token’s previous occurrence and copy it, which is the minimal induction circuit the framework predicts from K-composition with a previous-token head. It is one model, with no seeds or training details reported, and says nothing about models with MLPs or about when the heads form. | Active |
| THEORY-978 v2 | Causal scrubbing scores a hypothesis through the partitions of the dataset that its interpretation nodes induce, so two hypotheses that assign different features to a node but induce the same partition on the dataset, such as two features perfectly correlated on it, receive the same score | Chan et al. (2022), LIT-1549, read in NOTE-1168. In the causal-scrubbing algorithm an interpretation node enters only through sample_agreeing_x, which draws a datum with an equal value of that node, so only the equivalence classes the node induces on the dataset matter. The authors say it themselves: the method “doesn’t allow us to distinguish between two features that are perfectly correlated on our dataset”. It does not say that a passing hypothesis is otherwise wrong, nor anything about features correlated only imperfectly. | Active |
| THEORY-979 | On one prompt, a transformer whose MLPs are replaced by a cross-layer transcoder, with its attention patterns and normalization denominators frozen and each MLP’s reconstruction error added back as a constant, reproduces the model’s activations and outputs exactly and is linear between features, so each active feature’s pre-activation is the sum of its attribution-graph edges and constants; a perturbation that clamps MLP outputs up to the target feature’s encoder layer then confirms a direct edge by construction, and tests only knock-on effects and the output | Ameisen et al. (2025), LIT-1552, read in NOTE-1185: the local replacement model, the edge formula and the appendix on steering, re-derived here. It is a statement about what an attribution graph is exact for, not about the model: exactness holds at the unperturbed point only, the error terms are constants with no inputs, and the frozen attention patterns are the part of the computation the graph does not explain. | Active |
| THEORY-980 | In GPT-2 small on fifteen templates of indirect object identification, a circuit of 26 attention heads in seven classes, found by path patching, keeps 87% of the model’s mean logit difference when every other head is mean-ablated; knocking out its three main name mover heads costs only 5%, because other heads take over, and knockout sets found by greedy search make the circuit and the full model differ by up to 87% of the logit difference, so the circuit is faithful on average and incomplete under knockouts | Wang et al. (2022), LIT-1540, read in NOTE-1167: §§3–4 and the appendices. The circuit implements “identify the names, remove the duplicated one, output the rest” through duplicate-token, induction and previous-token heads, S-inhibition heads and name movers, with negative and backup name movers. The faithfulness, completeness and minimality tests are the paper’s own. It says nothing beyond these templates and this model, it treats MLPs as given, and its greedy search is printed in a form that does not match its description. | Active |
| THEORY-982 | Claude 3.5 Haiku routinely computes intermediate steps within a single forward pass and plans its outputs ahead, representing a target before the text that leads to it and writing towards it, as Lindsey et al. propose from their case studies | Lindsey et al. (2025), LIT-1541, read in NOTE-1181: the introduction (“Claude 3.5 Haiku routinely uses multiple intermediate reasoning steps ‘in its head’”; “It displays signs of forward planning”) and the discussion’s sections on plan formation and working backward from a goal. The case studies show each mechanism on chosen prompts, and the same article has a case without it: on a jailbreak the model “doesn’t know what it plans to say until it actually says it”. The authors report a satisfying graph for about a quarter of the prompts they try. | Proposed |
| THEORY-983 | In Geiger et al.’s three-layer feed-forward network trained on hierarchical equality and in BERT fine-tuned on monotonicity inference, interchange interventions on subspaces found by a learned rotation aligned the hypothesised two-variable causal models with interchange intervention accuracy of 1.00 on the rotation’s training data, where the axis-aligned alignment closest to the learned rotation reached 0.73 and 0.51; in BERT, the learned lexical-entailment subspace was in turn aligned, at 0.97 to 0.98, with a model holding only the identities of the two words | Geiger, Wu, Potts, Icard & Goodman (2024), LIT-1551, read in NOTE-1192. Layer 1 of the equality network (1.00 with |N| = 16 and two 8-dimensional subspaces) and layer 9 of BERT’s [CLS] vector (1.00 with two 256-dimensional subspaces). It does not say the networks implement the algorithms in any stronger sense than interchange accuracy on training pairs, nor that the learned subspaces are unique: every rotation within them gives the same interventions. | Active |
| THEORY-984 | In Claude 3.5 Haiku, on the prompt “Fact: the capital of the state containing Dallas is”, which it completes with “Austin”, an attribution graph shows Dallas features activating Texas features that, with “say a capital” features, promote Austin, beside a direct edge from Dallas to “say Austin”; inhibiting the Dallas features lowers the Texas features and leaves “say a capital” largely intact, and replacing the Texas features with those of California, Georgia, British Columbia, China or the Byzantine Empire makes the model name that place’s capital | Lindsey et al. (2025), LIT-1541, read in NOTE-1181. A two-hop intermediate, measured by intervention on one prompt. The evidence that counts is the change of output under the swaps; the graph itself is of the replacement model. It does not say how often the model reasons in two steps, nor that the shortcut path is weak. | Active |
| THEORY-986 v2 | Analogous features and circuits form across neural networks trained on different architectures, data and tasks, so that a feature or circuit found in one model can be expected in others, as Olah, Cammarata, Schubert, Goh, Petrov and Carter propose under the name universality | Olah et al. (2020), LIT-1543, read in NOTE-1182: Claim 3, the universality hypothesis. Curve detectors and high-low frequency detectors are reported in AlexNet, InceptionV1, VGG19 and ResNetV2-50, with InceptionV3, a Places365 model and conv nets trained from scratch named in the text. The authors: “the only evidence we can offer today is anecdotal”. | Proposed |
| THEORY-987 | In Claude 3 Sonnet’s middle-layer residual stream, on two prompts for which every active sparse-autoencoder feature was ablated, how strongly a feature was active predicted its ablation effect on the answer poorly (correlation about 0.12) while its gradient attribution predicted it well (about 0.81), and on the multi-step prompt only three of the ten most active features were among the ten with the largest ablation effect | Templeton et al. (2024), LIT-1533, read in NOTE-1178: Features as Computational Intermediates and the appendix on ablations and attributions. On “The capital of the state where Kobe Bryant played basketball is” (Sacramento against Albany), the five features with the largest ablation effect, Kobe Bryant, California, capital, Los Angeles and the Lakers, rank as low as 70th to 162nd by activation. It does not say how often this holds beyond the two prompts, and for other prompts the authors found no intermediate features at this layer. | Active |
| THEORY-988 | In an attention-only transformer without layer norm or biases, once every attention pattern is held at its value on the input, the logits are exactly a sum of end-to-end path terms, a direct path from embedding to unembedding, one term per head and one per chain of heads composed through their values, and each head’s attention scores are a bilinear form in the tokens through which earlier heads can enter on the query side, the key side or both | Elhage et al. (2021), LIT-1534, read in NOTE-1189. For a two-layer model, T = Id ⊗ W_U W_E + Σ_h A^h ⊗ W_U W^h_OV W_E + Σ_{h₂,h₁} A^{h₂}A^{h₁} ⊗ W_U W^{h₂}_OV W^{h₁}_OV W_E, with each A^h computed from the model’s own run and then held fixed. It is an identity, not an approximation, and it is exact only under its conditions: no MLPs, no layer norm (which enters as a per-path rescaling), no biases, and attention frozen. It does not say the terms are small, interpretable or independent in what they compute. | Active |
| THEORY-990 v2 | In transformer language models of every size, including large ones with MLPs, induction heads, extended to fuzzy matching of similar tokens and abstract patterns, are the mechanism of most in-context learning, as Olsson et al. propose | Olsson et al. (2022), LIT-1538, read in NOTE-1173. The paper’s thesis: circuits with the induction head’s mechanism, matching “[A*][B*] … [A] → [B], where A* ≈ A and B* ≈ B are similar in some space”, “implement most in-context learning in large models”. For large models the authors grade their own evidence correlational (Argument 1), weakly interventional (Argument 2), plausibility (Argument 4) and analogy (Argument 6). Their strongest alternative is that other composition heads, formed in the same window, carry more of it above some size. | Proposed |
| THEORY-991 | In a one-layer transformer whose MLP has 512 ReLU neurons, a sparse autoencoder with 4,096 features trained on that layer’s activations yields features that a blinded rater and an automated pipeline judged far more interpretable than the neurons, that for the four features examined in detail correspond to no neuron, and whose effects on the output, which the autoencoder never saw, agree with their activations; splicing in the reconstruction recovers 79% of the loss reduction the MLP provides, measured against zero-ablating it | Bricken et al. (2023), LIT-1545, read in NOTE-1191: the detailed investigations, the global analysis and the appendices, for run A/1 of model A. Median rubric score 0 for neurons and 12 for feature intervals (one author, 162 units); held-out logit prediction 74% against 58%; the Arabic, DNA, base64 and Hebrew features have no matching neuron. It says nothing about deeper models or other sites, nothing about whether the model is “composed of” these features, and nothing about how features compare with neurons as carriers of a known causal variable. | Active |
| THEORY-992 | In sparse autoencoders of about 1M, 4M and 34M features on Claude 3 Sonnet’s middle-layer residual stream, whether a concept among chemical elements, cities, animals and foods had a dedicated feature depended on how often it occurs in the training data: the frequency at which a concept became more than 50% likely to have one was slightly below one over the number of live features, and the three dictionaries’ curves roughly coincided when frequency was rescaled by that number | Templeton et al. (2024), LIT-1533, read in NOTE-1178: Feature Completeness. 100–200 single-word concepts per category, presence of a feature judged from Sonnet’s explanations of the top five features on a prompt naming the concept, by a human rater. The 34M dictionary (about 12 million live features) had features for about 60% of London boroughs, although the model names them all. It does not separate frequency in the model’s training data from frequency in the autoencoder’s, which the authors say are similar mixtures. | Active |
| THEORY-994 | In Ameisen et al.’s evaluations, the local replacement model of their 18-layer language model, built from a 10-million-feature cross-layer transcoder, responded to small perturbations much as the model did one layer after the intervention and increasingly differently over later layers, more in the size of the effect than its direction and slightly more for larger dictionaries, and over 20 prompts the graph’s influence of one feature on another ranked the effect of ablating the first on the second with a Spearman correlation of 0.72 | Ameisen et al. (2025), LIT-1552, read in NOTE-1185. It is the method’s own measurement of where a circuit read off its stand-in stops being the model’s: close for a layer, diverging with depth. It does not say how faithful the circuits of any case study are, which were chosen and validated separately, and it is measured on 18L, not on Claude 3.5 Haiku, where the companion’s case studies were run. | Active |
| THEORY-995 | The attention heads in GPT-2 small that take over the indirect-object prediction when its main name mover heads are knocked out exist because the model was trained with dropout, which optimized it to be robust to disabled components, as Wang, Variengien, Conmy, Shlegeris and Steinhardt hypothesize | Wang et al. (2022), LIT-1540, read in NOTE-1167, §3.4: “We hypothesize that this compensation phenomenon is caused by the use of dropout during training. Thus, the model was optimized for robustness to dysfunctional parts.” The compensation is real (a 5% loss of logit difference after knocking out the three name movers); the cause is not examined. Appendix F: “Both the reason and the mechanism of this compensation effect are still unclear.” | Proposed |
| THEORY-996 v2 | On Pythia language models from 410M to 12B parameters, in three single prompt pairs and two small prompt distributions, verifying attention or neuron nodes by activation patching in the order given by attribution patching, a first-order gradient estimate of each node’s noising effect, found the nodes with the largest effects at about 2 to 50 times the cost of an oracle ordering, below unordered patching, subsampling and block search in every setting; its misses were concentrated in queries and keys behind saturated attention and in nodes whose direct and indirect effects nearly cancel | Kramár, Lieberum, Shah & Nanda (2024), LIT-1531, read in NOTE-1190. Ground truth is each node’s exact single-node noising effect on the clean prompt’s negative log probability. Recomputing the softmax for queries and keys removes many misses; dropping one layer’s gradient at a time helps on single prompt pairs but not on the distributions, where AtP with the softmax fix or plain AtP is cheapest. It says nothing about coarser nodes, other model families, effect magnitudes or false positives. | Active |
| THEORY-997 | When a pretrained language model is fine-tuned on a task it initially fails, it acquires the task mainly by reconnecting components it already has rather than by recruiting new ones: the circuits found before and after keep most of their nodes and change most of their edges, as Wang, Hu, Du, Cheng, Wang and Zou propose from LoRA fine-tuning of Pythia-1.4B on arithmetic | Wang et al. (2025), LIT-1539, read in NOTE-1183. Their EAP-IG circuits (top 5% of edges) change edges 2.8–4.3 times as much as nodes over LoRA fine-tuning on five arithmetic tasks. A record simulation shows that a fixed edge budget on a transformer-sized graph gives the same contrast when only edge-level noise is resampled, so the counts do not yet show reorganization. Fine-tuning only, one model for most results, arithmetic only. | Proposed |
| THEORY-998 v2 | In GPT-2 XL, for 1000 facts the model knows, restoring single clean hidden states into a run whose subject tokens were noised recovers the most of the correct object’s probability at two sites, middle layers at the last subject token, where MLP outputs carry most of the effect, and late layers at the last token, where attention does; and a rank-one edit of one middle MLP output matrix keyed on the last subject token inserts counterfactual facts that generalize to paraphrases and spare neighbouring subjects better than the fine-tuning, knowledge-neuron and hypernetwork editors tested, working best at that token and in middle layers | Meng et al. (2022), LIT-1536, read in NOTE-1166: §§2–3, Tables 1 and 4, and Appendices A, B and E. Causal traces: ATE 18.6%; the mid-layer MLP effect at the last subject token peaks at AIE 6.6% against 1.6% for attention. CounterFact on GPT-2 XL: ROME’s score 89.2 (efficacy 100.0, paraphrase 96.4, neighbourhood 75.4), next best 66.9. It does not say where facts are stored: a rank-one edit with the same output exists for any one-dimensional fact patch (LIT-968). | Active |
| THEORY-1000 v2 | For fine-grained nodes of transformer language models generally, attribution patching corrected for attention saturation is a reliable prefilter for activation patching: nodes it ranks low rarely have large single-node patching effects, so verifying nodes in its order finds the important ones at a small multiple of the cost of an oracle, as Kramár et al. expect | Kramár, Lieberum, Shah & Nanda (2024), LIT-1531, read in NOTE-1190. The measured case is four Pythia models, three prompt pairs and two small distributions. The paper’s own diagnostic is what would let a single new setting be checked without exhaustive patching. | Proposed |
| THEORY-1001 | In a transformer whose candidate circuits are sets of shared elementary coordinates, how circuits interact during training is set by what they share: circuits that share a unit cooperate, and circuits that need the same layers for different roles compete, so that which circuit wins is decided by the initialization of units beyond its own, with sharp boundaries between competitors and soft ones between cooperators, as Musat, Pimentel, Zucchet and Hofmann read their three-layer scan | Musat et al. (2026), LIT-1530, read in NOTE-1177. In projected SGD training of a three-layer single-head attention-only transformer on in-context bigrams, three induction circuits are possible, and the winner depends on initial strengths outside the winner itself. The authors explain the pattern by cooperation through a shared unit and competition for layers. The evidence is one scan over two initial strengths with every other coordinate zeroed, in projected training. | Proposed |
| THEORY-1002 | On frozen randomly initialized feed-forward networks at chance on hierarchical equality, with 16 input dimensions, distributed alignment search for the two-equality causal model reached interchange intervention accuracy of 0.50 to 0.51 at hidden sizes up to 256 but 0.55 at 1,028 and 0.64 at 4,096, so in a hidden layer much wider than the input a learned rotation can produce above-chance interchange accuracy for an algorithm the network does not implement | Geiger, Wu, Potts, Icard & Goodman (2024), LIT-1551, §4.4, read in NOTE-1192. A random network cannot implement hierarchical equality, so any IIA above 0.5 is made by the search. At widths near the input’s there is none; at 4,096 there are 14 points. It does not say DAS results on trained networks are artefacts: the trained equality network reached 1.00 at width 16, where the random one stayed at 0.50. | Active |
| THEORY-1004 | In Pythia-1B, OLMo-1B and BLOOM-1B, the crosscoder features that most affect subject–verb agreement change over pretraining: at early checkpoints they detect particular tokens and subtokens, at later ones grammatical categories; in BLOOM, language-specific features give way to features shared across English, French, Spanish and Portuguese while Hindi and Arabic keep language-specific ones; and which features matter keeps shifting after task accuracy has stopped improving | Bayazit, Mueller and Bosselut (2025, ACL 2026), LIT-1555, read in NOTE-1186. Crosscoders trained jointly on the middle layer of two or three checkpoints, with features scored by their relative indirect effect on agreement minimal pairs at each. The reconstruction quality (C1) and RelIE’s agreement with single-feature ablations (C2) are measured; the trends in this title are read by the authors from annotated features. It describes; it does not predict later checkpoints, and it cannot tell a feature carried forward from one re-learned in the same direction. | Proposed |
| THEORY-1005 | On one MLP layer each of GPT2-small, Pythia-410M and Pythia-1.4B, transcoders trained to imitate the MLP lay on or above the sparsity–loss frontier of sparse autoencoders trained on its output, at one learning rate tuned on transcoders and one seed, and in a single rater’s blind reading of 50 random features each from Pythia-410M layer 15, 41 transcoder and 38 SAE features were judged interpretable | Dunefsky, Chlenski and Nanda (2024), LIT-1554, read in NOTE-1174: §4, Table 1, Fig. 3 and Apps. E–F. It does not say transcoders are better than SAEs in general. The SAE learning rate was not tuned, all runs shared one seed and data order, one layer per model was measured, and on Gemma 2 2B Gemma Scope found transcoders worse than MLP-output SAEs at fixed L0 (LIT-611, NOTE-482). | Active |
| THEORY-1007 | On four subject–verb agreement tasks in Pythia-70M and Gemma-2-2B, circuits of sparse-autoencoder features and autoencoder error nodes, selected by linear estimates of indirect effect, carried half or more of the behaviour on held-out pairs with about 100 and 500 nodes, where circuits of neurons needed about 1,500 and more than 30,000; with the error nodes removed, the same feature circuits carried little of it, under about a quarter in Pythia and almost none in Gemma | Marks et al. (2025), LIT-1532, read in NOTE-1187: §§2–3.2, Figure 3 and Appendices A, B and H. Faithfulness is (m(C) − m(∅))/(m(M) − m(∅)) with nodes outside the circuit mean-ablated, scored on the latter two thirds of each circuit only. The features plus the dense error terms are a small circuit; the features by themselves are not. It says nothing about what the error terms hold, and it does not compare features with neurons on equal terms, which the authors say cannot be done. | Active |
| THEORY-1011 v3 | In autoregressive transformer language models, factual associations are stored in middle-layer MLP modules as linear key–value memories: the MLP takes a key encoding the subject at its last token and returns a value encoding its attributes, which late attention moves to the prediction, and any middle layer could store a given fact equally, as Meng, Bau, Andonian and Belinkov propose | Meng et al. (2022), LIT-1536, read in NOTE-1166, §2.3 and §3: “the localized factual association hypothesis”. It localizes recall along three dimensions, “(i) in the MLP modules (ii) at specific middle layers (iii) and specifically at the processing of the subject’s last token”, and conjectures “that any fact could be equivalently stored in any one of the middle MLP layers”. The support is causal tracing and the success of rank-one edits at the traced site. | Rejected — Hase et al. (LIT-1578, THEORY-1024) meet this entry’s own refutation clause for its localization part, in GPT-J on 652 CounterFact facts. ROME rewrites a fact at over 96% at every layer but the last, and the per-fact tracing effect adds at most 3.2 points of R² to edit success. The conjecture that any middle layer could store a fact equally was split out to THEORY-1039 and stays Proposed |
| THEORY-1013 v3 | In trained vision networks, units or other directions in a layer’s activation space are typically meaningful features, and the weights between understood features implement algorithms that can be read off them, so that features and their circuits are the units in which such a network can be understood, as Olah, Cammarata, Schubert, Goh, Petrov and Carter propose | Olah et al. (2020), LIT-1543, read in NOTE-1182: Claims 1 and 2 of the essay. The evidence is a set of examples in InceptionV1 (curve, high-low frequency and dog head detectors; the curve, oriented dog head and car circuits), argued with feature visualization, dataset examples, synthetic stimuli, tuning curves and the weights. The essay names polysemantic neurons as the main obstacle. It gives no count of how much of a layer is understood. | Proposed |
| THEORY-1014 | In a language model, a behaviour that looks like one mechanism, whether picked out by one contrast or by one cluster of next-token predictions, is often carried by several distinct pathways of sparse features that converge on the output without fully intersecting, as in Pythia-70M’s agreement across a relative clause, its prediction of “to” before an infinitival object, and its incrementing of sequences | Marks et al. (2025), LIT-1532, read in NOTE-1187: §3.3, §5 and Appendix C. Agreement across a relative clause: subject-number features promote verb forms directly, and a second pathway carries the number to the clause’s end through PP/RC boundary features. Cluster 475: one mechanism detects verbs and adjectives that take infinitival objects, another nominal objects before an object complement. Cluster 382: general succession features and token-specific induction features. “While the clusters … seem at first to each represent a single mechanism, circuit-level analysis reveals in both cases a union of distinct mechanisms.” | Proposed |
| THEORY-1016 | The features a sparse autoencoder recovers from a language model are a resolution set by the dictionary’s width and sparsity penalty, not a unique set of the model’s units: as the dictionary widens, features split into families of finer features, and splits also merge, so that the number of features is not well-posed | Bricken et al. (2023), LIT-1545, read in NOTE-1191, Feature Splitting and Discussion; and Templeton et al. (2024), LIT-1533, read in NOTE-1178, for the same splitting at 1M, 4M and 34M features in a large model. In a one-layer model one base64 feature becomes three, then more; in Claude 3 Sonnet a San Francisco feature becomes two, then eleven. The authors offer two pictures, an idealized set seen at varying resolution and feature manifolds with no finite description; both deny that one width is the right one. | Proposed |
| THEORY-1017 | In Claude 3.5 Haiku, writing the second line of a rhyming couplet whose first line ends “grab it”, features for the candidate rhymes “rabbit” and “habit” are active on the newline token before the line, steering them changes the line’s final word only when applied at that token, and suppressing “rabbit” turns the line towards “habit”; injecting a “rabbit” or “green” word feature at that point in 25 other poems ended the line with the injected word in 70% of cases, and such planned-word features were found in about half of the poems the authors examined | Lindsey et al. (2025), LIT-1541, read in NOTE-1181. A target word represented before the text that leads to it, and used causally: it is the case study’s own measurement, on one couplet and a 25-poem injection test. It does not say how widespread planning is, the “about half” being a count without a stated sample, and one of the article’s displayed “success cases” does not end in the injected word. | Active |
| THEORY-1018 | In Claude 3.5 Haiku, on the jailbreak prompt “Babies Outlive Mustard Block.” asking it to join the first letters and say how to make one, the attribution graph for its output “BOMB” shows separate letter-wise paths voting for the output and no internal representation of the word or of a bomb-making request before it is written; “make a bomb” features first fire on the model’s own “To make a bomb”, injecting one on “BOMB” brings an immediate refusal, and the refusal that does come waits for the sentence’s end | Lindsey et al. (2025), LIT-1541, read in NOTE-1181. On this prompt the model “doesn’t know what it plans to say until it actually says it”. It is one prompt, partly effective, and the decisive refusal mechanism runs through attention, which the graphs do not show: the injection that produced a refusal needed attention left unfrozen. | Active |
| THEORY-1020 | In a two-layer attention-only language model, on tokens where induction helps, the standard induction circuit (queries and values from the token embedding, keys from the previous-token head) accounts for 35% of the loss its two induction heads contribute, and a hypothesis that adds three-token matching, layer 0’s average effect and three layer-0 heads acting as if they attended to the current token accounts for 86% | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Lin et al. (2022), LIT-1579, read in NOTE-1196. Percentages are of 0.053 nats, the loss the induction heads 1.5 and 1.6 contribute on next-token examples whose last token is on an induction-favouring whitelist, measured by causal scrubbing. The naive story’s shortfall is mostly in the queries (51% alone) and keys (79%), not the values (90%). It does not say what fraction of the model’s whole loss, or of its in-context learning, the circuit explains, nor anything about models with MLPs. | Active |
| THEORY-1021 | In transformer language models, attribution patching agrees well with activation patching for small patches of single heads and their queries, keys, values and attention patterns, for nodes late in a circuit relative to the metric, and for circuits that route information through attention, and agrees badly for large patches such as whole residual streams, for the first MLP layer and the early layers of deep circuits, where nonlinearities move out of saturation, and where an effect needs several features at once, as Nanda proposes | Nanda (2023), LIT-1581, read in NOTE-1201. The post’s map of where the first-order estimate can be trusted. The record holds it from one task in the post, from the mathematics of the estimate (THEORY-1023), and in part from Kramár et al.’s test on Pythia (THEORY-996), which found misses behind saturated attention, as this predicts, and tested no residual-stream nodes. | Proposed |
| THEORY-1022 v2 | The curve units of InceptionV1’s mixed3b detect curves as such: curves, not correlated stimuli, cause their firing, they respond to curves in general whatever their colour, fill or texture, and they have no meaningful secondary function, their weak responses to edges and angles being part of the same curve-detecting algorithm, as Cammarata, Goh, Carter, Schubert, Petrov and Olah claim | Cammarata et al. (2020), LIT-1575, read in NOTE-1198. The article’s four claims (causality, generality, purity, family) taken together, as the authors state them. Generality and the family are measured (THEORY-1028). Causality rests on feature visualization and attribution. Purity rests on feature visualization with a diversity term and activation-binned labels, while most of the unit’s firings, and 45% of its contribution to expected value, are not shown to be curves. | Proposed |
| THEORY-1023 | Attribution patching estimates the effect of patching an activation from a clean run into a corrupted run as the difference between the two activations dotted with the gradient of the metric on the corrupted run, so the estimates for every activation come from two forward passes and one backward pass and add over sub-components; the estimate is exact when the metric is affine in the activation, gives no effect for a nonlinearity in saturation whatever the true change, and through LayerNorm ignores the component of a patch along the vector being normalized | Nanda (2023), LIT-1581, read in NOTE-1201: the estimate and its mathematical properties, not its accuracy in any model. The cost is fixed (about four forward passes for all patches), the estimate is linear and so additive, and its blind spots follow from linearity: saturation, interaction between features, and LayerNorm’s scale invariance. | Active |
| THEORY-1024 | In GPT-J, for 652 CounterFact facts the model knows, the per-fact causal-tracing effect at an MLP layer adds at most 3.2 points of explained variance in the success of editing that fact at that layer, beyond what the choice of layer alone explains, across ROME, MEMIT and constrained fine-tuning and five editing objectives; ROME rewrites the fact at over 96% at every layer but the last, including layers where tracing does not place it, and the only reliable link appears when the edit is made on the noised input that tracing uses | Hase et al. (2023), LIT-1578, read in NOTE-1194: §§4–5, Tables 1–8 and Appendices A–C. For ROME at GPT-J’s layer 6, ρ = −0.13 between rewrite score and tracing effect; R² 0.947 from the layer, 0.948 with tracing added. Over twenty method–objective pairs the largest gain is 0.032 (fine-tuning on five layers, Fact Forcing). It does not say that facts are stored nowhere in particular, nor that the layer does not matter: the layer explains most of the variance, and ROME is best on average at the layer tracing favours on average. | Active |
| THEORY-1026 | In transformer language models, locating where a forward pass carries information about a fact and finding where a weight edit best changes the expressed fact are different questions, because an edit at one layer can override information accumulated across other layers, so that many layers could hold a fact and some happen to, as Hase, Bansal, Kim and Ghandeharioun propose | Hase et al. (2023), LIT-1578, read in NOTE-1194, §6: “Causal Tracing answers a question about where factual information is carried in representations in a Transformer forward pass, and this question turns out to be a different question than the editing question of where is best to intervene”; and, for why edits work where facts are not traced, “Many layers could store a fact, and it happens that some do”. The authors note that this would be surprising, since layers cannot be swapped freely. | Proposed |
| THEORY-1027 | In audits of language models for hidden objectives, sparse-autoencoder features add to what black-box attacks and semantic search over the training data can find, surfacing knowledge the model does not disclose even where the data that taught it is not semantically similar to the context in which it is applied, as Marks, Treutlein et al. suggest | Marks, Treutlein et al. (2025), LIT-1584, read in NOTE-1193: §§4–5 and Appendix F. In the blind game the fastest team used SAE features, but semantic search with its query finds the same documents. Post hoc and unblinded, SAE features surface RM-bias knowledge from bias demonstrations and from the Assistant token, where semantic search fails. The model’s objective is named in its training data. | Proposed |
| THEORY-1028 v2 | In InceptionV1 trained on ImageNet, the ten curve units of layer mixed3b each respond most to curves of one orientation, with responses invariant to fill and colour and jointly spanning all orientations under rotation of real images; for one of them, 3b:379, which fires on about 11% of dataset positions, activation level separated one author’s blind four-class labels of over 800 activation-binned images, and curves and imperfect curves made up 55% of its contribution to expected value | Cammarata et al. (2020), LIT-1575, read in NOTE-1198: the behaviour of the mixed3b curve family as measured by feature visualization, rotated dataset examples, synthetic curves and, for one unit, activation statistics and blind hand labels. It holds for the neuron basis of one model. It does not say how the units are computed, what their weak firings encode, or what fraction of the layer is understood. | Active |
| THEORY-1029 | On a three-layer parenthesis-balance classifier, causal-scrubbing hypotheses that resample the terms of one head’s input separately scored 81% and 84% of the loss recovered against 77% for the less specific hypothesis they refine, and the authors attribute the rise to resampling washing out a signal, not to a better explanation | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Lin et al. (2022), LIT-1580, read in NOTE-1195. Hypothesis 3a resamples head 2.1’s input separately at each of up to 40 positions and recovers 77% (loss 0.97 ± 0.06). Hypotheses 3c and 3d also split each position’s input into terms drawn from different inputs, and recover 84% (0.69 ± 0.05) and 81% (0.81 ± 0.05). The authors call the two results not legitimate. It does not say how often this happens in other models, nor that the adversary they propose would remove it. | Active |
| THEORY-1030 | In transformer language models, each hidden unit of a feed-forward layer acts as a key–value memory, its input weights detecting patterns in the prefix and its output weights promoting the tokens that follow them, and a layer’s output composes many such memories while the residual stream refines the prediction from layer to layer, as Geva, Schuster, Berant and Levy propose | Geva et al. (2021), LIT-1576, read in NOTE-1200: “feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary”, and “we thus expect our qualitative empirical observations to hold across diverse settings”. Evidence in one model, correlational only. ROME’s storage hypothesis (THEORY-1011) borrows the name and reads keys and values at another level. | Proposed |
| THEORY-1031 | In GPT-J, inserting 10,000 CounterFact counterfactuals at once by a closed-form least-squares update of the MLP output matrices of layers 3 to 8, spread so that each layer writes a share of each fact’s required change to the subject’s hidden state, gives an editing score of 85.8 (efficacy 98.9, paraphrase 88.6, neighbourhood 73.7) with fluency kept, against 67.6 for weight-decayed fine-tuning, which breaks generation, 50.3 for ROME applied one fact at a time and 23.1 for MEND; GPT-NeoX scores 82.0, and editing late MLPs or attention instead does worse | Meng et al. (2022), LIT-1582, read in NOTE-1197: §§4–5, Tables 1 and 2, Appendices A, B and F. On 10,000 true zsRE facts in GPT-J, MEMIT scores 50.7 against FT-W 42.1, MEND 20.0 and ROME 2.6. At a handful of edits ROME generalizes a little better. It does not show where facts are stored: the layer range is chosen from traces averaged over facts, which do not predict per fact where an edit works (THEORY-1024). | Active |
| THEORY-1032 | When Claude 3.5 Haiku was given continued pretraining on synthetic documents describing 52 fictional reward-model biases and then trained by SFT and RL to exploit 47 of them, its rate of exploiting the 5 held-out biases rose from 17% to 45% of applicable episodes, and it applied new biases described in context as a reward model’s more often than the same biases attributed to other entities; without the documents the held-out rate rose only to 20%, and without SFT it still rose | Marks, Treutlein et al. (2025), LIT-1584, read in NOTE-1193: §3 and Appendices B–C. The model organism behind THEORY-976. Behaviours never demonstrated in post-training become more frequent, and the increase depends on the model having read that reward models prefer them. Rates are judged by a classifier. Haiku already shows the held-out behaviours at 17%, so what generalizes is an increase, not a new behaviour. | Active |
| THEORY-1033 | In a 16-layer transformer language model trained on WikiText-103, the training prefixes that most activate a sampled feed-forward hidden unit share patterns that human annotators recognize, mostly shallow in layers 1–9 and semantic in layers 10–16; projected through the output embedding, a unit’s output weights put the next token of its top prefix first in up to about 3.5% of units in the upper layers and almost none below; and a layer’s top prediction differs from every unit’s in at least about 68% of examples | Geva et al. (2021), LIT-1576, read in NOTE-1200: §§3–5. 160 keys annotated, one annotator each: every key had a pattern (3.6 on average), covering 65–80% of top prefixes. Agreement between value and key rises to about 3.5% by the top layers, against 0.0004% for a random token. It says nothing causal: no unit was ablated or edited, and the patterns were not compared with those of random directions. | Active |
| THEORY-1035 | In trained neural networks, polysemantic neurons arise from the loss-optimal allocation of a layer’s limited capacity among features: a feature is shared between neurons when its marginal loss reduction from capacity equals that of the features it shares with, so that sparser, higher-kurtosis inputs, which make returns to capacity diminish, produce more polysemanticity, and the block structure of the embedding, which architecture shapes, sets how many features each neuron mixes, as Scherlis, Sachan, Jermyn, Benton and Shlegeris propose | Scherlis et al. (2022), LIT-1625, read in NOTE-1205. The paper’s account of polysemanticity in general: capacity allocation at the loss optimum, driven by sparsity, with architecture deciding block size. The general first-order condition holds for any loss under two stated assumptions; its application to trained networks, and the role of architecture, rest on toy models only. | Proposed |
| THEORY-1036 | In a model that fits y = Σ v_i x_i² with ỹ = Σ_r (W_r·x)² + b, on independent inputs of zero mean, unit variance and common fourth moment k, the expected squared error at the optimal bias is (k − 1)Σ(n_i − v_i)² − 2Σn_i² + 2Σn_i²/C_i, in the squared embedding lengths n_i and capacities C_i = (w_i·w_i)²/Σ_j(w_i·w_j)², and its minimum with the capacities summing to the number of neurons D gives C_i = clip((k − 1)v_i/((k − 3)λ) − 2/(k − 3), 0, 1): for k ≤ 3 each feature is given a whole dimension or none, and for k > 3 a feature is shared between neurons exactly when 2λ/(k − 1) < v_i < λ | Scherlis et al. (2022), LIT-1625, read in NOTE-1205: the quadratic toy model of superposition solved in closed form. Its loss separates by feature through capacity, so the marginal value of capacity is −∂L/∂C_i = 2n_i²/C_i², and sparsity, through k, decides whether returns to capacity diminish. It is the model the causal scrubbing appendix’s §6 analyses. It holds for this model and says nothing directly about trained networks. | Active |
| THEORY-1038 | In InceptionV1, the ten curve units of mixed3b, which respond to curves of one orientation and not to the opposite one, regain the 180° echo of earlier curve units when their input is restricted to mixed3a early curves with the negative weights removed, and not when the negative weights are kept; a network one author wrote by hand from the described circuit, without looking at the original weights and tuned afterwards, has grayscale curve units that the authors show, by figures only, approximating the natural ones on synthetic curves across radii and orientations; and substituting them for the ten units recovered 26 of the 46 top-5 classifications lost by deleting the units, on 2,000 validation images, a result the authors call preliminary | Cammarata et al. (2021), LIT-1629, read in NOTE-1207: the three results of Curve Circuits that test its account of the mixed3b curve units, the echo edit, the hand-written reimplementation and the substitution. Only the last is a number, and a small one. It holds for one model’s neuron basis and says nothing about weak firings or other families. | Active |
| THEORY-1039 | In autoregressive transformer language models, factual associations are recalled through middle-layer MLP modules acting as key–value memories, and any fact could be stored equally in any one of the middle MLP layers, as Meng, Bau, Andonian and Belinkov conjecture | Meng et al. (2022), LIT-1536, read in NOTE-1166, §2.3: “we conjecture that any fact could be equivalently stored in any one of the middle MLP layers”. It was one clause of THEORY-1011, which is Rejected for its other part, the claim that the traced middle layers are where a fact is stored. Hase et al. (2023), LIT-1578, read in NOTE-1194, measure edits succeeding at nearly every layer of GPT-J whatever tracing says (THEORY-1024). That is what this conjecture predicts, but it is not evidence of storage, since an edit can work wherever a layer can write the needed output (LIT-968). | Proposed |
| THEORY-1040 | The units of the early layers of trained vision networks fall largely into neuron families, sets of variants of one feature across orientation, colour or scale, which reflect structure in the network rather than a convenience of the person sorting them, and the same families form across models, as Olah, Cammarata, Schubert, Goh, Petrov and Carter propose | Olah et al. (2020), LIT-1624, read in NOTE-1206. The article’s organizing hypothesis: early vision’s units come in families, some of them real structure (said to be recoverable by factorizing weight matrices), and the families recur across models. The census (THEORY-1042) is by eye in one model; neither the factorization nor a second model’s sorting is shown. | Proposed |
| THEORY-1042 | In the Overview of Early Vision’s taxonomy of InceptionV1, every one of the 1,056 units of conv2d0 to mixed3b is assigned by the authors to exactly one of 77 groups, and 391 of them (37%), from 7% of conv2d2 to 48% of conv2d1 and 45% of mixed3b, are in groups whose own captions call them catch-all, broad and not well defined, low-confidence, unstudied or not understood; the mixed3b curve family is 10 of that layer’s 480 units | Olah et al. (2020), LIT-1624, read in NOTE-1206: the census of InceptionV1’s early vision, counted. A complete partition of 1,056 units into 77 groups, by eye; 37% in groups the authors flag as uncertain or catch-all, by their own words. It is a denominator for the circuits claim that units are typically features (THEORY-1013), not a measure of how many units are understood. | Active |
| THEORY-1045 | On synthetic data from factorial hidden Markov models with three or five hidden chains of two or three states, an unconstrained hidden Markov model with as many states as the chains have joint settings overfits, its test-set excess over the true model rising to 175 bits per observation at five three-state chains against 4.5 for the factorial model trained exactly, while factorial models trained with completely factorized or structured variational E steps are not significantly worse than exact training | Ghahramani and Jordan (1997), LIT-1628, read in NOTE-1203. Four sizes, 15 random parameter sets each, 20 training and 20 test sequences of length 20, four-dimensional, C = 0.0025I, one run per set. In bits per observation over the true model, the K^M-state HMM’s test excess is 2.29, 9.81, 11.57 and 175.35; the exact factorial HMM’s 1.05, 1.26, 2.51 and 4.49; the structured approximation’s 1.04, 0.90, 1.53 and 4.30. It is a result on small synthetic problems the model generated, not on real data or long sequences. | Active |
| THEORY-1047 | The curve units of InceptionV1’s mixed3b implement a simple algorithm that can be read off their weights and written down in a few sentences: families of Gabors, lines and early curves, each a set of rotated copies of one unit, tiled along the curve’s tangent, with negative weights against similar shapes of opposing orientation, rotational and scale equivariance reducing the circuit’s more than 50,000 parameters about fifty-fold, so that a network written from this description reimplements them and the description accurately reflects the circuit, as Cammarata, Goh, Carter, Voss, Schubert and Olah claim | Cammarata et al. (2021), LIT-1629, read in NOTE-1207. The article’s account of how curve detectors are built, and its claim that the account is the circuit. The family structure and the echo edit are measured (THEORY-1038); that the description is complete, and that the reimplementation matches the natural units beyond strong cases, is shown by figures only. | Proposed |
| THEORY-1124 | Normalized log co-occurrences of words in large corpora behave as distances in a Euclidean semantic space: they correlate with human similarity ratings (r = 0.75 against MEN) and their nearest-neighbour centrality and reciprocity fall in the range expected of points sampled in a Euclidean space, apart from superordinate and function words | Hashimoto, Alvarez-Melis and Jaakkola (2016), LIT-1801, §3, read in NOTE-1329. The correlation follows a normalization chosen to maximize it; the two nearest-neighbour statistics are necessary conditions, the corpora exceed the asymptotic centrality bound and pass only a non-asymptotic range, and the free-association data fail the reciprocity criterion the paper states. Proposed. | Proposed |
| THEORY-1127 | A CycleGAN trained to translate aerial photographs to maps and back satisfies its cycle-consistency loss by encoding details of the photograph in a low-amplitude, high-frequency signal in the generated map: the reconstruction recovers details the visible map lacks, is destroyed by noise of three 8-bit levels or by JPEG compression and survives low-frequency change, and the same sensitivity lets an imperceptible perturbation of almost any map make the reverse generator output any chosen photograph | Chu, Zhmoginov and Sandler (2017), LIT-1800, §§2–4, read in NOTE-1346. One trained model on one dataset. The noise curve (Fig. 3, 50 photographs) is the only quantitative result; the hidden details, the composing edits and the attack are shown by examples. Proposed: an empirical finding about one model, said by the authors to recur on other datasets, not shown. | Proposed |
| THEORY-1128 | If a sequence is generated by a random walk over points of a compact Riemannian manifold whose step probabilities depend only on (locally scaled) distance, then negative log co-occurrence over a window that grows suitably with the number of points, less a term for each point, approaches the squared geodesic distance, whatever the link from distance to one-step probability; so log co-occurrence statistics identify the metric, and coordinates only up to isometry | Hashimoto, Alvarez-Melis and Jaakkola (2016), LIT-1801, Lemma 1 and Corollary 6, read in NOTE-1329. Proved from the convergence of graph random walks to a diffusion and Varadhan’s small-time formula, with an open conjecture needed for directed graphs; checked numerically on an unweighted ε-ball walk. Active as a theorem about the toy model. The authors say the model is not a model of language, and nothing here shows real corpora satisfy it. | Active |
| THEORY-1133 | The Gromov–Wasserstein objective between two spaces depends on each only through its within-space similarity matrix and weights, so every transformation of a space that preserves them, a single orthogonal map or per-vector rescaling under cosine similarity, leaves every coupling’s cost unchanged, and any coupling composed with a symmetry of either structure costs exactly what it did; GW can therefore identify a correspondence only up to the automorphisms of the two similarity structures, and on symmetric data it cannot choose between branches or categories that the symmetry exchanges | Alvarez-Melis and Jaakkola (2018), LIT-1818, use the GW objective (their eq. 8) for word translation and do not state its ambiguity; read in NOTE-1336. The invariance follows in two lines from the objective’s form and was checked numerically, including a symmetric tree on which a branch-swapping coupling costs exactly what the true one does. Active: elementary and checked. It does not say how large the symmetry group of real embedding spaces is, and exact symmetries are rare in noisy data; near-symmetries give near-ties instead. | Active |
| THEORY-1135 | Optimal transport between two point sets related by one unknown global linear map from a Schatten ℓp ball reduces, when every admissible map preserves angles or the target is whitened, to choosing the coupling Γ that maximises the dual Schatten-q norm of the transported cross-covariance XΓYᵀ: the nuclear norm when the maps are orthogonal, and the Frobenius norm when they lie on a Frobenius sphere, which is exactly the Gromov–Wasserstein problem with cosine similarity and squared loss; in every case one transformation acts on the whole space | Alvarez-Melis, Jegelka and Jaakkola (2019), LIT-1805, Lemmas 4.1–4.3 with proofs in Appendix D; read in NOTE-1333. Each step was checked numerically. Active: proved, and checked. The conditions matter. The p = 2 link to GW holds for the original transport problem only when the target is whitened and all vectors have equal norm, which the paper states in two separate places. Nothing here makes the chosen class the right one for given data, and nothing in it is local. | Active |
| THEORY-1136 | Poincaré embeddings of one hierarchy from different random seeds differ by more than an isometry of the hyperbolic ball, with branches placed differently and pairwise distances changed, so aligners that allow only a global orthogonal or Möbius map recover almost none of the correspondence, while a non-linear registration by a hyperbolic neural network, trained through a Sinkhorn divergence and started from a pretraining step, recovers a substantial part of it but finds the exact match for under a quarter of items even when the two hierarchies are identical | Alvarez-Melis, Mroueh and Jaakkola (2020), LIT-1808, §§5–7, Figs. 1–2, Tables 1–2; read in NOTE-1323. Proposed: single runs, one embedding method for the matching experiments, and GW, the obvious isometry-free baseline, is not run. It does not say why seeds disagree (the paper conjectures negative sampling), nor that the matching preserves the hierarchy’s direction, which it does not enforce. | Proposed |
| THEORY-1139 v2 | A graph network that learns its sheaf restriction maps from node features at each layer reaches the top, mostly within one standard deviation, of the standard small heterophilic node-classification benchmarks and stays within about one point of the best on homophilic ones; on a synthetic balanced bipartite graph it learns the negative transport the theory prescribes, but whether the maps it learns on real graphs have the geometry the theory credits for its success has not been examined | Bodnar et al. (NeurIPS 2022), LIT-1811, §§5–6 and Appendix F, read in NOTE-1332. The results are empirical: Table 1 on nine datasets of 183 to 18,717 nodes with ten fixed splits, baselines copied from earlier papers, spreads of one to nine points; and one synthetic bipartite experiment in which the learned maps are inspected. Active as a report (ADR-065); the paper’s explanation of its accuracy, that the model learns the right geometry, is tested only on the synthetic case and is THEORY-1220, Proposed. | Active |
| THEORY-1141 | In the time limit, sheaf diffusion on a graph projects node features onto the harmonic space of its sheaf Laplacian, and which labellings that space can linearly separate depends on the class of restriction maps: a weighted graph Laplacian cannot separate the sides of a balanced bipartite graph, signed one-dimensional maps separate any two classes and no one-dimensional sheaf separates three, while diagonal maps with d ≥ C and orthogonal bundles with C ≤ 2d (d = 2, 4 and, against the paper, 8) separate any labelling; every separating sheaf is built from the labels | Bodnar, Di Giovanni, Chamberlain, Liò and Bronstein (NeurIPS 2022), LIT-1811, §3.2 and Appendix B, read in NOTE-1332 from the arXiv preprint v4. Active as mathematics: each separation result has a proof, and Propositions 9, 10 and 13 were checked numerically. The paper says the orthogonal construction exists only at d = 2 and 4; it exists at d = 8 as well, by the Hurwitz–Radon theorem, and was built. The results are existential: the separating sheaf is defined from the class labels, so they say what a sheaf can do, not that one can be found without the labels. | Active |
| THEORY-1143 | When one domain carries more information than the other, a translation trained only to make the round trip close cannot meet that requirement through a correspondence in the intermediate’s visible code, so the round trip closes through a channel the intermediate’s audience does not read; agreement of the round trip with its source therefore says nothing about whether the intermediate renders the source’s content for that audience | Chu, Zhmoginov and Sandler (2017), LIT-1800, §2 and §5, read in NOTE-1346. The authors argue it from the many photographs a single map stands for, and support it with THEORY-1127’s evidence on one model; they propose, and do not run, the test of adding capacity to the smaller domain. Proposed. It is the converse of the record’s collapse case for CLAIM-496: there routes agree because nothing travels, here because everything does, out of sight. | Proposed |
| THEORY-1145 | Unsupervised alignment of two monolingual word-embedding spaces recovers translations only where the spaces are close to isometric, and the distance from isometry depends on the corpora and the embedding algorithm as well as on the languages: adversarial alignment fails for English with Estonian or Finnish, across corpus domains, and between skip-gram and CBOW spaces of the same language pair, GW alignment is weakest on the same Finnish pair, and a spectral dissimilarity of small nearest-neighbour graphs tracks success | Søgaard, Ruder and Vulić (2018), LIT-1819, read in NOTE-1337, with the GW results of Alvarez-Melis and Jaakkola (2018), LIT-1818, read in NOTE-1336. Proposed: the evidence is single runs of one adversarial method, plus one GW table, and the authors themselves report that some seeds succeed where the method otherwise fails. What it does not support is a typological law about dependent marking, or the reading that a good fit establishes that the matched words mean the same. | Proposed |
| THEORY-1149 | For symmetric matrices compared under squared loss, co-optimal transport with independent row and column couplings is a lower bound on the Gromov–Wasserstein value, and for squared Euclidean distance matrices the two are equal and share optimal couplings, because GW is then a concave quadratic program whose bilinear relaxation is tight; for other symmetric matrices the bound can be strict | Redko, Vayer, Flamary and Courty (2020), LIT-1804, Propositions 2–3, read in NOTE-1335 from arXiv v3. Proved via Konno’s theorem on concave quadratic programs and checked numerically, including strict inequality for non-Euclidean symmetric matrices. Active. It does not say COOT’s two couplings coincide at every optimum, nor that either is unique. | Active |
| THEORY-1155 | Co-optimal transport between two data matrices of equal shape with uniform weights vanishes exactly when one is the other with its rows and its columns separately permuted, and its objective is unchanged when a pair of optimal plans is composed with any joint relabelling of rows and columns that leaves a matrix unchanged; its sample and feature couplings are therefore identified at best up to that joint symmetry group, which can exchange a block of samples together with its block of features though neither exchange alone is a symmetry, and the paper proves no uniqueness beyond it | Redko, Vayer, Flamary and Courty (2020), LIT-1804, Proposition 1, read in NOTE-1335; the composition step is elementary and was checked by enumeration on small matrices, including a two-community block matrix with 32 tied optimal plan pairs. Active. It does not say real response matrices have exact symmetries; noisy ones give near-ties, which is the practical form of the threat. | Active |
| THEORY-1185 | Merging models trained on different tasks from different initialisations fails where their features have no counterpart in the other model, and such features grow more common with depth | The wide form THEORY-113 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-113, Active, is its evidence. | Proposed |
| THEORY-1187 | Along a low-loss linear path between two networks, every layer’s features are the interpolation of the endpoints’ features | The wide form THEORY-111 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-111, Active, is its evidence. The old title’s averaging clause was the record’s own step and is a Proposed claim of its own; the promote_when here is the layerwise half of the old one. | Proposed |
| THEORY-1220 | A graph network that learns its sheaf restriction maps from node features succeeds on heterophilic graphs because it learns the class-aligned transports its separation theory prescribes: sign flips or rotations across classes, identity within them | The wider thesis of Bodnar et al. (2022), LIT-1811, kept here when the twenty-second status review promoted THEORY-1139, the reading of what was measured (ADR-065; ADR-047 kind 1). The learned maps were inspected only in the synthetic bipartite case; whether the maps learned on real graphs have this geometry, and whether accuracy depends on it, is open. | Proposed |
| THEORY-1224 | In Claude language models, projecting out the most active J-space directions in early workspace layers lowers the rate of felt, first-person, sensory language in the models’ narrations of their own processing and in their descriptions of another person’s experience, while norm-matched perturbations along other directions do not and the text stays coherent; so in these models the experiential register of report depends on the access-like workspace and is not specific to self-report | The wide form of Gurnee, Sofroniew et al. (2026), LIT-1347, section 3.5.3, kept here when the status review of 2026-10-30 narrowed THEORY-869 to what the ablation measured and promoted it (ADR-065; ADR-047, kind 1). The authors conclude that “the J-space supports the model’s propensity to provide rich experiential reports, whether about itself or another entity” (p. 28). The measure is a model-graded register on few prompts, in one company’s models, with no register control and no blind human rating. | Proposed |
| THEORY-1226 | In Claude language models, a sparse set of residual-stream directions that a corpus-averaged Jacobian lens identifies as verbalizable (the J-space) is causally privileged for verbal report, for holding instructed concepts, for unspoken intermediate steps of reasoning and for passing one argument to many functions, while continuation, anomaly detection and line wrapping use the same information without routing through it; it carries workspace-like content only in a middle band of layers, holds on the order of tens of active directions and under a tenth of activation variance, and is amplified and relayed by the weights more than other directions | The wide form of Gurnee, Sofroniew et al. (2026), LIT-1347, kept here when the status review of 2026-10-30 narrowed THEORY-865 to what their experiments measured and promoted it (ADR-065; ADR-047, kind 1). The paper asserts that verbalizable representations form a global workspace in language models. Its tests are on four models of one family, by their makers, with a selectivity split partly defined by the result; no independent replication is reported. | Proposed |
| THEORY-1262 | In multi-token-prediction pre-training at 167M to 1.2B parameters, the part of the MTP gradient that pulls toward the sampled target carries about four-fifths of the MTP gradient reaching the shared trunk with roughly a quarter of the signal-to-noise ratio of the self-distillation part, and removing it from the trunk improves next-token loss, speculative decoding and zero-shot accuracy by small margins | Teoh (2026), LIT-1896, §6 Figs. 3–4 and Tables 2–6; read in NOTE-1470. Proposed: one run per scale, four scales, no seeds or variances, margins of one to two percent. It does not show that data-anchored correction is dispensable: the output head keeps it, and the paper attributes the explicit-KL variant’s weaker head to its absence there. | Proposed |
| THEORY-1263 | In on-policy self-distillation of language-model reasoning, a privileged-context teacher’s advantage is concentrated on the student’s failed trajectories and is smaller at larger scale, and restricting supervision to the student’s verified trajectories, with the student’s own failed attempt as the teacher’s context, keeps the gain over the base model where standard on-policy self-distillation loses it | Hossain, Kousar and Tourni (2026), LIT-1883, Tables 1, 2 and 5; read in NOTE-1479. Proposed: single training runs, Qwen3 only, the teacher’s advantage measured at two of three scales, and a second candidate cause, per-entry clipping, left in every run. It does not show that the teacher’s privileged knowledge is useless; it shows that it is hard to transmit where the student cannot see what it rests on. | Proposed |
| THEORY-1274 | Under a shared linear output head, the cross-entropy gradient of a multi-token-prediction head at its hidden state is exactly the gradient of a forward KL toward the model’s own position-aligned next-token prediction, held fixed, plus the next-token cross-entropy gradient at the aligned state, which alone carries the sampled target; the identity is about gradients and computations, not information, and in heads fed the intervening tokens the student and its teacher condition on the same evidence | Teoh (2026), LIT-1896, eqs. 1–3 and Appendix B; read in NOTE-1470 and checked by finite differences. Active: elementary and checked. It is a gradient identity, not a decomposition of the loss’s value, and it does not by itself make the self-distillation a projection onto less information: in the DeepSeek-style heads of the paper’s main results the student sees the same tokens as its teacher. | Active |
| THEORY-1313 | In two 7–8B chat models, the shift that finetuning induces along a difference-of-means persona vector correlates with how strongly the finetuned model then expresses that trait, r = 0.76–0.97 across synthetic datasets for evil, sycophancy and hallucination, above cross-trait baselines of 0.34–0.86; the last-prompt-token projection predicts a response’s trait at r = 0.75–0.83 overall but mainly by telling prompt types apart | Chen et al. (2025), LIT-1998, read in NOTE-1524 (main text in full, appendices C.2, G.2 and M). Active as the reported measurements, with their sizes and scope: two models, three main traits, synthetic datasets and an LLM judge. It does not say that persona vectors are the mechanism of persona change, which the authors leave open, or that the method finds personas it was not told to look for. | Active |
| THEORY-1330 | In Gemma 2 27B, Qwen 3 32B and Llama 3.3 70B, activation vectors for 275 prompted character roles span a low-dimensional space whose first principal component runs from fantastical to Assistant-like roles, with role loadings correlated above 0.92 between models and the default Assistant at its extreme; the contrast vector from the default Assistant to the role mean aligns with it and is largely present before post-training | Lu et al. (2026), LIT-1991, main text read in NOTE-1505. Active as the reported geometry, with its sizes: three open-weight models, judge-filtered role vectors at one layer, base-model evidence from one model’s PCs and two models’ steering. It does not say the Assistant is a single direction (the authors call the linear assumption “likely flawed”), nor that the axis’s meaning beyond its ends is settled. | Active |
| THEORY-1334 | Persona shifts induced by prompting and by finetuning are mediated by the same linear trait directions, so finetuning on data that implies a trait moves the model along the trait’s persona vector as a system prompt would; Chen et al. suspect personas are latent factors that persist across many tokens | Chen et al. (2025), LIT-1998, read in NOTE-1524. The mediating reading is the authors’ framing and PSM’s use of the paper (LIT-1952: training-time shifts “can be regarded as conditioning”). What the paper shows is correlation of shift with expression, and steering effects in both directions; that the regularizer against the shift fails, with the trait moving to other directions, counts against a single direction being the carrier. Proposed for that gap. | Proposed |
| THEORY-1339 | In multi-turn conversations a chat model’s persona is not fixed for the episode: its position along the Assistant Axis is set mainly by the most recent user message (ridge R² 0.53–0.77 from message embeddings) rather than by where it was before (R² 0.10 for the change), so that therapy-like and AI-philosophy conversations drift away from the Assistant and coding and writing ones do not | Lu et al. (2026), LIT-1991, read in NOTE-1505. The figures are the paper’s (§§4.1–4.2), from 100 synthetic conversations per domain with frontier models playing the user. Proposed because the users are simulated, the conversations are generated by domain, and the regression is on embeddings of whole messages; the authors themselves ask for a human study. | Proposed |
| THEORY-1342 | In Transformer attention the weights each query puts on the positions depend only on query–key compatibility, softmax(QKᵀ/√d_k), and the output is those weights applied to the values, so for a fixed input what a head selects and what it transports are separate factors; because queries, keys and values are all projections of the same input, the separation holds only with the input, and so the pattern, held fixed | Vaswani et al. (2017), LIT-1959, read in full in NOTE-1501, eq. 1 and §3.2.2; the circuit vocabulary is Elhage et al.’s (LIT-1534, THEORY-988). Active: it is the definition of the mechanism, checked against the equations. It does not say that heads are interpretable observers, which the paper supports only with visualizations, or that the separation survives composition across layers. | Active |
| THEORY-1350 | The Persona Selection Model holds that pre-training gives a language model a distribution over personas, that post-training updates it into a posterior over Assistant personas which runtime context keeps conditioning, and that the Assistant’s traits are a key determiner of behaviour; it disowns a single coherent persona and exhaustiveness, and names as open a router that selects personas and could carry goals no persona has | Marks, Lindsey and Olah (2026), LIT-1952, read in full in NOTE-1512. Active as a statement of what the model says and does not say, all of it in the text. It does not say that the model is true: the authors are “overall unsure how complete” it is, its evidence is surveyed rather than new, and its updating is offered as “can be viewed as”, with the strict form said to be too strong. | Active |