Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Representation learning — how learned systems come to represent their data — self-supervised and contrastive objectives, spectral embeddings, information bottlenecks, knowledge-graph embeddings, and whether representations converge.
171 of 2004 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-226 v2 | The Conditional Entropy Bottleneck | Fischer (2020), ARXIV-2002.05379. The Conditional Entropy Bottleneck minimizes I(X;Z|Y) − γI(Y;Z), which is the Information Bottleneck with γ = β − 1, aimed at the “Minimum Necessary Information” point I(X;Y) = I(X;Z) = I(Y;Z). On Fashion MNIST it improves accuracy, adversarial robustness, OoD detection and calibration over VIB and deterministic baselines, and it does not memorize random labels. On CIFAR-10 it improves accuracy and robustness. That excess retained information causes these failures is stated as a hypothesis and supported only empirically. | Active — read 2026-10-14 (NOTE-754) in full; worth reading as the paper that names the Minimum Necessary Information point I(X;Y) = I(X;Z) = I(Y;Z), shows that its objective is the Information Bottleneck with γ = β − 1, and gives variational bounds (the residual information ReX) that make the distance to that point observable. Its CatGen decoder is InfoNCE with the backward encoder as critic, capped by log K. Read it knowing that the causal claim, that retained information causes adversarial vulnerability, miscalibration, poor OoD detection and memorisation, is a hypothesis supported only by Fashion MNIST and one selected CIFAR-10 model per setting, and that the arXiv v1 prints the CatGen inequality (Eq. 30) with the wrong sign. |
| LIT-227 v2 | Contrastive Learning Is Spectral Clustering On Similarity Graph | Tan et al. (2023), ARXIV-2303.15103. Over a finite set of n objects with a symmetric augmentation matrix π, SimCLR’s InfoNCE loss with a Gaussian kernel on the unit sphere equals tr(Zᵀ L(π) Z) + log R(Z), a Laplacian quadratic form plus the log of InfoNCE’s normalizers. The paper calls this spectral clustering under a definition that admits any regularizer, and does not characterize the minimizer. CLIP is treated the same way on a bipartite pair graph. Mixtures of exponential kernels gain 1 to 4 linear-probe points on CIFAR-10/100 and TinyImageNet, not uniformly. | Active — read 2026-10-14 (NOTE-846) in full; worth reading as an exact identity: SimCLR’s InfoNCE loss, over all n objects with a Gaussian kernel on the unit sphere and a symmetric augmentation matrix π, equals tr(Zᵀ L(π) Z) + log R(Z), a Laplacian quadratic form plus the log of InfoNCE’s normalizers. Read it knowing that its “spectral clustering” admits any regularizer (Def. 2.7), so no minimizer is characterized and no eigenvector or density-ratio statement follows. The CLIP extension is a two-sentence proof, the batch-size reading is a remark, and the kernel gains are 1 to 4 points on three small datasets. |
| LIT-228 v2 | Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods | Balestriero & LeCun (2022), ARXIV-2205.11508. On a finite sample with a positive-pair graph G, idealized VICReg is Laplacian eigenmaps, a Euclidean SimCLR variant is a generalized MDS, and linear Barlow Twins is CCA. All three align the representation’s leading left singular vectors with G’s. SimCLR and Barlow Twins force rank(Z) = rank(G), and VICReg can stay full rank. Any of them is optimal for a linear probe on the training set iff G’s spectrum spans the task’s. | Active — read 2026-10-14 (NOTE-752) in full; worth reading for its method-by-method dictionary from joint-embedding SSL to spectral embedding on a finite positive-pair graph G. VICReg with a squared variance term is Laplacian eigenmaps (LPP when linear). A Euclidean SimCLR variant is a generalized MDS whose optimum is the top eigenvectors of G scaled by √eigenvalue. Linear Barlow Twins is CCA. All of them align Z’s left singular vectors with G’s, and they differ in rank. Read it knowing three things. The closed forms hold for modified losses, on a finite sample, with free or linear Z. Theorems 6, 9 and 10 are sketched or unproved. And the practical advice (prefer low-invariance VICReg under a misaligned G) rests on an unproved bound and synthetic experiments with N = 256. |
| LIT-231 v2 | The Hidden Uniform Cluster Prior in Self-Supervised Learning | Assran et al. (2022), ARXIV-2210.07277. Joint-embedding methods whose anti-collapse term maximizes the volume of the representation carry a prior toward features that split the data into equal-sized clusters: explicitly in MSN (a KL to uniform over mean cluster assignments) and SwAV (an equipartition constraint), through an informal reduction to K-means for VICReg, and by a cited equivalence for SimCLR. With mini-batches of two to eight ImageNet classes, SimCLR, VICReg and MSN lose 11–18 points of linear ImageNet accuracy while MAE and data2vec do not. Replacing MSN’s uniform target with a power law (PMSN) gains about a point on long-tailed iNaturalist-18, three with the true class distribution. | Active — read 2026-10-14 (NOTE-850) in full; worth reading for the clearest statement that a joint-embedding objective’s anti-collapse term is not neutral: MSN’s mean-entropy regularizer is a KL penalty pulling cluster sizes toward uniform, and SwAV enforces equal sizes outright. Read it knowing that the same prior is reached for VICReg only through an informal reduction of a simplified loss to K-means, and for SimCLR only through a cited equivalence; that the imbalance experiment uses two to eight ImageNet classes per batch, not a naturally long-tailed dataset; and that the iNat18 gains are single runs of 1–3 points, the largest using the true class labels. The account is THEORY-306; the practical advice belongs in the anthology. |
| LIT-232 v2 | Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding | Gebhart, Hansen and Schrater (2021; AISTATS 2023), ARXIV-2110.03789. A knowledge-graph embedding is a cellular sheaf of vector spaces on the schema graph, with restriction maps per relation type, plus a 0-cochain of its pullback to the knowledge graph. The summed Structured Embedding score is the sheaf Laplacian form, and TransE, TransR and RotatE are special cases; bilinear models are not. Since the zero cochain is always a section, training needs negatives and a margin. Composite multi-hop queries are scored without extra training by harmonic extension, a Schur complement, which beats naive TransE summation on ip and pi queries in untuned experiments. | Active — read 2026-10-14 (NOTE-843) in full; worth reading as the clean dictionary between coboundary-type knowledge-graph scores (Structured Embedding, TransE, TransR, RotatE) and cellular sheaves on the schema, with the summed score as a sheaf Laplacian form and multi-hop queries scored by harmonic extension, a Schur complement. Read it knowing that bilinear models fall outside it, that the zero cochain is always a section so all content is in the contrastive margin, that “approximate global section” means only small Laplacian energy, and that the experiments are preliminary, untuned and well below query-embedding baselines. It shares the word “sheaf” with the contextuality line and nothing else of substance. The account is THEORY-299. |
| LIT-235 v2 | When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis | Sun et al. (2023), ARXIV-2308.05017. A spectral contrastive loss over an augmentation graph that adds labelled same-class pairs (NSCL) is equivalent to factorizing its adjacency. At that optimum, a novel labelling’s least-squares residual is exactly its component on the discarded eigenvectors that their labelled rows cannot absorb, which can be zero. In a toy, known classes sharing the target attribute remove the residual, and ones sharing a rival attribute raise it. | Active — read 2026-10-14 (NOTE-833) in full; worth reading as an exact account, within HaoChen et al.’s spectral contrastive framework, of how labelled data from other classes changes which partition of unlabelled data the learned features express: the novel labelling’s residual is its part outside the top eigenspace that the labelled rows of the discarded eigenvectors cannot absorb, and a toy shows known classes can help or hurt. Read it knowing that Theorem 5.5 is printed as ≤ though its proof gives equality, that everything is at the exact optimum of a population graph, that the general bound (Theorem 5.6) is approximate, pools all known classes into one averaged block and has notational slips, and that the help-or-harm examples have one sample per kind. |
| LIT-240 v3 | The Role of the Information Bottleneck in Representation Learning | Vera et al. (2018), DOI-10.1109/ISIT.2018.8437679. The generalization gap of the cross-entropy loss for an encoder is bounded by a term in the information the representation keeps about the input, so minimizing empirical risk plus that bound is equivalent to the Information Bottleneck on the empirical distribution — offered as an explanation of why noise injection such as dropout helps. | Unavailable — searched 2026-10-14 for a lawful full text of the ISIT 2018 paper and none was reached, so it is not read and no NOTE is filed. OpenAlex: closed, with one other location, the CentraleSupélec HAL record hal-01756003, which the HAL API gives as submitType “notice” (metadata only, no file); Unpaywall: not open access; IEEE Xplore returned a bot check (202), failing on access, not availability, though the paper is subscription-only there; arXiv: a search of Piantanida and Vera lists no paper with this title, only the related, distinct arXiv:1802.05355; CONICET’s repository, which lists an event-paper record for it, returned 502 and 503, so whether it holds a file is unknown; a web search for the title found no author-posted copy. It stays Unavailable until a copy is supplied. |
| LIT-242 v3 | On the Stepwise Nature of Self-Supervised Learning | Simon et al. (2023), ARXIV-2303.15438. For a simplified Barlow Twins loss on a linear or fixed-kernel model, gradient flow from small initialization learns the top eigenvectors of a symmetrized cross-correlation operator one at a time and reaches the minimum-norm zero-loss solution, kernel PCA on K_Γ. Exact from aligned initialization; informal for generic small initialization. ResNets trained with Barlow Twins, SimCLR and VICReg show the same stepwise rank growth, as an experimental finding. | Active — read 2026-10-14 (NOTE-741) in full; worth reading for the dynamics that the optimum-based spectral accounts lack. For a simplified Barlow Twins loss (λ = 1, no batch norm, symmetrized cross-correlation) on a linear or fixed-kernel model, gradient flow from small initialization learns the top eigenvectors of Γ, or of K_Γ, one at a time and reaches the minimum-norm zero-loss solution: kernel PCA. Read it knowing three things. The step for generic initialization (Result 4.6) is an informal derivation. Its operator is built from the model’s kernel and the sample pairs, not from K⁺. And its evidence for SimCLR, VICReg and ResNets is experimental: the initial NTK predicts them poorly, and the final empirical NTK captures about half the embedding subspace. |
| LIT-245 v4 | Minimum Description Length and Generalization Guarantees for Representation Learning | Sefidgaran et al. (2023), ARXIV-2402.03254. The expected generalization error of an encoder–decoder classifier is bounded by 2√((2·MDL + K + 2)/n) + ε for any decoder. Here MDL is the KL divergence between the encoder’s joint law of train-and-ghost latents and a prior symmetric under label-preserving permutations, not I(X;U). The quantity reflects the encoder’s structure and is finite for deterministic encoders under lossy compression. It reduces to sample-wise mutual information for a product prior, and the predicted-label version contains the f-CMI and VC bounds. | Active — read 2026-10-14 (NOTE-742) in full; worth reading as the theorem that bounds the generalisation error of an encoder–decoder learner by the encoder alone: 2√((2·MDL + K + 2)/n) + ε, where MDL is the KL divergence of the joint train-and-ghost latents to a prior symmetric under label-preserving permutations (Theorem 4). The bound holds for any decoder and stays finite for deterministic continuous encoders under lossy compression. With a product prior it reduces to sample-wise mutual information, which is why it can be tighter than IB-style bounds. Read it knowing that the bound is never evaluated numerically, that the prior may depend on the unseen ghost data and is only approximated in practice, that a K + 2 term sets a floor, that the proofs other than Theorem 4’s were not checked here, and that the experiment is one CIFAR-10 CNN at 62–65% accuracy. |
| LIT-246 v3 | On Variational Bounds of Mutual Information | Poole et al. (2019), ARXIV-1905.06922. Every variational lower bound on mutual information in use — Barber–Agakov, Donsker–Varadhan/MINE, NWJ/f-GAN-KL, InfoNCE — is one family, tight at a critic that is a function of the log density ratio log p(y|x)/p(y), and InfoNCE is the multi-sample member that trades variance for a hard ceiling of log K. | Active — read 2026-10-14 (NOTE-739) in full; worth reading as the single derivation that puts Barber–Agakov, Donsker–Varadhan, NWJ, MINE and InfoNCE in one family, and as the exact proof that InfoNCE is a lower bound on mutual information for every critic when the K batch pairs are i.i.d. from the joint, capped at log K. Read it knowing that Eq. 10 is printed with its indices transposed relative to its own derivation (the bound holds either way; the optimal critic log p(y|x) + c(y) belongs to the derived form), that the InfoNCE optimal critic is stated on Ma & Collins’s authority rather than proved, that the optimal critic of the interpolated bound I_α is a conjecture, and that every experiment is on 20-dimensional Gaussian toys or dSprites. |
| LIT-247 v3 | On Mutual Information Maximization for Representation Learning | Tschannen et al. (2019), ARXIV-1907.13625. The success of InfoNCE-style representation learning cannot be attributed to mutual information itself: MI is invariant under invertible reparametrizations, invertible encoders that maximize true MI can be worse than raw pixels, and tighter MI bounds from higher-capacity critics can give worse representations. | Active — read 2026-10-14 (NOTE-749) in full; worth reading as the controlled case that the downstream quality of InfoNCE- and NWJ-trained encoders does not track mutual information: invertible encoders improve while true MI is constant, a higher NWJ bound from a bigger critic gives worse features, and at matched bound values the encoder architecture decides. Also the proof that InfoNCE is neither a lower nor an upper bound when batch pairs are dependent, and the identity between InfoNCE with an inner-product critic and the multi-class K-pair loss. Read it knowing that the experiments are MNIST image halves (CIFAR10 for two of them) under a linear probe, that the tighter-bound result is shown for NWJ only (InfoNCE saturates near log K for every critic), and that the metric-learning account is argued, not tested. |
| LIT-248 v2 | Self-Supervised Learning with Kernel Dependence Maximization | Li et al. (2021), ARXIV-2106.08320. With image identity as a one-hot label, HSIC(Z,Y) = (Δl/N)(E_pos k − E k), and the mean squared MMD between the per-image view distributions is 2(N/Δl)·HSIC(Z,Y) (Thm A.1, App. B.2). The large-batch InfoNCE loss is exactly −HSIC(Z,Y) plus a log-mean-exp of the centred critic (eq. 28), and for a bounded kernel it bounds −HSIC(Z,Y) + γHSIC(Z,Z) up to a second-order remainder (Thm B.1). SSL-HSIC, −HSIC(Z,Y) + γ√HSIC(Z,Z), is proposed as a loss. | Active — read 2026-10-14 (NOTE-830) in full; worth reading as the place where an SSL objective becomes a statement about kernel mean embeddings: with image identity as a one-hot label, HSIC(Z,Y) is the gap between the mean kernel on positive pairs and on all pairs, (1/2N²)Σ MMD²(i,j) = (N/Δl)·HSIC(Z,Y), and the large-batch InfoNCE loss is exactly −HSIC(Z,Y) plus a log-mean-exp of the centred critic. Read it knowing that Theorem B.1 prints its remainder with the wrong sign, that the InfoNCE it analyses is the large-batch form and not the K-negative estimator, that “InfoNCE works because it estimates kernel dependence” is an interpretation no experiment isolates, and that the experiments are single runs. |
| LIT-249 v4 | Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss | HaoChen et al. (2021), ARXIV-2106.04156. Minimizing the population spectral contrastive loss L(f) = −2E[f(x)ᵀf(x⁺)] + E[(f(x)ᵀf(x⁻))²] is, up to a constant, a rank-k factorization of the normalized adjacency matrix of the augmentation graph, so its minimizers are the top-k eigenvectors (eigenfunctions, in the infinite case) of that graph up to a row scaling and an invertible linear map — neither of which changes linear-probe accuracy. | Active — read 2026-10-14 (NOTE-740) in full; worth reading as the founding statement that a contrastive loss is a low-rank factorization of the normalized augmentation graph. Lemma 3.2 makes the spectral contrastive loss equal to ‖D^(−1/2)AD^(−1/2) − FFᵀ‖²_F up to a constant, so its population minimizers are the top-k eigenvectors up to a row scaling and a right factor, which Lemma 3.1 shows a linear probe cannot see. Theorem 3.8 then bounds linear-probe error by Õ(α/ρ²_⌊k/2⌋). Read it knowing that everything is proved on a finite augmentation space at the population optimum of the spectral loss alone. The infinite case (App. F) assumes a uniform bound on the density ratio, not only Hilbert–Schmidt, and asserts rather than proves the extension of the main theorems. The symmetry of the pair distribution is used throughout and never stated. |
| LIT-250 v3 | Duality of Bures and Shape Distances with Implications for Comparing Neural Representations | Harvey et al. (2023), ARXIV-2311.11436. The Procrustes shape distance between two representations (optimal rotational alignment of units) equals the Bures distance between their centred linear kernel matrices, so the rotation-invariant geometry of a representation is exactly the geometry of its kernel, and through Uhlmann’s theorem the fidelity of two kernels is the maximum overlap over all representations consistent with them. | Active — read 2026-10-14 (NOTE-753) in full; worth reading as the clean finite-dimensional statement that, for centred linear kernels, the Procrustes distance equals the Bures distance and normalized Bures similarity is the cosine of the Riemannian shape distance, so the kernel is a complete invariant of a representation up to translation and orthogonal maps, and as the proof that CKA and NBS are related only within a rank-dependent envelope. Read it knowing that it covers linear kernels only, assumes the orthogonal group rather than arguing for it, and treats the infinite limits by a law-of-large-numbers sketch and a citation; it does not reach an RKHS or GNS statement. |
| LIT-251 v2 | Contrastive learning, multi-view redundancy, and linear models | Tosh, Krishnamurthy & Hsu (2021), ARXIV-2008.10150. The logistic contrastive problem of telling a same-datum pair of views from an independent pair is solved over all functions by the pointwise mutual information, whose exponential g*(x,z) = p(x,z)/(p(x)p(z)) is the change of measure from p(z) to p(z|x). When the two views are redundant for a label, the infer-then-predict function E[E[Y|Z]|X=x] = ∫ E[Y|Z=z] g*(x,z) p(z) dz is within εX + 2√(εXεZ) + εZ of the Bayes predictor, so linear functions of the landmark embedding (g*(x,Z_1), …, g*(x,Z_m)) are near-optimal, with error O(1/m). | Active — read 2026-10-14 (NOTE-841) in full; worth reading as the redundancy account of contrastive learning: when two views are redundant for a label, the infer-then-predict function µ(x) = E[E[Y|Z]|X=x] is within εX + 2√(εXεZ) + εZ of the Bayes predictor (Lemma 1), and µ is linear in the density ratio g*(x,·) = p(x,z)/(p(x)p(z)), “the change-of-measure from the marginal distribution of Z to the conditional distribution of Z given X = x”. Random landmark evaluations of g* then give near-Bayes linear predictors at rate 1/m. Read it knowing that it is population-level, that the logistic problem has a single negative, that its error propagation assumes a uniform bound on the ratio, and that its low-dimensional direct embeddings are shown to exist, not to be reached. It never assumes the views are exchangeable. |
| LIT-252 v2 | Spectral Inference Networks: Unifying Deep and Spectral Learning | Pfau et al. (2018), ARXIV-1806.02215. For a symmetric kernel k on a space with sampling density p, the span of the top-N eigenfunctions of Kf = E_x′[k(x,x′)f(x′)] on L²(p) maximizes the generalized Rayleigh quotient Tr(Σ⁻¹Π), with Σ = E[u uᵀ] and Π = E[k(x,x′)u(x)u(x′)ᵀ]; a Cholesky-masked gradient orders the eigenfunctions, argued informally, and moving averages of Σ and its Jacobian correct the minibatch bias. Slow Feature Analysis is the graph-Laplacian case on adjacent frames, where the slowest features are the bottom of the Laplacian’s spectrum. | Active — read 2026-10-14 (NOTE-824) in full; worth reading as the generic neural solver for eigenfunctions of a symmetric kernel operator on L²(p): the top-N span maximizes Tr(Σ⁻¹Π), a Cholesky-masked gradient orders the eigenfunctions, and moving averages of Σ and its Jacobian remove the minibatch bias. Slow Feature Analysis is the graph-Laplacian case. Read it knowing that symmetry of k is a precondition placed on the kernel, not derived from a pair distribution, that the ordering argument is informal by the authors’ own statement, that only the hydrogen-atom experiment has ground truth, and that it never mentions positive pairs, contrastive learning or the density ratio. |
| LIT-253 v2 | When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective | Nielsen et al. (2025), ARXIV-2506.03784. For softmax models p(y|x) ∝ exp(f(x)ᵀg(y)) under a diversity condition, the model distribution determines the embeddings and unembeddings up to an invertible linear map (f = Af′, g₀ = A⁻ᵀg′₀). The determination is not stable under KL divergence, though: two models can be ε-close in KL, even both near maximum likelihood, while their embeddings are far from linearly equivalent. Under a log-likelihood variance distance it is stable: d_LLV ≤ ε implies a representational dissimilarity of at most 2Mε. | Active — read 2026-10-14 (NOTE-738) in full; worth reading as the exact statement that softmax models are identified only up to the paired map f = Af′, g₀ = A⁻ᵀg′₀, and as the proof that this identification is not stable under KL divergence but is stable under a log-likelihood variance distance (d_LLV ≤ ε ⇒ d_f,g ≤ 2Mε). Read it knowing that the KL counterexample needs the unembedding norm to grow without bound, that how far apart its embeddings are is shown numerically rather than proved, that everything concerns the final embedding and unembedding only, and that the experiments use 2- to 5-dimensional representations. |
| LIT-254 v4 | Similarity of Neural Network Representations Revisited | Kornblith et al. (2019), ARXIV-1905.00414. Any similarity index invariant to invertible linear maps gives the same value for all representations of rank n, the number of examples (Theorem 1). The paper argues, without a theorem, for invariance to orthogonal maps and isotropic scaling only, and proposes centered kernel alignment, normalized HSIC of centred Gram matrices, which has exactly those invariances. R²_CCA and linear regression are CKA of re-weighted Gram matrices. Linear CKA alone matches corresponding layers across CNN initializations (99.3%, against at most 45.4% for the other linear indexes). | Active — read 2026-10-14 (NOTE-743) in full; worth reading as the source of CKA for comparing representations and of the argument that a similarity index should be invariant to orthogonal maps and isotropic scaling but not to all invertible linear maps. That argument rests on Theorem 1: any index with that wider invariance is constant once rank reaches the number of examples. It also shows R²_CCA, linear regression and linear CKA to be one family, differing in eigenvalue weighting, with canonical ridge interpolating between them. Read it knowing that the case for the orthogonal group is argued from training dynamics, not proved, that it does not state the converse that equal Gram matrices imply an orthogonal relation, and that RBF CKA is scale-invariant only because its bandwidth is set from the median distance. |
| LIT-256 v2 | Mutual Information Neural Estimation | Belghazi et al. (2018), ARXIV-1801.04062. Mutual information is stated measure-theoretically as the expectation under the joint of log dP_XZ/d(P_X⊗P_Z), and its Donsker–Varadhan dual is tight exactly at T* = log dP/dQ + C, so a neural critic trained on the dual is a log Radon–Nikodym-derivative estimator. | Active — read 2026-10-14 (NOTE-828) in full; worth reading as the paper that states mutual information as the expected log Radon–Nikodym derivative and proves the Donsker–Varadhan bound tight at that log derivative plus a constant. Read it knowing that its consistency theorem is for the exact empirical supremum over a compact network family, that two steps of its proof have gaps, that its gradient-bias fix is a heuristic, and that the finite-sample estimate is not a bound (LIT-246). |
| LIT-257 v2 | What Representational Similarity Measures Imply about Decodable Information | Harvey et al. (2024), ARXIV-2411.08197. For linear readouts regularised by w ↦ wᵀG(X)w, the optimal decoded signal is K_X z with K_X = X G(X)⁻¹ Xᵀ / M, and the expected alignment of two networks’ readouts is Tr(K_X K_z K_Y). With K_z = I that gives linear CKA and mean squared CCA as normalized averages, ENSD as the average, and the GULP distance as the average squared readout distance. The Procrustes distance of normalized representations and the average decoding distance bound each other with constants set by the participation ratio of K_X − K_Y. All at finite M. | Active — read 2026-10-14 (NOTE-834) in full; worth reading as the decoding reading of representational similarity: an optimal regularized linear readout outputs K_X z, so linear CKA, mean squared CCA, ENSD and the GULP distance are task-averaged readout statistics, and Procrustes distance and decoding distance bound each other through a participation ratio. Read it knowing that every identification fixes the target covariance to the identity at finite M, and that its population appendix is an outline that does not reach that case. |
| LIT-258 v2 | Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions | Johnson et al. (2022), ARXIV-2210.01883. On a finite augmentation space, the NT-Logistic and spectral contrastive losses are minimized over unconstrained kernels by the positive-pair kernel K⁺(a₁,a₂) = p⁺(a₁,a₂)/(p(a₁)p(a₂)), and InfoNCE by K⁺ up to a constant on each communicating class. Kernel PCA under K⁺ returns exactly the eigenfunctions of the positive-pair Markov chain, and the top-d of those span the linear-predictor subspace that minimizes worst-case approximation error over approximately view-invariant targets. | Active — read 2026-10-14 (NOTE-751) in full; worth reading as the cleanest statement linking contrastive losses, one Mercer kernel and a minimax-optimal eigenbasis. The NT-Logistic and spectral losses are minimized over unconstrained kernels by K⁺ = p⁺/(p⊗p), and InfoNCE by K⁺ up to a constant per communicating class (over symmetric functions, with the minimizer form taken from Van den Oord et al.). Kernel PCA under K⁺ returns λ_i^(1/2) times the positive-pair chain’s eigenfunctions (Thm 3.2). The top-d span minimizes worst-case L²(p) error over approximately view-invariant targets, at ε/(2(1 − λ_(d+1))), provided d is at least the number of communicating classes (Thm 4.1). Read it knowing that everything is proved on finite spaces at the unconstrained population optimum. The supporting experiments are synthetic, and the authors report that they could not extract the eigenfunctions from real-data models. |
| LIT-267 v3 | The Lattice Representation Hypothesis of Large Language Models | Xiong (2026), ARXIV-2603.01227. Thresholding linear attribute directions (Fisher-LDA probes) gives a crisp object–attribute incidence, and the paper’s Theorem 1 is the standard fact that any such incidence has a complete concept lattice. The geometric meet is half-space intersection; the join is defined as the set union of two cones and “approximated by the conic hull” of their directions, which as written is not a lattice join. On five WordNet domains with GPT-4o-annotated attributes, the probes recover the incidence at F1 69.7–83.2 on three 7–8B models, and profile-based subsumption reaches F1 57.1–77.1. No lattice law is ever tested. | Proposed — read in full 2026-09-27 (NOTE-240); promising as a bridge between probe geometry and formal concept analysis, and unproven as a hypothesis about LLMs. It would be settled by three things. The attributes would have to come from something other than another LLM’s annotations. The region-level join would have to be defined correctly, as the Galois closure of the shared intent. And lattice identities (closure, absorption, meet/join agreement with the recovered context) would have to be tested, not pairwise subsumption and candidate ranking alone. |
| LIT-269 v2 | Attention Approximates Sparse Distributed Memory | Bricken & Pehlevan (2021), ARXIV-2111.05498. Kanerva’s SDM read weights each stored pointer by the number of neurons in the intersection of two Hamming balls of radius d. The paper shows that this weight is roughly log-linear in Hamming distance for close patterns. As a result, with L²-normalised vectors and a regression-fitted β, softmax attention reproduces SDM’s retrieval behaviour on random and learnt-projection data (App. B.7). Two things are weaker than they look. The analytic derivation (App. B.2, Eqs. 16–21) gets the exponent wrong: by my recomputation its β is ≈2–2.5× too small. The biological case is Kanerva’s 1988 cerebellar mapping, restated rather than re-argued. It also needs r = 2⁶⁴ ≈ 1.8×10¹⁹ neurons in the Attention setting, which the paper itself calls “biologically implausible” (App. B.5). | Active — read in full 2026-09-27 (NOTE-242); worth reading as the clearest statement of the SDM–attention correspondence. Its exact circle-intersection formula and its eight-algorithm convergence comparison are the parts to take away. Treat its analytic exponential derivation, its GPT-2 “confirmation” and its cerebellar implementation of attention as argument, not demonstration. |
| LIT-272 v2 | Reasoning about Meaning in Natural Language with Compact Closed Categories and Frobenius Algebras | Kartsaklis et al. (2014), ARXIV-1401.5980. The paper sets the sentence space equal to the noun space (F(s)=F(n)=W). It then uses the basis-copying map σ: v_i ↦ v_i⊗v_i of the Frobenius algebra fixed by W’s basis to lift corpus-built verb matrices Σ_i sbj_i⊗obj_i into rank-3 tensors. So a transitive sentence becomes sbj⊙(verb·obj) (“copy-subject”) or obj⊙(verbᵀ·sbj) (“copy-object”), a vector in W. On the Grefenstette–Sadrzadeh verb-disambiguation set, copy-object gets Spearman ρ=0.172, against 0.168 (Kron), 0.163 (Multp), 0.143 (copy-subject), 0.050 (additive) and an inter-annotator bound of 0.620. On a 112-term definition-matching task the best F1 is 0.21 (nouns) and 0.27 (verbs). No significance test is reported. The disambiguation numbers are identical to the authors’ COLING 2012 paper, and the definition numbers are the same data re-reported. | Active — read in full 2026-09-27 (NOTE-246); worth reading for the construction, not for its evidence. It is the citable categorical statement of the Frobenius lift of DisCoCat verbs: the functorial recasting, S=N=W, and the copy-subject and copy-object maps. Its experiments are the authors’ COLING 2012 numbers, re-reported without that paper’s caveat that copy-object’s lead is statistically insignificant. The best disambiguation ρ is 0.172, against a human bound of 0.620. The construction depends on a choice of basis, which the paper cites but never draws the consequence of. |
| LIT-302 v2 | The Platonic Representation Hypothesis | Huh et al. (2024), ARXIV-2405.07987. A position paper with one argument and three measurements. Measured with mutual 10-nearest-neighbour overlap, vision models that solve more VTAB tasks agree more with each other (78 models). Language models with lower bits-per-byte agree more with DINOv2, CLIP and others on Wikipedia image–caption pairs, and this alignment tracks HellaSwag and GSM8K scores. The cross-modal score “only reaches 0.16” out of a maximum of 1 (§6), and a CKA version “revealed a very weak trend” (App. A). The formal argument (§4.2, App. F) shows only this: in an idealised world of discrete events seen through bijective observations, the Bayes-optimal NCE critic is the pointwise-mutual-information kernel plus a constant, and that kernel is representable as an inner product under a sufficient smoothness condition (Prop. F.1). So every such modality shares one kernel, up to an additive constant. | Active — read in full 2026-09-29 (NOTE-286); worth reading as the source of the “Platonic representation hypothesis” and of the one clean formal statement behind it (Bayes-optimal NCE critics equal the PMI kernel, so bijective modalities share a kernel). Read the empirical case as local nearest-neighbour agreement of modest size, chosen after CKA showed little. Read the endpoint, “a statistical model of reality”, as a conjecture that the paper’s own §6 bounds by the mutual information between modalities. |
| LIT-304 v2 | The Linear Representation Hypothesis and the Geometry of Large Language Models | Park et al. (2023), ARXIV-2311.03658. Three readings of “concepts are directions” are made precise with counterfactual pairs, and proved to line up. The shared direction of word-pair differences in the unembedding space is a logit-linear probe (Thm 2.2). The embedding-space direction is a steering vector that leaves causally separable concepts unchanged (Thm 2.5). Under any inner product making separable concepts orthogonal, the Riesz map sends the first to the second (Thm 3.2). Softmax training fixes the representation only up to γ ↦ Aγ + β, λ ↦ A⁻ᵀλ, so no inner product is identified, and the paper chooses M = Cov(γ)⁻¹ by setting a free diagonal D to the identity (Thm 3.4). Evidence: 27 concepts on LLaMA-2 7B (26 show a common direction; thing⇒part does not), read from histograms and heatmaps with no test statistic, plus a Gemma-2B heatmap comparison. | Active — read in full 2026-09-29 (NOTE-302); worth reading as the short, precise source of the counterfactual-pair formalisation of the linear representation hypothesis and of the observation that softmax training leaves the representation’s inner product unidentified. Read its theorems as statements about the final-layer context vector and the unembedding only. Read the choice M = Cov(γ)⁻¹ as a stipulation (D = I), not as a result, and the experiments as qualitative. |
| LIT-305 v2 | On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups | Kondor & Trivedi (2018), ARXIV-1802.03690. Theorem 1 says that a feed-forward network whose layers are linear maps followed by pointwise nonlinearities, with every layer indexed by a homogeneous space G/H_ℓ of a compact group G, is G-equivariant (every layer transforming by the induced action) if and only if each linear map is a generalised convolution f ∗ χ with a filter on the double coset space H_{ℓ−1}\G/H_ℓ. The “if” direction is a short coset computation. The “only if” direction goes through the group Fourier transform: equivariant maps preserve isotypic components (Lemma 6) and act on each Fourier matrix by right multiplication, M ↦ MB (Lemma 10), which the convolution theorem (Prop. 2) turns back into a convolution. | Active — read in full 2026-09-29 (NOTE-282); worth reading as the standard ML statement of “equivariance ⇔ convolution” for compact groups and as a compact introduction to isotypic decomposition and noncommutative Fourier analysis for neural networks. Read the “only if” proof as an outline. It is written out for finite or countable groups and scalar channels only, and one step (removing the pointwise nonlinearity) does not hold as stated for ReLU. |
| LIT-314 v2 | Group Equivariant Convolutional Networks | Cohen & Welling (2016), ARXIV-1602.07576. Group convolution (eqs. 10–11) replaces translation by the elements of a discrete group G, so feature maps become functions on G and every layer is G-equivariant (eq. 12, with pooling and nonlinearities shown to commute with the action). With parameter counts held roughly fixed, p4-CNNs cut rotated-MNIST test error from 5.03% (planar baseline) to 2.28% (previous best 3.98%), and p4m convolutions cut plain-CIFAR10 error from 9.45% to 6.46% on a ResNet44 (Tables 1–2). | Active — read in full 2026-09-29 (NOTE-288); worth reading as the short, self-contained origin of the group-convolution layer and of the “equivariance as weight sharing over a symmetry group fixed in advance” design. Read the CIFAR “state of the art” claim as holding for un-augmented CIFAR10 only, and the paper as containing no representation theory (no irreps, characters or Fourier analysis on groups). |
| LIT-319 v2 | Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges | Bronstein et al. (2021), ARXIV-2104.13478. The authors cast CNNs, GNNs, Deep Sets, Transformers, spherical, mesh and gauge CNNs, and LSTMs as instances of one “blueprint” (p. 29). Each is a stack of linear G-equivariant layers, pointwise nonlinearities and local pooling, capped by a G-invariant global pooling, with the domain Ω and its symmetry group G given in advance (p. 27: “a symmetry group G”; p. 4: “exploiting the known symmetries”). The two priors are symmetry and scale separation. The Fourier transform is derived as the joint eigenbasis of shift-commuting (circulant) operators, which requires distinct eigenvalues (p. 37). Its group-theoretic generalisation via irreducible representations is announced and deferred to “future work” (p. 43). | Active — read in part 2026-09-29 (NOTE-274); worth reading as the standard systematisation of symmetry-as-architecture in deep learning and for its clean derivation of convolution and the Fourier basis from translation symmetry (§4.2). Read it as a proto-book that defers its representation theory (irreps, group Fourier transforms) and its learning theory to “future work”. It takes the symmetry group as known and never addresses discovering one. |
| LIT-322 v2 | Not All Language Model Features Are One-Dimensionally Linear | Engels et al. (2024), ARXIV-2405.14860. Defines a feature as irreducibly multi-dimensional when no rotation and translation makes its distribution either a product of independent parts (separable) or a mixture with a lower-dimensional component (Def. 2). Softened versions are the separability index S(f) and the ε-mixture index M_ε(f) (Def. 3). Clustering sparse-autoencoder dictionary elements by cosine similarity then finds circular representations of weekdays and months in GPT-2 (also 20th-century years) and in Mistral 7B. Replacing the circle coordinates at the day or month token causally changes the answer on “N days/months from X” prompts in Mistral 7B and Llama 3 8B, and in early layers the circle patch nearly matches patching the whole layer. The models’ task accuracy is modest on weekdays (29/49 Llama, 31/49 Mistral), high on months (143/144, 125/144), and near-trivial for GPT-2 despite its circles. | Active — read in full 2026-09-29 (NOTE-273); worth reading as the first causal demonstration that a pretrained language model computes with an irreducibly two-dimensional (circular) feature, and for a clean statistical definition of feature reducibility. Read the discovery as largely manual: circles for days, months and years, found by eye among about 2,500 inspected clusters, with the automatic score ranking them 9th, 28th and 15th of 1,000. Read “irreducible” as statistical, not representation-theoretic. |
| LIT-323 v3 | Toy Models of Superposition | Elhage et al. (2022), ARXIV-2209.10652. In a toy autoencoder x′ = ReLU(WᵀWx + b), with n sparse synthetic features compressed into m < n dimensions, a linear model keeps only the top-m features, but with the output ReLU and sparse enough features the model stores more features than dimensions (“superposition”), in non-orthogonal directions whose interference the ReLU and a negative bias filter out. Whether a feature is dropped, superposed or given a dedicated dimension changes discontinuously with sparsity and importance, and in the n = 2, m = 1 theory this is a first-order phase change. With uniform features the directions settle into uniform polytopes, with feature dimensionalities clustering at 3/4, 2/3, 1/2, 2/5 and 3/8. A ReLU hidden layer creates a privileged basis in which monosemantic and polysemantic neurons coexist, and a model can compute |x| for more features than it has neurons. Everything is shown in toy models only, and the authors expect the geometry and learning-dynamics results to generalise least. | Active — read in full 2026-09-29 (NOTE-278); worth reading as the founding statement of the superposition hypothesis and of the vocabulary the field now uses: features as directions, privileged versus non-privileged bases, polysemanticity as interference, and feature dimensionality. It is also the cleanest demonstration that a nonlinearity plus sparsity makes it optimal to store more features than dimensions. Its evidence is toy models alone, which the paper says plainly. Its phase-change and polytope results are exact only for n = 2, m = 1 and uniform features. |
| LIT-324 v2 | Opening the Black Box of Deep Neural Networks via Information | Shwartz-Ziv & Tishby (2017), ARXIV-1703.00810. The paper studies small tanh networks (layer widths down from 12 to 2) trained by SGD on a 12-bit, O(3)-symmetric synthetic rule, with mutual information estimated by binning each neuron into 30 bins over all 4,096 inputs. It reports two phases: a short “ERM” phase of a few hundred epochs in which I(T;Y) rises, and a long “compression” phase in which I(X;T) falls. The phases coincide with a drop in per-layer gradient SNR (from high-SNR drift to low-SNR diffusion, at about 350 epochs), and the converged layers sit on the IB curve at fitted β. Everything is shown on small synthetic tasks, with the explanations argued informally. | Active — read in full 2026-09-29 (NOTE-298); worth reading as the source of the IB account of deep-learning dynamics and of the drift/diffusion gradient-SNR phase picture, which McCandlish et al. cite. Its compression and generalisation claims are contested: the anthology’s ANTH-LIT-509 reports them refuted, which I have not checked. Read it as the claim, not the result, and never without the critique. |
| LIT-330 v2 | The approximation of one matrix by another of lower rank | Eckart & Young (1936), DOI-10.1007/BF02288367. For a real n×N matrix a, the rank-r matrix β nearest in the least-squares (Frobenius) distance is obtained by writing a = u′λU (u, U orthogonal, λ diagonal ≥ 0, “Theorem I”) and zeroing all but the r largest diagonal elements. The solution is unique unless λ_r = λ_{r+1}. The squared error is Σ_{i>r} λ_i², so the error is at most l(1−r/R)^{1/2}, with l the length of a and R its rank. The paper also argues that the resulting canonic components cannot straightforwardly be read as psychological factors. | Active — read in full 2026-09-29 (NOTE-301); worth reading as the short, readable origin of the Frobenius-norm low-rank approximation theorem in the form still used. The proof is a first-order stationarity argument plus an unproved simultaneous-diagonalisation theorem, and it covers only the least-squares norm. For the general unitarily-invariant-norm statement, cite Mirsky (1960, not in the record). |
| LIT-334 v3 | Conceptual Spaces: The Geometry of Thought | Gärdenfors (2000), DOI-10.7551/mitpress/2076.001.0001. Proposes a geometric level of representation between symbolic and connectionist models, in which concepts are regions of spaces built from quality dimensions and similarity is distance (per publisher abstract). | Unavailable — seeded 2026-09-29 from metadata and published descriptions, not read: registered because the owner’s prior-art novelty map cites it. A close reading was attempted on 2026-09-29 and no legitimate full text could be reached; it stays Unavailable until a copy is supplied. |
| LIT-337 v3 | Boolean Concept Logic | Wille (2000), DOI-10.1007/10722280_22. Extends Formal Concept Analysis with negation by introducing semiconcepts and protoconcepts, which form double Boolean algebras, as a Boolean concept logic (per the standard description of the paper; abstract not retrieved). | Unavailable — seeded 2026-09-29 from metadata and published descriptions, not read: registered because the owner’s prior-art novelty map cites it. A close reading was attempted on 2026-09-29 and no legitimate full text could be reached; it stays Unavailable until a copy is supplied. |
| LIT-341 v2 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power et al. (2022), ARXIV-2201.02177. Small decoder-only transformers (2 layers, width 128, 4 heads, ~4·10⁵ non-embedding parameters) trained on binary-operation tables over abstract symbols can reach 100% validation accuracy long after fitting the training set. In the headline run (division mod 97, 50% data, Adam with no weight decay, a budget of 10⁶ steps), training accuracy is near-perfect before 10³ steps, validation accuracy only near 10⁶, with little progress before 10⁵. The optimisation time to generalise rises steeply as the training fraction falls (on S₅ near 25–30% data, −1% data costs +40–50% median steps), and weight decay is the most data-efficient intervention tried. | Active — read in full 2026-09-29 (NOTE-287); worth reading as the paper that named grokking and measured its data and optimisation dependence. It is a testbed paper with ablations and no mechanism, so it supplies the phenomenon row 15 cites, not the resolvability or singular-learning-theory reading. |
| LIT-344 v5 | Formal Concept Analysis: Mathematical Foundations | Ganter & Wille (1999), DOI-10.1007/978-3-642-59830-2. The standard mathematical monograph on Formal Concept Analysis: formal contexts, concept lattices and their structure theory (per publisher record). | Unavailable — seeded 2026-09-29 from metadata and published descriptions, not read; a second attempt on 2026-10-09 again reached no lawful full text. Springer sells the 1999 book and the 2024 second edition (OpenAlex: both closed access, no repository copy); HathiTrust holds the 1999 edition search-only; the Internet Archive has no copy. The front matter Springer serves freely (preface and contents) was read; the main text was not. No NOTE is filed. It stays Unavailable until a copy is supplied, not on merit. |
| LIT-345 v2 | Progress measures for grokking via mechanistic interpretability | Nanda et al. (2023), ARXIV-2301.05217. A one-layer ReLU transformer (P = 113, 30% of pairs, full-batch AdamW with λ = 1, 40k epochs) that groks modular addition implements a “Fourier multiplication” algorithm. The embedding maps a and b to cos/sin(w_k·a), cos/sin(w_k·b) at five key frequencies k ∈ {14, 35, 41, 42, 52}, with w_k = 2πk/113. Attention and MLP form cos/sin(w_k(a+b)), and the neuron-logit map W_L (≈ rank 10) reads out Σ_k α_k·cos(w_k(a+b−c)). Progress measures built from this (restricted and excluded loss) show training splits into memorisation (0–1.4k epochs), circuit formation (1.4k–9.4k) and cleanup (9.4k–14k), with the test-accuracy jump inside cleanup. | Active — read in full 2026-09-29 (NOTE-283); worth reading as the fullest mechanistic account of a grokking network. Its learned structure is exactly the character basis of ℤ/113 (five real 2-dimensional irreps), which makes it the nearest existing empirical instance of the map’s row-7 “predicate harmonics = characters” prediction, in a setting where the group was known in advance. |
| LIT-362 v2 | Learning the Irreducible Representations of Commutative Lie Groups | Cohen & Welling (2014), arXiv:1402.4437. Proposes “Weyl’s principle” (via Kanatani 1990) as a definition of disentangling: the elementary components of data are the irreducible subspaces of the representation of a symmetry group acting on it. It learns such a decomposition for one class of group fixed in advance, compact commutative subgroups of SO(D) (“toroidal”), from pairs (x, y) of raw data vectors related by an unobserved group element, y = W R(φ) Wᵀx + ε. The orthogonal basis W (2-D invariant subspaces) is fitted by SGD on a closed-form marginal likelihood, and the integer weights ω_j are then read off a batch rotated by a known 0.1°. The only experiment trains 100 filters on 250,000 white-noise 16×16 patches paired with random rotations of themselves (learned ω from −11 to 12), and uses the invariant √κ̂ for 1-NN on rotated MNIST, reported in a figure with no numbers. Spectral degeneracy plays no role. | Active — read in full 2026-09-29 (NOTE-307); worth reading as the earliest ML paper in this batch to state that the “elementary components” of a representation are the irreps of a symmetry group and to learn them from data; read it as learning the irreducible decomposition of an assumed abelian group class from transformation pairs on raw pixels, not as detecting an unknown symmetry in a trained network, and with an experiment that is a one-group proof of concept on noise patches. |
| LIT-363 v2 | Understanding image representations by measuring their equivariance and equivalence | Lenc & Vedaldi (2014), arXiv:1411.5908. Given a fixed representation φ (HOG, or AlexNet Conv1–Conv5 before ReLU) and a transformation g of the input image chosen by the experimenter (flips, 90° and other rotations, rescaling by 2^{±1/2}, affine warps), it fits a linear map M_g on the representation space with φ(gx) ≈ M_g φ(x), by sparse or structured-sparse regression or by a “transformation layer” trained through the rest of the network, and fits “stitching layers” E with φ′(x) ≈ E φ(x) between different networks. In AlexNet, learned M_g recover most of the classifier’s accuracy for vertical flips and 90° rotations (top-1 error 0.75 uncompensated vs 0.43–0.51 and 0.44–0.53 across Conv1–Conv5, original 0.43), channels scored invariant rise to 99.61% for horizontal flips at Conv5, and Conv1–Conv2 are interchangeable between AlexNet and networks trained on ImageNet, Places or both. It detects no group and uses no spectrum. | Active — read in full 2026-09-29 (NOTE-309); worth reading as the origin of two standard probes of trained networks, fitted linear equivariance maps and model stitching, and as the closest prior art for “measuring symmetry in a trained network’s representation”; read it knowing that every transformation tested is named in advance and applied to the input, that each is fitted separately with no group structure, and that its deep-layer “equivariance” is judged by recovered task accuracy through a high-capacity learned layer. |
| LIT-371 v2 | How do language models learn facts? Dynamics, curricula and hallucinations | Zucchet et al. (2025), arXiv:2503.21676. ‘An 8-layer, 44M-parameter decoder-only transformer trained with AdamW on synthetic biographies (64k individuals by default) learns factual recall in three phases. It first fits the marginal attribute-value distribution, then sits on a plateau exactly at the no-knowledge baseline, then acquires individual-specific associations. Plateau end grows with population as 0.43·N^0.81 (R² = 0.998, N = 4k–256k, 5 seeds). Attention patching from post-plateau checkpoints removes the plateau, and attention from the recall position to name tokens rises through it, so the plateau is when the attention extraction circuit forms. Nothing in the three phases is a compression of input information: the last phase adds individual-specific information.‘ | Active — read in full 2026-09-30 (NOTE-319); worth reading as a clean, intervention-backed decomposition of a loss plateau into circuit formation followed by knowledge storage, in a setting where knowledge is countable; its curriculum and fine-tuning results are single-seed and synthetic, and it measures no information quantity, so it bears on phases of training but not on the IB account. |
| LIT-372 v2 | On the Information Bottleneck Theory of Deep Learning | Saxe et al. (2018), DOI-10.1088/1742-5468/ab3985. Replicating Shwartz-Ziv & Tishby with their own code, the paper shows the information-plane “compression phase” appears only for double-saturating nonlinearities measured through an imposed binning/noise model: tanh compresses, ReLU/softplus/linear do not (binning, KDE, Kraskov; SZT task and MNIST), and re-binning the same tanh run evenly in net input removes it. Compression and generalisation dissociate in all four combinations, full-batch GD compresses as much as SGD, and the drift→diffusion gradient-SNR transition occurs in networks that cannot compress (ReLU, MNIST, a 1-1-1 linear net), so it is real but unrelated to compression. | Active — read in full 2026-09-30 (NOTE-320); worth reading as the direct experimental test of all three claims of the IB account of deep learning (two phases; compression causes generalisation; SGD diffusion causes compression), each refuted as a general statement with the original code, and for its Appendix C argument that I(h;X) is infinite in a deterministic network so every finite information-plane value belongs to the analyst’s noise model. |
| LIT-374 v3 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li (2026), arXiv:2506.20699. ‘Proposes “width”, the least number of cells in an open cover on each of which some predictor is γ-contractive with conditional risk ≤ δ, as a structural complexity separate from VC dimension. From an additive Rademacher bound, √(2d_str log(e(K−1)n/d_str)/n) + LK·R_n(G) (Thm 4.5), it argues that context routing (the “scaffold”) should be trained by different signals from within-context prediction (the “flow”), and that alignment failures are scaffold failures. The routine results are proved: coupon-collector Ω(w log w), Lipschitz composition, penalised model selection, multiclass Sauer–Shelah. The new ones are not. The phase transition at width is assumed in its proof. The bouquet witness of width/VC separation contradicts its own construction. The “fundamental theorem” is false as stated, because infinite pseudo-dimension does not prevent learnability. No theorem links the CS-Laplacian estimator to width. There are no experiments.‘ | Proposed — re-read jointly with LIT-373 and arXiv 2602.07974 on 2026-10-01 (NOTE-322); promising and unproven. Read jointly with the companion arXiv 2602.07974, which carries the fuller statements and several definitions this paper leaves implicit. The formal core holds in repaired form: width as a structural complexity incomparable with the per-cell VC dimension; an error floor below width; the Ω(w log w) discovery cost; an additive Rademacher bound for routed predictors. The flagged slips are repairable and not load-bearing: the bouquet cover, the non-degeneracy assumption in the appendix, Massart, and pseudo-dimension against fat-shattering in Thm 4.6. What remains unproven are the claims that would make it matter: that the CS estimator converges to width itself and not to a predictor-relative count; that a monolithic model cannot match K ≥ w routing; above all, that router/scaffold training should be decoupled from task loss. This is presented as derived from the additive bound, but the bound holds for joint training too. The alignment section is asserted. What would settle it: on synthetic benchmarks of known width, (i) a K-sweep showing the floor at K < w for routed learners and its absence or presence for capacity-matched monolithic models; (ii) CS width recovery from a cold start; (iii) decoupled against joint router training, compared on forgetting and boundary error. |
| LIT-375 v2 | Structural Learning Theory: A Metric-Topology Factorization Approach | Li (2026), arXiv:2602.07974. >- The formal core of Xin Li’s structural-learning programme. Width w(P; γ, δ) is the least number of open cells on each of which some predictor is γ-contractive with conditional risk ≤ δ. The paper proves that width is incomparable with the per-cell VC dimension (Thm 2.2). It shows that with K < w cells there is an n-independent error floor η(w, K), under a non-degeneracy assumption stated in the body (Thm 2.4, Assumption 2.3). Width is at least c·β₁·L/D₀ under a bounded-overlap assumption (Thm 2.5), and discovering all basins costs Ω(w log w) samples (Thm 2.6). The CS-Laplacian eigenvalue count converges uniformly to the predictor-relative count w_G(P) when n ≳ (H_G + d_X log(1/r_x) + log(1/δ′))/(r_x^{d_X} g_eff²) (Thm 3.5); no result shows that w_G equals width. There are no experiments. | Proposed — read in full 2026-10-01 (NOTE-324); >- Promising and unproven. It is the programme’s fullest and most careful formal statement, and it carries the learning theory of LIT-374 §3 with the assumptions in the body. Its routine results hold, and its bouquet slip is repairable. What would settle it: - a theorem that the CS count for a learned predictor equals w(P; γ, δ), not w_G(P); - a K-sweep on benchmarks of known width, showing the floor at K < w for routed learners, and testing whether a capacity-matched monolithic model shows it; - the decoupled-versus-joint training comparison it motivates but does not run. |
| LIT-411 | Could a Large Language Model be Conscious? | Chalmers (2023), arXiv:2303.07103; Boston Review, 9 August 2023; from a NeurIPS 2022 invited talk. Asks for a feature X that LLMs have (or lack) and that indicates consciousness (or its absence). Self-report, seeming-conscious, conversation and general intelligence give at most weak evidence for it. Biology, senses and embodiment, world and self models, recurrent processing, a global workspace and unified agency are the evidence against. All but biology are temporary. On mainstream assumptions: under 10% credence for current LLMs, 25% or more for conscious LLM+ systems within a decade. Twelve challenges are offered as roadmap or red flags. | Active — read in full 2026-10-02 (NOTE-345), from arXiv v3 (the Boston Review text), with v1’s conclusions compared; worth reading as the clearest short statement of the case for and against consciousness in LLMs, and as the origin of the “theory-balanced” credence approach that Butlin et al. built on. Its method is a regimented request for a feature X, and its verdicts are credences offered “for illustrative purposes”: under 10% for current LLMs, 25% or more for conscious LLM+ systems within a decade, on “mainstream assumptions”. It does not break the deadlock THEORY-023 describes. It is an instance of it: mimicry undercuts self-report, the main obstacles are theory-derived, and biology is set aside rather than answered. |
| LIT-427 | Pandemonium: A Paradigm for Learning | Selfridge (1959), Mechanisation of Thought Processes Vol. I (HMSO), pp. 513–526 plus discussion to p. 531. Pattern recognition by a “pandemonium”: data demons hold the input, computational subdemons compute features, each cognitive demon “shrieks” a weighted sum of them, and a decision demon picks the loudest. The system learns by hill-climbing on the weights and by “natural selection” on the subdemons, culling those of low worth and breeding new ones by mutation and by “conjugation” of pairs. It is applied to telling dots from dashes in hand-keyed Morse. In the discussion McCarthy proposes reading the demons’ private computation as unconscious and their public shouting as conscious thought. | Active — read in full 2026-10-02 (NOTE-360); worth reading as the first published architecture of a mind-like process built from many quasi-independent “demons”, each doing one narrow job, with no part holding the whole’s competence. That is the decomposition template that Minsky’s agents inherit. Cite it as an engineering proposal with a preliminary result reported only in discussion, not as a theory of mind. Its one remark on consciousness is McCarthy’s, in the discussion, and it is an aside. |
| LIT-608 | Deep Equals Shallow for ReLU Networks in Kernel Regimes | Bietti & Bach (2021), ICLR 2021. Dot-product kernels on the sphere are diagonal in spherical harmonics, with one eigenvalue µ_k per degree k shared by N(d, k) ~ k^(d−2) harmonics. Theorem 1 reads the asymptotic decay of µ_k off the kernel function’s expansion at ±1: a non-integer exponent ν there gives µ_k ~ k^(−d−2ν+1). The arc-cosine kernels have ν = 1/2 and 3/2, and composing them preserves ν, so the NTK of a deep ReLU network decays as k^(−d) and its random-feature kernel as k^(−d−2), exactly as for two layers. Deep and shallow ReLU kernels therefore have the same RKHS, the same as the Laplace kernel’s for the NTK. | Active — read 2026-10-03 (NOTE-475): main text and Appendix A in full, the proofs in Appendices B and C followed at the level of their lemmas. Worth reading as the general rule behind the NTK eigenvalue decays the record holds: for a dot-product kernel on the sphere, the decay of its spherical-harmonic eigenvalues is fixed by the kernel function’s non-integer exponent at the endpoints ±1 (Theorem 1). Deep ReLU kernels keep the shallow exponent, so depth leaves the spectrum’s shape unchanged: µ_k ~ C(d, L) k^(−d) for the NTK and k^(−d−2) for the random-feature kernel, with only the constant growing in L (Corollaries 2–3). |
| LIT-611 | Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 | Lieberum et al. (2024), Google DeepMind technical report. Gemma Scope releases JumpReLU sparse autoencoders trained on the attention, MLP and residual-stream activations of every layer of Gemma 2 2B and 9B, and of three layers of 27B: over 400 SAEs, over 2,000 counting sparsity variants, and over 30 million latents. Each latent fires only above a learned threshold θ_i, and training penalises the L0 count of active latents directly, through straight-through estimators. Evaluation is by sparsity–fidelity curves (delta LM loss and FVU against mean L0). Base-model SAEs transfer to the instruction-tuned model, wider SAEs split latents, and the authors state that no agreed metric of SAE quality exists. | Active — read in full 2026-10-03 (NOTE-482). Worth reading as the technical report for the open SAE suite that later SAE studies, including the anthology’s, are run on, and as the clearest statement of what a JumpReLU SAE is. A JumpReLU SAE is a learned hard threshold per latent, z ⊙ H(z − θ), trained against squared reconstruction error plus λ‖f(x)‖₀ (Eqs. 3–4). The report releases over 400 such SAEs (over 2,000 counting the sparsity variants) on every layer and sublayer of Gemma 2 2B and 9B and on three layers of 27B. Its evaluations are sparsity–fidelity curves, with no criterion offered for choosing a width or a sparsity; the authors say there is no consensus on what measures SAE quality. |
| LIT-612 | The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies | Basri, Jacobs, Kasten & Kritchman (2019), NeurIPS 2019. In the linearised dynamics of a wide two-layer ReLU network, on uniform data on a sphere, the Gram matrix H∞ is a convolution, so its eigenvectors are Fourier modes or spherical harmonics. Its eigenvalues are computed in closed form per frequency. Without bias, every odd frequency k ≥ 3 has eigenvalue zero, so the network cannot represent or learn it (Theorems 2 and 4). With a zero-initialised bias all frequencies pass. The eigenvalues fall as 1/k² on S¹ and, numerically, about 1/k^(d+1) on S^d, so the time to learn frequency k grows as k² and k³ on the circle and the 2-sphere. Deep and residual networks show the same exponents. | Active — read 2026-10-03 (NOTE-481): main text in full, the supplementary derivations of the S^d eigenvalues skimmed. Worth reading for the closed-form NTK eigenvalues per frequency and the exact null space they reveal. On the circle the bias-free two-layer ReLU kernel has eigenvalue 2(k²+1)/(π²(k²−1)²) on even k ≥ 2 and exactly zero on odd k ≥ 3 (Eq. 12), so such a network cannot represent cos 3θ at all (Theorems 2 and 4). A zero-initialised bias fills the null space, with 1/(π²k²) on odd k (Eq. 13). Convergence time then grows as k² on S¹ and about k³ on S², and shallow, deep and residual networks match: fitted exponents 1.93–2.15 on S¹ and 2.74–3.13 on S² (Figs. 6–7). |
| LIT-613 | Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra | Papyan (2020), JMLR 21(252). Cross-entropy indexes every deepnet quantity by example, class and would-be class, a two-way MANOVA layout. In the spectra of features, backpropagated errors, gradients, the Fisher, the Hessian and weights, knockouts attribute the top outlier to the global mean, C−1 outliers to between-class second moment, a mini-bulk of C(C−1) to between-cross-class covariance, and the main bulk to within-cross-class covariance. Feature and error class means separate from the bulk and become orthogonal with depth. Gradient class means are approximately feature class means ⊗ error class means, layer by layer; K-FAC’s single Kronecker product misaligns the Fisher’s outliers and the class-wise CFAC aligns them. For multinomial logistic regression on a Gaussian class model the expected Fisher spectrum is derived exactly. | Active — read 2026-10-03 (NOTE-474); worth reading as the general account of class/cross-class structure in deepnet spectra: features, backpropagated errors, gradients, Fisher, Hessian and weights all show one outlier from the global mean, C−1 from class means, a mini-bulk of about C(C−1) from cross-class means, and a main bulk from within-class variation. Per layer, gradient class means are approximately Kronecker products of feature class means and error class means, which is why K-FAC, pairing every class with every class, misaligns the Fisher’s outliers, and CFAC, pairing each class with itself, aligns them. Proved only for multinomial logistic regression on a Gaussian model. |
| LIT-618 | Prevalence of Neural Collapse during the terminal phase of deep learning training | Papyan, Han & Donoho (2020), PNAS 117(40). Training past zero error (the terminal phase, TPT) drives four linked phenomena in the last layer: within-class variability collapses (Tr(Σ_WΣ_B†)/C → 0), centred class means become equinorm and equiangular with cosines −1/(C−1), a simplex ETF; classifier rows become proportional to the means; and the decision becomes nearest class mean. Measured on MNIST, Fashion-MNIST, SVHN, CIFAR-10/100, STL10 and ImageNet with VGG, ResNet and DenseNet, 480 models. Test accuracy and DeepFool robustness keep improving during TPT. Theorems: NC1–NC2 imply NC3–NC4 for the Webb–Lowe and Soudry max-margin classifiers, and the simplex ETF uniquely maximizes the small-noise error exponent. | Active — read in full 2026-10-03 (NOTE-476); worth reading as the source of neural collapse: in the terminal phase of training, last-layer features collapse to their class means (NC1), the centred means form a simplex equiangular tight frame (NC2), the classifier aligns with them (NC3) and the decision becomes nearest class centre (NC4). Measured on 7 datasets × 3 architectures; NC1–NC2 imply NC3–NC4 for the Webb–Lowe and max-margin classifiers, and the simplex ETF is the unique optimal codebook in a vanishing-noise Gaussian channel. It explains Papyan’s Hessian outliers as class means outgrowing collapsing within-class variation. |
| LIT-624 | Towards Understanding the Spectral Bias of Deep Learning | Cao, Fang, Wu, Zhou & Gu (2019). For a two-layer ReLU network trained by gradient descent on squared loss, the training residual projected onto the eigenfunctions of the NTK integral operator with the r_k largest eigenvalues decays like (1 − λ_(r_k))^T (Theorem 4.2). This holds for any labelling, provided the sample size beats the eigengap and the width is polynomial in λ_(r_k)^(−1). On uniform data on S^d the eigenfunctions are spherical harmonics, with µ_k = Ω(k^(−d−1)) for k ≫ d and Ω(d^(−k+1)) for d ≫ k on even degrees, and µ_k = 0 on odd k ≥ 3 (Theorem 4.3). Lower degrees are learned first, faster, and with fewer samples and narrower networks. | Active — read 2026-10-03 (NOTE-483): main text and Appendix A in full, the proofs in Appendix B followed at the level of their lemmas. Worth reading as the theorem that makes “spectral bias” a statement about NTK eigendirections: for a two-layer ReLU network trained by gradient descent, the residual projected onto the top r_k eigenfunctions of the NTK integral operator shrinks like (1 − λ_(r_k))^T (Theorem 4.2), for any target, with sample size governed by the eigengap λ_(r_k) − λ_(r_k+1). On uniform spherical data the eigenfunctions are spherical harmonics, the odd degrees above 1 have eigenvalue zero, and µ_k = Ω(max(k^(−d−1), d^(−k+1))) for even k (Theorem 4.3). |
| LIT-657 | The Generalization Ridge: Information Flow in Natural Language Generation | Chang, Deng & Chen (2025), COLM 2026. InfoRidge estimates, layer by layer and through fine-tuning, the matrix-based mutual information between a transformer’s last-token hidden state and the embedding of the target token, and between each block’s residual update and that target. In GPT-2 Small and Medium, Qwen-2.5 0.5B and Llama 3.1 8B on CLUTRR, ECQA and a synthetic arithmetic task, it rises, peaks in the upper-middle layers and falls. On the synthetic task the peak coincides with the best OOD early-exit accuracy while ID accuracy keeps rising. Middle blocks add the most information, attention to task tokens peaks there, and residual scales fitted on OOD data downweight the last layers. The profile persists over 50 decoding steps on CNN/DailyMail. | Active — read 2026-10-03 (NOTE-536); worth reading as a layer-wise information profile of fine-tuned language models: a matrix-based Rényi estimate of I(Zℓ; Y), between the last-token hidden state at layer ℓ and the input embedding of the true next token, rises through the early layers, peaks in the upper-middle layers and falls in the last few, the “generalization ridge”. On a synthetic arithmetic task with GPT-2 Small the peak (0.27 at layer 10 of 12) is where OOD early-exit accuracy is best (56.9%) while ID accuracy keeps rising to 93.2% at layer 12. Learned residual scales put less weight on the final layers when fitted on OOD data. The estimator is a kernel-alignment quantity, not Shannon mutual information: at layer 12 it falls to 0.019, near its layer-6 value, where the same layer decodes the target with 93% ID accuracy. |
| LIT-665 | ZipIt! Merging Models from Different Tasks without Training | Stoica, Bolya, Bjorner, Ramesh, Hearn & Hoffman (2023), ICLR 2024. Merging differently initialised models trained on disjoint tasks fails under permutation, because the permuted model sits in a basin for its own task and not the other’s. ZipIt! instead merges correlated features within and across models through merge and unmerge matrices, and can stop merging part-way, giving a shared trunk with task heads. The abstract reports a 20–60% improvement over prior work on CIFAR and ImageNet splits; it comes near the ensemble with partial zipping, and proves a tighter LMC barrier bound than Entezari et al. when units are redundant. | Active — read in full 2026-10-03 (NOTE-523), main text and Appendices A–F, the proof in Appendix G followed to its reduction to Entezari et al.; worth reading as the case that permutation is the wrong symmetry for merging models trained on different tasks. ZipIt! matches features by activation correlation within each model as well as across the two, and can stop merging at a chosen layer, leaving a multi-head model. On disjoint halves of CIFAR-10 it reaches 79.1% joint accuracy where Git Re-Basin reaches 46.2%, and partial zipping comes within a few points of the ensemble. Its theorem tightens Entezari et al.’s barrier bound when a model has redundant units. With full merging on ImageNet-1k it is no better than a permutation baseline (8.6%), so the gains there come from leaving later layers unmerged. |
| LIT-667 | Does Data Scaling Lead to Visual Compositional Generalization? | Uselis, Dittadi & Oh (2025), ICML 2025. Two labelled concepts with n values each form an n × n grid; training shows k combinations per value and testing uses the rest. On dSprites, 3DShapes, PUG, colored MNIST and a new FSprites, ResNet-50s trained from scratch reach near 100% on seen pairs and fall sharply on unseen ones; 4× more in-distribution data does not close the gap, while more values and more combinations do. Features pass from spurious, to discriminative but entangled, to linearly factored and near-orthogonal as combinatorial coverage grows. Exact linear factorization makes k = 2 sufficient for perfect generalization. Frozen DINO, DINOv2 and CLIP features are above chance but far from that ideal. | Active — read 2026-10-03 (NOTE-525); worth reading as a controlled test of whether more data buys visual compositional generalization: in an (n, k) grid where each of n values of two concepts is seen in only k combinations, ResNet-50s trained from scratch fail on unseen pairs whatever the data volume (4× more in-distribution data leaves 60–80% drops), and succeed only as the fraction of combinations seen rises, when their features become linearly factored (a pair’s embedding is the sum of per-concept vectors; R² > 0.8). Under exact linear factorization, k = 2 combinations per value suffice for a linear classifier to get all unseen pairs right (Proposition 4.1). DINO, DINOv2 and CLIP features are partly factored. Model selection is oracle, on the test set, by design. |
| LIT-669 | Mechanistic Mode Connectivity | Lubana, Bigelow, Dick, Krueger & Tanaka (2022), ICML 2023. Two models are mechanistically similar if they are invariant to the same interventions on the data-generating process. Models that use a spurious cue and models that ignore it are mode connected along quadratic paths but not along linear ones, even after permutation matching, and the paper conjectures that a linear barrier implies mechanistic dissimilarity. For a one-hidden-layer ReLU network it proves only the converse: mechanisms of different complexity cannot be linearly connected. Naive fine-tuning on clean data stays linearly connected to the pretrained model and keeps its reliance on the cue; a fine-tune that forces a barrier removes it. | Active — read 2026-10-03 (NOTE-538); worth reading as the paper that ties a loss barrier to a difference in what two models compute. It defines two models as mechanistically similar when they are invariant to the same interventions on the data-generating process, finds that models relying on a spurious cue and models ignoring it are connected by quadratic paths but not by linear ones, even after permutation, and conjectures that a linear barrier always signals mechanistic dissimilarity. The conjecture is proved only for a one-hidden-layer ReLU network on a synthetic process; the evidence is synthetic cues on CIFAR-10, CIFAR-100 and Dominoes. |
| LIT-674 | Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity | Zhou, Yang, Yang, Yan & Hu (2023), NeurIPS 2023. When two trained networks are linearly mode connected, whether by spawning from a shared early checkpoint or by permutation alignment, the features of nearly every layer of the weight-interpolated network are proportional to the interpolation of the two networks’ features (LLFC). LLFC on the output layer implies LMC. Weak additivity of ReLU and a commutativity condition between layers’ weights and features together imply LLFC, and Git Re-Basin’s two matching objectives are each one factor of the commutativity residual. | Active — read 2026-10-03 (NOTE-512); worth reading as the finding that linear mode connectivity reaches inside the network: whenever two trained networks are linearly connected, by spawning or by permutation, the feature map of almost every layer of the interpolated network points in the direction of the interpolated feature maps (LLFC). Two conditions, weak additivity of ReLU on the two networks’ pre-activations and a commutativity of each layer’s weights with the other network’s features, imply exact LLFC (Theorem 1), and Git Re-Basin’s weight and activation matching each minimise one factor of the commutativity residual. Image classification only. |
| LIT-678 | Intermediate Layer Classifiers for OOD generalization | Uselis & Oh (2025), ICLR 2025. Linear probes on the frozen features of an intermediate layer (ILCs) are compared with last-layer retraining for out-of-distribution classification on CMNIST, Waterbirds, CelebA, MultiCelebA, CIFAR-10C, CIFAR-100C and ImageNet variants, with ResNets and ViTs. Few-shot (probe trained on OOD data), the best intermediate layer usually wins, by more when OOD data are scarce. Zero-shot (probe trained on ID data, OOD only for selecting the layer), it beats retraining the last layer, sharply under subpopulation shift. Layer L − 2 beats L − 1 in every dataset shown. A distance-ratio sensitivity score finds intermediate features less displaced by the shift than penultimate ones, most of all for minority groups. | Active — read 2026-10-03 (NOTE-541); worth reading as the measurement that, under distribution shift, a trained classifier’s best linear probe is usually not on the penultimate layer: Intermediate Layer Classifiers (linear probes on frozen layer-l features, l ≤ L − 2) beat last-layer retraining in the few-shot setting on most of nine shifted datasets, and, trained only on in-distribution data, raise worst-group accuracy from 79.4% to 87.1% on Waterbirds and 56.0% to 82.0% on CelebA. The proposed reason is that penultimate features move further under shift, especially for minority groups, while intermediate ones stay put. Selection of the layer uses OOD validation data, and the gains for ViTs are small. |
| LIT-792 | Transformers learn in-context by gradient descent | von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov and Vladymyrov (2022), ICML 2023. Proposition 1 constructs weights for which one linear self-attention layer transforms each token (x_j, y_j) to (x_j, y_j − ΔW x_j), with ΔW one gradient step on the context’s squared loss, so the query token’s read-out equals the prediction of the updated linear model. A trained single layer matches the construction up to scale; deeper models outperform K steps of gradient descent and match GD++, gradient descent on inputs transformed by I − γXXᵀ. Synthetic regression only. | Active — read 2026-10-09 (NOTE-621); worth reading here for one exact identity: with constructed key, query, value and projection matrices, one linear self-attention layer applied to tokens (x_j, y_j) changes every token exactly as one gradient-descent step on the context’s squared regression loss would change the targets (Proposition 1), so reading a context and updating an implicit model’s weights on it produce the same prediction. A trained single layer finds those weights up to scale; deeper trained models match a preconditioned variant (GD++) rather than plain gradient descent. Everything is on synthetic regression with small attention-only models. |
| LIT-796 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Akyürek, Damani, Zweiger, Qiu, Guo, Pari, Kim and Andreas (2024), ICML 2025. Test-time training here is a temporary, per-task LoRA update of a language model on a loss built from the task’s own demonstrations, discarded afterwards. On BIG-Bench Hard (10-shot, Llama 3.1 8B) it raises accuracy from 50.5% (in-context learning) to 57.8%; on ARC it takes a fine-tuned 1B model from 5% to 29% on an 80-task subset and an 8B system to 53.0% on the public validation set. Training on leave-one-out in-context tasks beats training on the bare pairs. | Active — read 2026-10-09 (NOTE-637); worth reading here as a controlled comparison of two uses of the same demonstrations: held as context (in-context learning) or turned into a temporary update of the model’s weights (test-time training, per-task LoRA, discarded after the task). With nothing added but the update, BIG-Bench Hard 10-shot accuracy goes from 50.5% to 57.8% and fine-tuned ARC accuracy roughly sixfold on an 80-task subset (5% to 29%). The update works best when its training data keep the in-context format. The comparison is empirical, at 1B–8B parameters, on two benchmarks with known leakage risk. |
| LIT-801 | Learning to (Learn at Test Time): RNNs with Expressive Hidden States | Sun, Li, Dalal, Xu, Vikram, Zhang, Dubois, Chen, Wang, Koyejo, Hashimoto and Guestrin (2024), ICML 2025. Makes a sequence layer’s hidden state the weights W of an inner model f, updated per token by a gradient step on a learned self-supervised reconstruction loss, with output f(θ_Q x_t; W_t). With a linear f and batch gradient descent this is exactly linear attention (Theorem 1); with the Nadaraya–Watson estimator as inner learner it is exactly softmax self-attention (Theorem 2). TTT-Linear and TTT-MLP match or beat Mamba at 125M–1.3B and keep lowering perplexity to 32k tokens of context, where Mamba plateaus after 16k. | Active — read 2026-10-09 (NOTE-620); worth reading for its reframing of a sequence layer as a learner trained on its own context: the hidden state is the weights of an inner model, the update rule a gradient step on a self-supervised loss, the output rule the inner model’s prediction. Two exact identities follow: a linear inner model with batch gradient descent is linear attention (Theorem 1), and the Nadaraya–Watson estimator as inner learner is softmax self-attention (Theorem 2). The empirical part, TTT-Linear and TTT-MLP against Mamba and a Transformer at 125M–1.3B, is an architecture result that belongs to the anthology’s question. |
| LIT-814 | Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? | Min, Lyu, Holtzman, Artetxe, Lewis, Hajishirzi and Zettlemoyer (2022), EMNLP 2022. Replacing the gold labels of in-context demonstrations with random labels from the label set barely lowers accuracy (0–5 points absolute, over 12 model–method pairs including GPT-3 and 26 datasets). What the demonstrations do supply is the input distribution, the label space and the input-label pairing as a format; keeping the format with only inputs or only labels retains up to 95% of the gain. Meta-training for in-context learning sharpens the effect. | Active — read 2026-10-09 (NOTE-607); worth reading as the measurement that separates what a demonstration supplies from whether its labels are right: across 12 model–method pairs up to GPT-3 and 26 classification and multi-choice datasets, replacing gold labels with random ones costs 0–5 points, while removing the input distribution, the label space or the paired format costs much more. The finding is empirical and aggregate; some dataset–model pairs lose up to 14 points, and generation tasks are not tested. |
| LIT-833 | Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity | Lu, Bartolo, Moore, Riedel and Stenetorp (2021), ACL 2022. With the same four SST-2 examples, the 24 orders range from near chance (~50%) to above 85% accuracy, in GPT-2 and GPT-3 from 0.1B to 175B. Good orders are model-specific (pairwise Spearman correlations between sizes all between −0.35 and 0.27, on one example set), the variance persists with up to 32 examples and after calibration, and failing orders mostly predict one label. Ranking orders by the label entropy they induce on a self-generated probing set recovers an average 13% relative gain across eleven classification tasks. | Active — read 2026-10-09 (NOTE-626); worth reading as the measurement that the order of the same few-shot examples, with content held fixed, can move a language model from near chance to near the fine-tuned state of the art, at every size from GPT-2 Small to GPT-3 175B. Good orders do not transfer between model sizes (rank correlations near zero), more examples do not remove the variance, and failing orders mostly collapse the predicted label distribution. The rest of the paper is a selection method (entropy over a self-generated probing set) whose interest here is secondary. |
| LIT-844 | Quantum-Like Contextuality in Large Language Models | Lo, Sadrzadeh & Mansfield (2024), ARXIV-2412.16806, Proc. R. Soc. A 481:20240399 (2025). Instantiates a three-sentence anaphora schema with the support of a PR prism 51,966,480 times from Simple English Wikipedia and takes referent probabilities from BERT: 0.148% of the models are sheaf-contextual by a signalling-corrected test and 71.1% are CbD-contextual. The support is fixed by the schema, so each verdict turns on how uncertain BERT is in three sentences. | Active — read in full 2026-10-09 (NOTE-648) from the arXiv preprint (the journal version could not be reached); worth reading as the large-scale test of language-model output for contextuality, and for what its numbers rest on. A PR-prism schema of three sentences (“It is X1 and the same one is X2”, ”… X2 … same … X3”, ”… X3 and the other one is X1”) is instantiated with 866,108 Simple English Wikipedia noun pairs and their adjectives; BERT gives one referent probability per sentence, and the support is fixed by the schema. Of 51,966,480 models, 77,118 (0.148%) pass the signalling-corrected sheaf test (SF < 1/6) and 36,938,948 (71.1%) the CbD test (Δ < 2). Both verdicts are fixed by three numbers per instance; the CbD one holds whenever BERT is uncertain and roughly consistent across the three sentences. The link to embedding distance is a softmax identity plus weak correlations (R² ≤ 0.08). |
| LIT-849 | Relative representations enable zero-shot latent space communication | Moschella, Maiorca, Fumero, Norelli, Locatello & Rodolà (2022; ICLR 2023), arXiv:2209.15430. Re-encodes each sample as its cosine similarities to a fixed set of anchor samples, which is invariant by construction to every cosine-preserving map of the latent space (rotations, reflections, rescaling), not to translations or general linear maps. With anchors matched across two encoders (parallel anchors), a decoder trained on one encoder’s relative vectors runs on another’s with no training: across seeds, architectures, latent sizes and languages. Stitching works far better than with raw latents but is lossy and uneven, and the assumption that independently trained spaces differ by an angle-preserving map is supported only approximately. | Active — read in full 2026-10-09 (NOTE-652); worth reading as the source of “relative representations”: a sample is re-encoded as its vector of cosine similarities to a fixed set of anchor samples, and a decoder trained on that vector runs, without any training, on a different encoder that sees corresponding anchors. Read it knowing what is guaranteed and what is observed. The invariance is by construction and one line long: the vector is unchanged by any map that preserves cosines, that is rotations, reflections and rescaling (and, though the paper does not say so, any per-sample positive rescaling), but not by translations, which it assumes normalisation removes, nor by general linear maps. That training noise, a change of architecture or a change of language moves a latent space only by such a map is an assumption, supported indirectly and only approximately: across FastText and Word2Vec the relative vectors agree at cosine 0.86 but share only 34–39% of their ten nearest neighbours, and between two ViTs on CIFAR-10 11–12%. Zero-shot stitching is not correspondence-free: it needs the anchors matched across the two sides (parallel anchors), which here means 300–768 translated or aligned sentence pairs or the same images. It beats absolute stitching everywhere absolute stitching can be run, but loses a great deal against the unstitched model in places: 98.6 to 80.5 ± 21.1 F1 on DBpedia across architectures, 90.1 to 65.7 for a Japanese encoder under an English decoder, 72.6 to 37.4 on ImageNet when ViT features feed a RexNet decoder. On a model that is already multilingual (XLM-R) it is worse than doing nothing. |
| LIT-850 | Revisiting Model Stitching to Compare Neural Representations | Bansal, Nakkiran & Barak (2021), NeurIPS 34. Revives Lenc and Vedaldi’s model stitching (LIT-363) as a measure of representations: the bottom of one frozen network feeds the top of another through a trained 1×1 convolution, and the change in test error is the “stitching penalty”. Independently seeded networks, and supervised and self-supervised ResNet-50s, stitch at all layers within a few points, where CKA says they differ. Representations trained on more data plug into a weaker network and lower its error; for width and training time this happens only near the top. All empirical, single runs, mostly CIFAR-10. | Active — read in full 2026-10-09 (NOTE-653); worth reading as the paper that made model stitching a standard way of comparing trained networks, and as the source of “stitching connectivity”: two networks of one architecture, trained from different seeds, can be joined at any layer by a learned 1×1 convolution with little loss of test accuracy. Read it knowing that it proves nothing. Its definitions are informal (“low penalty”, “comparable”), stitching connectivity is an informal conjecture, and every result is a single run, mostly ResNet-18 on CIFAR-10, read off plots with no error bars. Its “more is better” ordering is an ordering by one task’s loss through a fixed top network and a restricted map class, not an ordering by information: the paper itself says a pixel shuffle, which loses nothing, can stitch worse, and a negative penalty means the fitted map did not reproduce the layers it replaced. The width and training-time improvements appear only near the top of the network; at mid layers the wider and longer-trained representations stitch slightly worse. |
| LIT-855 | Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models | Karkada, Simon, Bahri and DeWeese (2025), ARXIV-2502.09863, NeurIPS 2025. The quartic approximation of word2vec’s loss, under symmetric reweighting with constant aggregate weight, is exactly unweighted factorisation of M*, the co-occurrence matrix’s relative deviation from independence, and from small initialisation gradient flow learns M*‘s eigenvectors one at a time in sigmoidal steps. The embeddings this predicts match word2vec’s features and benchmark scores far better than truncated PMI or PPMI factorisations do. | Active — read 2026-10-09 (NOTE-660); worth reading as the first closed-form account of what a contrastive word-embedding algorithm learns under a rank constraint, and in what order. Replacing word2vec’s logistic loss by its quartic Taylor approximation, and choosing symmetric reweightings whose aggregate weight is constant, turns the objective into unweighted symmetric matrix factorisation of M*, the matrix of relative deviations of word co-occurrence from independence (Theorem 1, proved). From vanishing random initialisation, gradient flow then learns the eigenvectors of M* one at a time, each by a sigmoidal step in time (ln(λ_k/σ²))/λ_k (Result 3, a derivation that drops terms, not a proof). The resulting embeddings track word2vec’s features and benchmark scores closely, and much more closely than truncated factorisations of PMI or positive PMI. The analogy part is empirical: difference vectors of semantic pairs look like a spiked random matrix, and the spike’s strength tracks analogy accuracy. |
| LIT-856 | Small Singular Values Matter: A Random Matrix Analysis of Transformer Models | Staats, Thamm & Rosenow (2024; NeurIPS 2025). Taking the Marchenko–Pastur law as the zero-information null for a weight matrix, trained transformer matrices (BERT, Pythia-410M, Llama-3.1-8B) show outliers above the bulk and, in rectangular matrices only, below it, since only there does the null spectrum have a positive lower edge. In the MLP projections the singular vectors of both kinds of outlier align with activation-covariance eigenvectors, and zeroing the smallest decile of a rectangular matrix can hurt more than zeroing most larger ones. A linear teacher–student ensemble yields a below-bulk outlier when the noise is suppressed along a learned direction. | Active — read 2026-10-09 (NOTE-659); worth reading here as the paper that uses the Marchenko–Pastur law as a null model for where a trained weight matrix holds information, and so notices what a magnitude-ordered reading misses: a rectangular matrix’s null spectrum has a lower edge above zero, so learning can push singular values below the bulk as well as above it. In three transformers the departures line up with activation-covariance directions, at the lower end in the MLP up and gate projections, and zeroing the smallest decile of a rectangular matrix can cost more than zeroing most larger deciles; in square matrices it never does. A two-layer linear teacher–student ensemble produces such a lower outlier when the loss suppresses the noise along a learned direction. The evidence is measured, not proved, and it is uneven: overlap and ablation importance disagree for several matrix types, and the decile ablation does not isolate the outliers it is meant to test. |
| LIT-858 | Towards Understanding Grokking: An Effective Theory of Representation Learning | Liu, Kitouni, Nolte, Michaud, Tegmark and Williams (2022), NeurIPS 2022. In a toy model that decodes the sum of two learned embeddings, grokking is representation learning: an unseen pair is answered correctly when its embedding sum equals a trained pair’s, and the critical training fraction (about 0.4 for addition with p = 10) is where the training set’s “parallelogram” equations first fix the embedding up to translation and scale. A posited effective loss on the normalised embeddings predicts that threshold and, through its third Hessian eigenvalue, the time to structure. Phase diagrams over decoder learning rate and weight decay place grokking between comprehension and memorization in the toy, a transformer and MNIST. | Active — read 2026-10-09 (NOTE-662); worth reading as an early mechanistic account of when grokking networks generalise, in a toy model that sums two learned embeddings and decodes the sum. An unseen pair is predicted correctly when its embedding sum coincides with a trained pair’s (a “parallelogram”), and the critical training fraction is where the training set’s parallelogram equations first fix the embedding up to translation and scale. The account is exact for the toy (1D embeddings, addition hard-coded in the architecture, an effective loss posited and derived only for a linear decoder); the transformer and MNIST evidence is qualitative (PCA pictures and phase diagrams), and the four “phases” are defined by accuracy thresholds, not by any order parameter. |
| LIT-859 | Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers | Mainali and Teixeira (2025), ARXIV-2504.12916. For a single linear attention layer trained by small-step SGD on in-context regression ℝᵈ → ℝᵈ, with the input covariance and every task matrix sharing one eigenbasis and the weights assumed to stay in it, the averaged dynamics decouple into one logistic equation per eigenmode: mode α is learned on a timescale τ_α = (ηP s_α²)⁻¹, giving plateaus and cliffs in the loss, and settles where p_α q_α = N / ((N+1)s_α + Tr S), so the trained predictor applies an inverse, finite-N-corrected training covariance to the context’s empirical covariance. A scaling symmetry conserves ‖p₂‖² − ‖q₁‖². Effective-rank and subspace measures applied to attention-only transformers and a grokking model are offered as qualitative parallels. | Active — read 2026-10-09 (NOTE-658); worth reading as an exactly solved case of the slow learning that produces a fast in-context learner: for one linear-attention layer trained by gradient flow on in-context vector regression, with tasks and inputs sharing one eigenbasis, each eigenmode of the input covariance learns by its own logistic curve on a timescale ∝ 1/s_α², the trained product of the two weight blocks settles at the inverse of a finite-context-corrected training covariance, so the learned in-context predictor is a preconditioned estimator (the fixed point Ahn et al. and Zhang et al. had found), and ‖p₂‖² − ‖q₁‖² is conserved. The decoupling rests on an assumed spectral alignment and on large N and many tasks. The second half, measures applied to attention-only transformers and a grokking model, is qualitative and correlational: no seeds, error bars or controls are reported, and “mechanistic explanation” overstates it. |
| LIT-860 | Symmetry in language statistics shapes the geometry of model representations | Karkada, Korchinski, Nava, Wyart and Bahri (2026), ICML 2026 (PMLR 306:56137–56172). If co-occurrence among words on a latent continuum depends only on their separation, a full-rank spectral word embedding places them on Fourier modes of the latent coordinate: circles for cyclic concepts, open curves with harmonic ripples for intervals, curved sheets for maps, all proved for that model with the exponential kernel solved exactly. Wikipedia’s month and year statistics fit the symmetry; Gemma 2 and EmbeddingGemma show matching PCA pictures; the geometry survives removing the words’ mutual co-occurrences, which a seasonal latent variable shared by many words explains. | Active — read 2026-10-09 (NOTE-664); worth reading as a derivation of the circles, rippled curves and curved sheets found in language-model representations of months, years and places from one property of the data: when the co-occurrence of two words on a latent continuum depends only on their separation, the matrix the embedding factorises is circulant or Toeplitz, so its eigenvectors are sinusoids of the latent coordinate. For a full-rank spectral word embedding the paper proves the principal components are Fourier modes with amplitudes set by the kernel’s Fourier transform, solves the open-boundary exponential kernel exactly (a Sturm–Liouville problem), and bounds linear decoding of the coordinate by the missing Fourier tail (ε² ≲ r^(−1/D)). It measures the symmetry in Wikipedia statistics, shows the month circle survives zeroing every month–month co-occurrence, and explains that by a seasonal latent variable that couples many words. The “word embeddings” are explicit factorisations of the co-occurrence matrix, not trained networks; the language-model evidence is PCA pictures of Gemma 2 2B and EmbeddingGemma; the robustness argument is a perturbation sketch. |
| LIT-862 | A mathematical theory of semantic development in deep neural networks | Saxe, McClelland and Ganguli (2019), PNAS 116(23):11537–11546, ARXIV-1810.10531. A two-layer linear network trained from small weights learns the singular modes of its input–output correlations one at a time, each in a sigmoidal step at a time of order (τ/s)ln(s/ε), where a shallow network learns them all at once. With data generated on a tree this gives coarse-to-fine differentiation in stages, transient illusory correlations and narrowing generalisation; the SVD also defines typicality, prototypes and category coherence, and minimum-norm weights fix the hidden similarity matrix across networks. | Active — read 2026-10-09 (NOTE-666); worth reading as the paper that turns the exactly solved learning dynamics of a two-layer linear network into a theory of semantic development. Trained by gradient descent from small weights on whitened inputs, the network learns the singular modes of the item–feature correlation matrix one at a time, each by a sigmoid at a time of order (τ/s)ln(s/ε), while a shallow network learns every mode on one exponential timescale. Hierarchically generated data have singular values that fall with depth in the tree, so broad distinctions are learned first in stage-like transitions, individual predictions can rise and fall between stages, and inductive generalisation narrows with development. The same SVD gives typicality, prototypes and a coherence threshold (a BBP transition), rings give Fourier modes, and minimum-norm weights, which small initialisation approximates, give identical hidden similarity matrices across networks. The trajectory is exact only from decoupled, balanced initial weights with white inputs; from random small weights it is checked by simulation, and the match to children and cortex is qualitative. |
| LIT-863 | On the Emergence of Linear Analogies in Word Embeddings | Korchinski, Karkada, Bahri and Wyart (2025), NeurIPS 2025. If each word is a vector of d binary attributes whose effects on co-occurrence are independent and multiplicative, the co-occurrence ratio is a Kronecker product and its logarithm, the PMI, has rank at most d + 1 with eigenvectors affine in the attributes, so a spectral PMI embedding realises parallelogram analogies exactly; the ratio itself does so only for weak, narrowly spread attribute signals. The structure survives noise, vocabulary pruning and deletion of all pairs of a relation, by perturbation arguments checked in simulation and on Wikipedia. | Active — read 2026-10-09 (NOTE-665); worth reading as a generative model of word co-occurrence in which the parallelogram analogies of word embeddings are derived rather than postulated. Each word is a point of the hypercube {−1, +1}^d of binary attributes, and each attribute multiplies its co-occurrence by an independent 2 × 2 factor, so the ratio M = P(i,j)/P(i)P(j) is a Kronecker product (proved) and the PMI log M is a bilinear form in the attributes, of rank at most d + 1 with eigenvectors affine in them. A spectral embedding of the PMI is then a linear image of the attribute hypercube and its analogies hold exactly at any signal strength; one of M holds them only when the signals are weak, narrowly spread and K ≤ d + 1, because M’s spectrum mixes in products of attributes. Robustness to noise, to pruning the vocabulary and to deleting every pair of an analogy relation is argued from eigenvalue perturbation bounds and checked numerically. The embeddings are eigendecompositions of a co-occurrence matrix, not trained word2vec or GloVe, and the agreement with Wikipedia is in the shapes of spectra and accuracy curves; no attribute is recovered from data. |
| LIT-864 | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks | Saxe, McClelland and Ganguli (2014), ARXIV-1312.6120, ICLR 2014. Solves gradient-descent learning in deep linear networks with whitened inputs from decoupled initial weights: each singular mode of the input–output correlation is learned by a sigmoid at a time of order (τ/s)ln(s/ε), with conserved differences of squared layer strengths. For any depth, the delay depth adds is set by the initial end-to-end mode strength, finite as depth grows when that strength is of order one and the step size is scaled with depth. Argues, with simulations, that scaled Gaussian layers are not isometric in product while orthogonal ones are, and that tanh networks with orthogonal weights near gain 1 stay close to isometric. | Active — read 2026-10-09 (NOTE-667); worth reading as the paper in which the learning dynamics of deep linear networks were solved exactly from decoupled starts at any depth (Fukumizu had solved one hidden layer for another class of starts) and the stepwise, mode-by-mode picture was set out that later work calls Saxe-style. Gradient descent on a linear network with one hidden layer and whitened inputs is a cubic system whose modes cooperate within a singular mode of the input–output correlations and repel across modes; from decoupled initial weights each mode’s strength follows a sigmoid reaching s at a time of about (τ/s)ln(s/ε), with a conserved difference of squared layer strengths and, in the appendix, the unbalanced case. Beyond what the 2019 restatement keeps, it solves arbitrary depth from decoupled starts: depth slows learning through the initial end-to-end mode strength, so from strength of order one the number of iterations stays finite as depth grows once the step size is scaled down with depth, while from strength ε the delay grows like s/ε. It argues that scaled Gaussian layers preserve norms but not isometry, that random orthogonal layers give depth-independent learning times on MNIST, and that tanh networks with orthogonal weights near gain 1 keep the end-to-end Jacobian’s singular values of order one. Random initial weights, the role of isometry in learning time and everything nonlinear rest on simulation, not proof. |
| LIT-866 | Hyperbolic Entailment Cones for Learning Hierarchical Embeddings | Ganea, Bécigneul and Hofmann (ICML 2018), PMLR 80:1646–1655. Represents a partial order in the Poincaré ball by nested cones: v is below u when v lies in u’s cone. Nesting forces apertures of at most π/2 that narrow toward the boundary (r sin ψ/(1 − r²) non-increasing), so cones are undefined near the origin; the widest family is sin ψ = K(1 − r²)/r, with a closed-form membership test. Overlapping cones admit several parents. On WordNet noun closure link prediction in 5–10 dimensions it beats Poincaré and order embeddings once some closure edges are in training; with none, all methods fail. | Active — read 2026-10-09 (NOTE-671); worth reading as the paper that represents a partial order, not only a tree, by containment of regions in hyperbolic space. Each concept is a point x of the Poincaré ball and owns a cone S_x, the image under the exponential map of the tangent vectors within an angle ψ(x) of the outward ray through x; v counts as entailed by u when v lies in S_u. Requiring the cones to be axially symmetric, rotation invariant, continuous in aperture and nested (v in S_u implies S_v inside S_u) forces ψ ≤ π/2 (Lemma 2) and makes r sin ψ(r)/(1 − r²) non-increasing (Theorem 3), so no cone can be defined near the origin; the widest family is sin ψ(x) = K(1 − ‖x‖²)/‖x‖ outside a ball of radius ε, and membership is a closed-form angle test (Theorem 5). That family is transitive (Theorem 4), but the proof is only said to be “similar” and is not given. Cones of incomparable points can overlap, so a node may sit below several parents. The evidence that learned cones fit a DAG is one WordNet-noun experiment: held-out edges of the transitive closure, test F1 at 5 and 10 dimensions, single runs, where hyperbolic cones lead once 10% or more of the closure is in training but every method fails (F1 27–43%) when only the transitive reduction is, and Euclidean cones initialised from Poincaré embeddings match the hyperbolic ones. |
| LIT-870 | Hyperbolic Diffusion Embedding and Distance for Hierarchical Representation Learning | Lin, Coifman, Mishne and Talmon (2023), ICML 2023 (PMLR 202), ARXIV-2305.18962. Recovers a latent tree from point-cloud or graph data without learning: diffusion densities at dyadic time scales are embedded in a product of Poincaré half-spaces, and the ℓ1 product distance, a scale-weighted Hellinger sum, is proved equivalent up to constants to a thresholded snowflake min{1, d_T^(2α)} of the tree metric when the operator behaves like the heat kernel on a tree. It keeps local tree neighbourhoods (best MAP on the benchmarks) but not global distances (higher distortion than hyperbolic MDS). | Active — read 2026-10-09 (NOTE-675); worth reading as a construction that recovers a latent tree from unlabelled data by distances alone, with a proof of what is recovered. Each point is represented by its diffusion densities P^(2^−k) e_i at dyadic times, each density is placed in a Poincaré half-space at a height set by its scale, and the hyperbolic diffusion distance is the ℓ1 sum over scales of the hyperbolic distances, 2 sinh⁻¹(2^(1−kα)‖φ_i^k − φ_j^k‖) with φ the square roots of the densities, a scale-weighted Hellinger distance. Theorem 1 (Appendix C, after Leeb and Coifman) says that for the heat kernel on a tree this is equivalent up to constants to min{1, d_T^(2α)}, 0 < α < 1/2: a thresholded snowflake of the tree metric, not the tree metric, and saturated beyond a fixed radius; the main text states it without the threshold. The tree heat-kernel bounds are stated, not proved, and one step of the Hölder bound (Proposition C.5) holds only for tree distances of at least 1, so the admissible range of α is not secured. Empirically it gives the best mean average precision on five graph benchmarks and two single-cell datasets and worse average distortion than hyperbolic MDS or Sarkar’s construction: local neighbourhoods are kept, global distances are not. Nothing in it concerns attributes or linear directions. |
| LIT-871 | Hyperbolic Representation Learning: Revisiting and Advancing | Yang, Zhou, Ying, Chen and King (2023), ICML 2023, PMLR 202. Tracks the hyperbolic distance from each node to the origin and finds that hyperbolic models trained on ordinary task losses do not by themselves place a tree’s root nearest the origin or spread its levels outward: on synthetic 3-ary trees the root sits well off the minimum and the distances are roughly bell-shaped, ordering node pairs by level only 69–75% of the time. Proposes HIE, which recentres the embedding on its hyperbolic centroid and pushes nodes outward; it improves that ordering and benchmark scores, most on the least tree-like graphs. | Active — read 2026-10-09 (NOTE-669); worth reading as an empirical check, on synthetic trees and four benchmark graphs, of the assumption that a hyperbolic model trained on an ordinary task loss lays out the data’s hierarchy by itself, root near the origin and leaves near the boundary. It does so only partly. On two 3-ary trees of 1,093 nodes, a hyperbolic graph convolutional network (HGCN) trained for link prediction or node classification puts the root nowhere near the smallest distance to the origin, and the nodes’ distances to the origin come out roughly bell-shaped rather than piling up at the leaves; read off as levels, those distances order random node pairs correctly 69–75% of the time. The proposed fix, HIE, recentres the embedding on its hyperbolic centroid and adds a loss that pushes every node outward in proportion to its current distance. It raises that ordering to 75–84% and lifts benchmark scores, most on the least tree-like graphs (Citeseer, Cora). There is no theory of why task losses fail to produce the hierarchy, and the fix itself uses no level information. |
| LIT-873 v2 | Mutual information, neural networks and the renormalization group | Koch-Janusz and Ringel (2017; Nature Physics 14, 578–582, 2018). Defines the relevant degrees of freedom of a spatial block as the coarse variable that maximizes mutual information with the environment beyond a buffer, and learns it with restricted Boltzmann machines from Monte Carlo samples (the RSMI algorithm). It recovers Kadanoff block spins in the 2D Ising model and the electric-field variables of the dimer model, ignores decoupled noise, and, iterated, gives the Ising flow and ν ≈ 1.0 ± 0.15; a distribution-fitting RBM does none of this. The link to a short-ranged effective Hamiltonian is proved only in 1D. | Active — read 2026-10-09 (NOTE-674); worth reading as the paper that turns “the relevant degrees of freedom” of real-space renormalization into an objective that can be optimized: a block’s coarse variable is chosen to maximize its mutual information with the system beyond a buffer around the block. Trained on Monte Carlo samples alone, the network rediscovers Kadanoff’s block spin for the 2D Ising model, finds the low-momentum electric fields of the 2D dimer model while ignoring a strongly patterned but decoupled noise, and, iterated, reproduces the Ising flow away from T_c (located to about 1%) with ν ≈ 1.0 ± 0.15. An RBM trained instead to fit the data distribution latches on to the noise. The equivalence of the objective with a short-ranged effective Hamiltonian is argued, and shown only for 1D and quasi-1D chains; in 2D the evidence is the two models, whose answers were known beforehand. |
| LIT-874 | Representation Tradeoffs for Hyperbolic Embeddings | Sala, De Sa, Gu and Ré (ICML 2018), PMLR 80:4460–4469. Trees embed in the hyperbolic plane with distortion 1 + ε (Sarkar’s construction), but a hyperbolic distance d costs about d bits, so the embedding needs Θ((ℓ/ε) log deg_max) bits per coordinate in the plane: logarithmic in branching, linear in depth (lower bound proved for H², with an approximation). Higher dimension, via coding-theoretic child placement, divides the upper bound by r. Also an exact hyperbolic MDS by PCA after centring at a new mean, with a perturbation bound growing as sinh²(diameter), and a local-convexity condition for hyperbolic PGA. | Active — read 2026-10-09 (NOTE-679); worth reading as a quantitative account of why trees fit hyperbolic space and what the fit costs. Following Sarkar, any finite tree embeds in the hyperbolic plane with worst-case distortion at most 1 + ε by placing each node’s children evenly on a hyperbolic circle of radius τ around it; but a hyperbolic distance d needs about d bits to write down, so the coordinates need O((ℓ/ε) log deg_max) bits, ℓ the longest path, and an explicit tree of deg_max chains shows Ω((ℓ/ε) log deg_max) is needed in the plane. Precision grows with the logarithm of the branching and linearly with depth: hyperbolic space is cheap for short bushy hierarchies and dear for long chains. In r dimensions, children placed on hypercube vertices chosen by a Hadamard code divide the cost by r up to r ≈ log deg_max (upper bound only). For general metrics it gives an exact hyperbolic MDS: cosh of the distances is uuᵀ − XXᵀ once the points are centred at a new pseudo-Euclidean mean, so PCA recovers them. The lower bound’s proof uses a small-angle approximation and covers only the plane; the PGA guarantee is local convexity, not global recovery; the experiments, including the WordNet MAP of 0.989 in two dimensions, are reported at ε = 0.1, where WordNet takes 2,877 bits per coordinate, while the introduction quotes the ε = 1 figure of about 500. |
| LIT-877 | How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model | Cagnetta, Petrini, Tomasini, Favero and Wyart (2023; Phys. Rev. X 14, 031001, 2024). Introduces the Random Hierarchy Model, a random context-free grammar of depth L with m synonymic expansions per symbol. Shallow and lazy networks need a fixed fraction of all data, exponential in the input dimension; deep feature-learning CNNs learn from about n_c m^L examples, polynomial in it, which is where hidden layers become invariant to swapping synonyms and where the class statistics of input patches, equal for synonyms, emerge from sampling noise. Without those correlations deep networks fail too. The law is measured; the link to correlations is a scaling argument plus a one-step-gradient model. | Active — read 2026-10-09 (NOTE-677); worth reading as the paper that introduces the Random Hierarchy Model, a random context-free grammar of fixed depth L and branching s in which each class and each higher-level feature has m interchangeable (“synonymic”) lower-level representations, and as a measured and partly derived account of how many examples a deep network needs to learn it. Two-layer networks and lazy (kernel-regime) deep networks need a fixed fraction of all the data, which is exponential in the input dimension d = s^L; deep convolutional networks in a feature-learning parametrisation need P* ≈ n_c m^L, a power of d. That number is measured, not proved. It coincides with the training-set size at which the networks’ hidden layers become insensitive to swapping synonyms, and with the size at which the class-conditional frequencies of input s-tuples, which are equal for synonyms, rise above sampling noise; a one-step gradient argument for a two-layer network on one patch links the last two. Removing the input–label correlations, with the hierarchy kept, makes deep networks fail again. The hierarchy is one of constituency (parts composing wholes), not of attributes implying attributes. |
| LIT-878 v2 | Statistical Physics through the Lens of Real-Space Mutual Information | Gökmen, Ringel, Huber and Koch-Janusz (2021; Phys. Rev. Lett. 127, 240603). Replaces the RBM estimator of LIT-873 with a neural InfoNCE lower bound co-trained with a convolutional coarse-graining filter (RSMI-NE), and uses the optimal filters at a single buffer size, not an iterated flow, as lattice forms of the most relevant operators. On the interacting dimer model they are the columnar and plaquette order parameters below T_BKT and the electric fields at high T; the maximal information marks the phases; one fitted scaling dimension (1.00037 against 1) is given. The optimality theorem is cited, not proved here. | Active — read 2026-10-09 (NOTE-681); worth reading as the paper that turns the real-space mutual-information principle of LIT-873 from an iterated coarse-graining into a single-shot extraction of operators: with the mutual information estimated by a neural InfoNCE bound and the coarse-graining a convolutional filter trained against it, the filters that best compress what a block shares with its distant environment are, across the whole phase diagram of the interacting dimer model, the columnar and plaquette order parameters below the BKT transition and the electric fields at high temperature, and the plaquette filter’s correlator gives a scaling dimension of 1.00037 against the predicted 1. The maximal information itself, log 4 in the ordered phase and decaying algebraically with the buffer in the critical one, traces the phase diagram. The theorem that the optimal filters are the most relevant operators is cited to another paper, and the model’s operators were known beforehand. |
| LIT-879 v2 | Hyperbolic Neural Networks | Ganea, Bécigneul and Hofmann (NeurIPS 2018), ARXIV-1805.09112. Builds neural-network layers in the Poincaré ball from Möbius gyrovector operations: closed-form exponential and logarithmic maps, parallel transport from the origin as a scaling, geodesic hyperplanes with a closed-form point-to-hyperplane distance, and from them hyperbolic logistic regression, feed-forward layers and GRUs, all reducing to the Euclidean ones as the curvature goes to zero. Linear maps and nonlinearities are lifted through the origin’s tangent space, so the curvature acts only through biases and the output distances. Gains over Euclidean models are clear only on a synthetic tree-like task and on separating WordNet subtrees in low dimensions. | Active — read 2026-10-09 (NOTE-686); worth reading as the paper that gives the Poincaré ball the operations a neural network is built from, and so shows exactly where hyperbolic geometry enters such a network and where it does not. From Ungar’s gyrovector formalism (Möbius addition and scalar multiplication) it derives the exponential and logarithmic maps at any point, parallel transport from the origin as the scaling λ₀/λ_x (Theorem 4), geodesic hyperplanes as exp_p of a tangent orthogonal complement, and the closed-form distance to one (Theorem 5); all of these check out numerically. A linear map or pointwise nonlinearity is lifted as exp₀ ∘ f ∘ log₀, so a stack of such layers is one Euclidean network in log₀ coordinates; the curvature acts only through the bias translations between layers and the distance-based softmax. The evidence is small: 5-dimensional sentence encoders on SNLI, where hyperbolic models are on par or worse, and on a synthetic noisy-prefix task, where they win most when the noise is lowest, each cell the best of three runs; and hyperbolic logistic regression separating WordNet subtrees in pre-trained Poincaré embeddings better than Euclidean hyperplanes do, mostly in 2 to 5 dimensions. |
| LIT-880 | How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model | Tomasini and Wyart (2024; ICML 2024, PMLR 235). Pads each informative feature of the Random Hierarchy Model with uninformative positions, so the label is unchanged by discrete displacements of features, a stand-in for smooth deformations. Deep networks learn this sparse model from a number of examples polynomial in the input dimension, measured as (s0 + 1)^L n_c m^L (times s^(L/2)) without weight sharing and (s0 + 1)^2 n_c m^L with it, and become insensitive to synonym swaps and to feature displacements at the same training-set size as they learn the task, which the paper offers as the reason deformation stability tracks test error. Fitted laws; the image case is by analogy. | Active — read 2026-10-09 (NOTE-684); worth reading as the sparse extension of the Random Hierarchy Model (LIT-877) and as a model-based account of why a network’s insensitivity to small smooth deformations of its input tracks its test error. Each informative feature of the RHM is padded with s0 uninformative “empty” positions and may sit anywhere in its sub-patch, so the data become very sparse (s^L informative positions among (s(s0 + 1))^L) and the label is unchanged by discrete displacements of the informative features, a stand-in for diffeomorphisms of an image. On this Sparse Random Hierarchy Model, deep locally connected and convolutional networks learn from a number of examples polynomial in the input dimension: measured, P* ≈ s^(L/2)(s0 + 1)^L n_c m^L without weight sharing and P* ≈ C(s0 + 1)^2 n_c m^L with it. At about that training-set size the hidden layers become insensitive both to swapping synonymous tuples and to displacing features, so the two invariances and good performance arrive together; the paper offers this as the explanation of the correlation between deformation stability and test error measured on CIFAR-10. The laws are fitted, the (s0 + 1)^L factor has a one-step-gradient argument for locally connected networks, the CNN’s (s0 + 1)^2 is unexplained, and the image case is an analogy: no synonym sensitivity was measured on images. |
| LIT-881 v2 | Optimal Renormalization Group Transformation from Information Theory | Lenggenhager, Gökmen, Ringel, Huber and Koch-Janusz (2018; Phys. Rev. X 10, 011037, 2020). Proves that a block coarse-graining keeping all the mutual information the block shares with the system beyond a buffer makes the coarse measure factorize across the block, so a finite-range Hamiltonian gains no range (1D; D dimensions under an extra assumption) and 1D disorder gains no correlations across it. For blocks of two spins of the 1D clean and dilute random Ising chain, where that condition is not met, couplings beyond nearest neighbour and disorder correlations shrink as more information is kept, vanishing at decimation. Encoding into a single binary spin explains why decimation wins in 1D and majority rule in a 2D toy model. | Active — read 2026-10-09 (NOTE-682); worth reading as the theory written for the RSMI algorithm of LIT-873: it proves that a coarse-graining which keeps all the information a block shares with the system beyond a buffer factorizes the coarse measure across that block, so a finite-range Hamiltonian does not gain range (in 1D; in D dimensions under an extra per-block assumption) and, for 1D disorder, no correlations are generated across the block. The full-capture condition is not met by the coarse-grainings it then studies: in its own 1D Ising example decimation keeps about half of the block’s information, and the decay of next-nearest-neighbour and four-spin couplings, and of disorder correlations, as information grows is shown numerically for blocks of two spins. Its other result is information-theoretic: a single binary block spin loses information by encoding, which is why decimation beats majority rule in 1D and majority rule wins in a 2D toy model. |
| LIT-882 | An exact mapping between the Variational Renormalization Group and Deep Learning | Mehta and Schwab (2014), arXiv preprint, never published in a journal. Identifies Kadanoff’s variational RG kernel with an RBM’s energy by T = −E + H, under which the coarse-grained Hamiltonian is the RBM’s hidden-marginal Hamiltonian for any parameters and an exact RG step is a perfect fit of the data. Illustrates it with a hand-built decimation network for the 1D Ising chain and a four-layer stacked RBM trained on 2D Ising samples near T_c, whose receptive fields are local and grow with depth. It concedes that, short of exactness, RG and RBM training use different variational schemes; LIT-873 shows the difference matters. | Active — read 2026-10-09 (NOTE-683); worth reading as the paper that first put deep restricted Boltzmann machines and Kadanoff’s variational renormalization group side by side, and as the position Koch-Janusz and Ringel (LIT-873) argue against. Its “exact mapping” is an identification: setting the RG kernel T = −E + H makes the coarse-grained Hamiltonian equal the RBM’s hidden-marginal Hamiltonian for every parameter value, and makes an exact RG step the same thing as a perfect fit of the data distribution. Away from that point the paper itself says the two procedures use different variational approximations, so the mapping does not show that training an RBM performs RG. The evidence that it does is a hand-built 1D decimation network and qualitative receptive fields of a stacked RBM on 2D Ising samples; no flow, fixed point or exponent is computed. |
| LIT-883 | Towards a theory of how the structure of language is acquired by deep neural networks | Cagnetta and Wyart (2024), NeurIPS 2024. In sequences generated by the Random Hierarchy Model, the correlation between two tokens shrinks by a factor of about m per level of their lowest common ancestor, and a sample of P sequences resolves it only above a noise floor of (v²P)^(−1/2), so a finite training set acts as an effective context window t*(P). Deep next-token learners show loss steps at P_ℓ ≈ v m^(2ℓ−1), where their representations come to encode the level-ℓ hidden symbol, and the steps add up to a power law. On character-level text the measured correlation decay predicts where the loss for a given context length levels off. | Active — read 2026-10-09 (NOTE-687); worth reading as the self-supervised sequel to the Random Hierarchy Model (LIT-877), and the first of that line in which the latent hierarchy is read off a co-occurrence matrix of observable tokens. For sequences generated by the RHM, the covariance between two token positions has zero mean over draws of the grammar and a root-mean-square size set by the depth ℓ of their lowest common ancestor, about ((1 − m/v^(s−1))/(v³ m^(2ℓ−1)))^(1/2) (derived for large v and m); the estimate from P samples carries noise of size (v²P)^(−1/2) (proved by a central-limit argument), so a finite training set resolves correlations only out to an effective distance t*(P). Deep transformers and CNNs trained to predict the last token show loss steps at about P_ℓ = v m^(2ℓ−1), where their hidden layers become insensitive to resampling the subtree below a level-ℓ symbol. That the networks use the correlations to build the hidden variables is argued, not proved. On character-level Shakespeare and WikiText-103 the measured correlation decays as t^(−β), and loss curves for different context lengths collapse under the t*(P) ~ P^(1/(2β)) it implies. No embedding, PMI or linear direction is studied. |
| LIT-885 | Relevance in the Renormalization Group and in Information Theory | Gordon, Banerjee, Koch-Janusz and Ringel (2020; Phys. Rev. Lett. 126, 240601, 2021). On a cylinder, with block and environment as slabs separated by a buffer of length L_B, the information-bottleneck encoder just past its first transition depends on the block only through the boundary weak value of the leading transfer-matrix eigenvector, the lowest-dimension primary at large circumference, and the transition is at β_c = (λ0/λ1)^(2L_B). Checked on a three-site critical Ising cylinder. With a large enough alphabet the encoder respects the data’s symmetry near that transition; a small one can break it. | Active — read 2026-10-09 (NOTE-690); worth reading as the theorem the real-space mutual-information programme (LIT-873, LIT-881, LIT-878) cites for its claim that the optimal coarse-graining picks out the most relevant operators. What it proves is narrower than that claim: for a short-range lattice model on a cylinder, with the block and the environment taken as whole slabs of the cylinder separated by a buffer, the information-bottleneck encoder just past its first transition depends on the block only through the boundary weak value of the leading transfer-matrix eigenvector, and the transition sits at β_c = (λ0/λ1)^(2L_B); for large circumference that eigenvector is the lowest-dimension primary. The planar block-in-a-shell geometry the algorithms use, the fixed-alphabet β → ∞ limit that is RSMI, and the regime beyond the first transition are not covered. A second result: past the first transition, with a large enough alphabet and no fine-tuning, the optimal encoder carries a representation of the data’s symmetry, and a two-symbol example with Z4 symmetry breaks it. |
| LIT-887 v2 | Why does deep and cheap learning work so well? | Lin, Tegmark and Rolnick (2016; Journal of Statistical Physics 168, 1223–1247, 2017). Argues that networks are cheap because physical data have low-order, local, symmetric Hamiltonians, and deep because data come from Markov generative hierarchies whose minimal sufficient statistics compose. Proves that a single hidden layer needs exactly 2^n neurons to multiply n inputs, against about 4n in a deep tree, and gives linear no-flattening examples. Holds that RG is supervised feature extraction, not unsupervised learning; arXiv v1–v2 carried a counterexample against LIT-882 that hit a typo in its Eq. 8, and was dropped later. | Active — read 2026-10-09 (NOTE-688); worth reading as the third position in the dispute over whether deep learning performs renormalization, and for two results that stand on their own. On the dispute it holds that RG is “essentially a feature extractor for supervised learning”: a coarse-graining is a renormalization only once the variables to keep are specified, and fitting the data distribution does not specify them. That holds, and it agrees with what LIT-882 concedes and LIT-873 shows; its sharper form was in an appendix of arXiv v1 and v2 only, dropped from v3 on and from the journal text, showing that a matched partition function does not give a matched distribution and (v2) that the exact trace condition is met by hidden variables that do not interact with the system at all. Its conclusion that RG has “little to do” with unsupervised learning is too strong: LIT-873 gets the relevant variables without labels, from the far environment. On its own merits: a two-variable product needs four neurons and an n-variable product needs exactly 2^n in one hidden layer against about 4n in a deep tree (proved, in a Taylor-matching sense, with weights that grow without bound); the minimal sufficient statistics of a Markov generative hierarchy compose level by level (proved, and nearly immediate); that physics favours low-order, local, symmetric Hamiltonians, and that this is why cheap networks work, is argued, not shown. |
| LIT-889 v2 | Symmetries and phase diagrams with real-space mutual information neural estimation | Gökmen, Ringel, Huber and Koch-Janusz (2021; Phys. Rev. E 104, 064106), the long companion of LIT-878. Derives the RSMI-NE estimator in full and applies it to the 2D Ising model (1 bit below T_c, exponential decay with the buffer above it, boundary filters at T_c), the interacting dimer model and a 1D non-equilibrium chipping model. Its new object is the ensemble of optimal filters from independent runs: it maps an RSMI-degenerate subspace whose projections show broken and emergent (U(1)) symmetries, and PCA of it recovers the pristine operators. Every model’s answer was known beforehand. | Active — read 2026-10-09 (NOTE-692); worth reading as the long companion of LIT-878 and the place where the real-space mutual-information method is shown on more than one model and its outputs are read as an ensemble: the maximal information counts the ordered sectors (exactly 1 bit in the 2D Ising ferromagnet, 2 bits in the columnar dimer phase), decays exponentially with the buffer in the Ising paramagnet and algebraically at criticality, and the optimal Ising filter goes uniform, boundary and random across T_c; 500 independent runs per temperature map an RSMI-degenerate subspace of filters whose projections show the broken C4 and translation symmetries of the dimer model and, at high temperature, its emergent U(1), and PCA of the ensemble recovers the pristine operators from a window where they never appear alone. The number of coarse variables is chosen where the information stops growing. A non-equilibrium chipping model is a one-figure validation, and its formal status is said to be missing. |
| LIT-891 | Comment on “Why does deep and cheap learning work so well?” | Schwab and Mehta (2016), arXiv comment, never published elsewhere. Reply to the appendix counterexample of Lin and Tegmark’s arXiv v1 (LIT-887): agrees that preserving the free energy does not recover the distribution, calls the “if and only if” of LIT-882’s Eq. 8 a typo (the trace condition implies ΔF = 0, not conversely), and shows that the counterexample violates the trace condition, so the mapping’s exact step is untouched. It does not reach the later objection that the trace condition fails to select the coarse variables. | Active — read 2026-10-09 (NOTE-694); worth reading as Mehta and Schwab’s own statement of how far their mapping (LIT-882) reaches, made in reply to the counterexample in arXiv v1 of Lin and Tegmark (LIT-887). It concedes two things: that preserving the free energy does not recover the distribution (“we entirely agree with this statement and never claimed otherwise”), and that the “if and only if” of LIT-882’s Eq. 8 is a typo, since the trace condition implies ΔF = 0 but not conversely. It argues one: that exact variational RG means the trace condition Tr_h e^T = 1 for every v, which the counterexample violates (checked), so the mapping’s Eq. 22 is untouched. That reply is right against v1. It locates the mapping’s content at the trace condition alone, which is the scope THEORY-197 gives it, and it does not answer the point Lin and Tegmark added in their v2 a fortnight later, that the trace condition is also met by hidden variables that do not interact with the system (THEORY-202). The “typo” did work in LIT-882: its Section I derives the exactness condition from ΔF = 0 through the direction the comment withdraws. |
| LIT-935 v2 | The Origins of Representation Manifolds in Large Language Models | Modell, Rubin-Delanchy & Whiteley (2025). A feature is modelled as a compact metric space and its representation as a map onto the unit sphere. If the map is continuous and injective, the representation is homeomorphic to the feature. If cosine similarity is also locally a smooth decreasing function of squared feature distance, every path on the representation manifold is a fixed multiple of the feature path’s length, so on-manifold geodesics recover the feature’s intrinsic geometry. Both conditions are assumed. Tests on GPT-2-small’s years and on text embeddings of dates and colours find homeomorphism, and isometry for years only on a logarithmic scale. | Active — read 2026-10-14 (NOTE-802) in full; worth reading for Theorem 1: if a feature is a compact metric space mapped continuously and injectively onto the unit sphere, and cosine similarity is locally a C² decreasing function of squared feature distance, path lengths on the representation equal the feature’s up to the scale √(−2g′(0)), so geodesics recover its intrinsic geometry. Read it knowing that both hypotheses are assumed rather than derived, that the title’s “origins” are an informal argument, and that the tests supply a candidate metric by hand: GPT-2’s years are homeomorphic to a linear and a logarithmic scale, and only the metric test separates them. Held as THEORY-284. |
| LIT-936 v3 | The information geometry of large language models is shared, learned, and controllable | Picozzi (2026), arXiv preprint. Each model is given the matrix of Fisher–Rao distances between its next-token laws over a fixed context battery. That matrix is a function of behaviour alone, so it is invariant to output-preserving reparameterisation, which activation geometry is not. Across ten models of three architecture kinds it agrees at a mean rank correlation of 0.88, against 0.62 for activations. This is proved to follow from each model’s small excess risk to a common law, and it is measured to come from model disagreement lying nearly orthogonal to the relational structure. The pulled-back metric gives minimum-disturbance steering whose advantage over Euclidean steering it predicts (median measured/predicted 0.967). | Active — read 2026-10-14 (NOTE-755) in full; worth reading as a measured, coordinate-free relational signature. The matrix of Fisher–Rao distances between a model’s next-token laws over a context battery is fixed by behaviour and invariant to any output-preserving reparameterisation, while activation geometry has a GL(d) gauge. It agrees across ten transformer, state-space and recurrent models at a mean rank correlation of 0.88, against 0.62 for mid-layer activations, and a probe on distances to 40 anchors transfers between models. Read it knowing three things. The signature is identified because it is a function of the observed laws, the in-probe case QUESTION-027 calls automatic. Its agreement is proved only as a consequence of each model fitting the same law, on a shared outcome space, and only about 9% of shared eigenvector structure goes beyond what the shared laws imply. It is a single-author v1 preprint whose code is on request. Most of its length is natural-gradient steering, editing and fine-tuning, which is machine-learning practice. |
| LIT-937 v2 | Post-Hoc Concept Disentanglement: From Correlated to Isolated Concept Representations | Erogullari et al. (2025). Concept activation vectors fitted one concept at a time come out aligned for attributes that co-occur in the labels, such as “beard”, “mustache” and “necktie” in CelebA, in blocks matching the label correlations. A joint fit with a non-orthogonality penalty ‖CCᵀ − I‖²_F finds mutually orthogonal directions that still separate each concept, with AUROC largely preserved, at the last convolutional layer of VGG16 and ResNet18. So the alignment comes from the separate fits, not from the representation. Steering along the orthogonal directions inserts or removes one concept with less effect on correlated ones. | Active — read 2026-10-14 (NOTE-816) in full; worth reading as a clean case where concept directions for correlated attributes align only because each is fitted alone: a joint fit with an orthogonality penalty finds mutually orthogonal directions that still separate each concept with AUROC largely preserved, at the last convolutional layer of VGG16 and ResNet18 on CelebA and FunnyBirds. Read it knowing that every result is in a figure with no values or error bars, that AUROC is the only check that a direction keeps its meaning, and that the attributes are correlated, not hierarchical. Registered for QUESTION-025, where it bears on the correlated branch only; THEORY-292 is filed from it. |
| LIT-942 v2 | Do different prompting methods yield a common task representation in language models? | Davidson et al. (2025), NeurIPS 2025. The paper extends function vectors from few-shot demonstrations to short instruction prompts and extracts instruction function vectors that improve zero-shot accuracy. Demonstration and instruction function vectors share only 4 to 7 of their top 20 attention heads, add rather than substitute, and lose accuracy when their heads and activations are swapped. It concludes that the two prompting forms do not induce a common task representation but partly overlapping mechanisms, and that the instruction route is built by post-training. | Active — read 2026-10-14 (NOTE-803) in full; worth reading as the direct test of whether a task given by demonstrations and the same task given by an instruction share one internal representation. Under function vectors they do not: mostly different heads, vectors that add, and an asymmetric cost to swapping them. Read it knowing that it tests prompt forms, not paraphrase or stance, on simple lexical tasks in small models, and that the authors say they cannot prove the negative. Filed THEORY-264; bears on QUESTION-024 without answering it. |
| LIT-943 v2 | From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks | Dominé, Anguita, Proca, Braun, Kunin, Mediano & Saxe (2025), ICLR. For a two-layer linear network with whitened inputs, trained by full-batch gradient flow from a λ-balanced start, W₂ᵀW₂ − W₁W₁ᵀ = λI, the dynamics of the network function, representations and finite-width NTK are solved exactly. The solution is a matrix Riccati equation. The relative scale λ is conserved and sets the regime. As λ → 0 learning is rich: sigmoidal, stage-like, with task-specific representations. As |λ| → ∞ it is lazy: exponential, with a static NTK. Its sign, together with the architecture, decides which layer stays task-agnostic, and in funnel networks it gives a delayed rich phase. Wide LeCun-initialized networks are λ-balanced in probability. | Active — read 2026-10-13 (NOTE-715): main text in full, and the statement of Appendix A.3 (random initializations are λ-balanced) without its proof; the proofs in Appendices B–C not checked. Its control parameter is the relative scale of the layers, not the absolute scale the seed entry named. The account filed from it is THEORY-212, and the question it raises under SGD is QUESTION-081. |
| LIT-944 v2 | Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis | Kumar, Clune, Lehman & Stanley (2025), a position paper. Three image-generating networks found by Picbreeder’s human-guided evolution are rewritten as dense MLPs. Networks of the same architecture are trained by SGD to reproduce the same images, and they do. The SGD networks show no reuse of the symmetric parts, and sweeping their weights distorts the image, where the evolved networks’ sweeps vary it meaningfully. The paper calls this fractured entangled representation, argues that open-ended search approaches a unified factored one, and conjectures that FER in large models harms generalization, creativity and continual learning. The evidence is qualitative. | Active — read 2026-10-14 (NOTE-771) in full; worth reading for its side-by-side comparison of networks that compute the same image in the same architecture: SGD reproduces three Picbreeder images without the evolved networks’ reuse of symmetric parts or factoring of independent ones, and perturbing its weights breaks the regularities that perturbing theirs preserves. Read it knowing that it is a position paper, that the evidence is curated neuron maps and weight sweeps for three images with no measure of fracture, that the evolved networks were selected by users rather than trained on the image, and that its explanation by open-ended search and its extension to large models are conjecture. Held as THEORY-241. |
| LIT-946 v2 | The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models | Kirsanov et al. (2025). Using manifold capacity from statistical physics, the paper studies how an instruction, demonstrations and soft prompts change the geometry of representations in base decoder-only models. Methods with similar accuracy act differently: demonstrations reshape intermediate representations of the input, driven by the example sentences rather than their labels; an instruction acts mainly on the last token; soft prompts act late. Few-shot failures from example choice or relabelled categories leave the classes separable, which the authors attribute to the readout. Demonstrations for one task help other tasks’ separability early and hurt it late. | Active — read 2026-10-14 (NOTE-789) in full; worth reading for its split of accuracy into representational separability and readout: demonstrations reshape how the input itself is represented, driven by the example sentences rather than their labels, an instruction acts mainly on the last token, and few-shot failures from example choice or relabelled categories leave the classes separable. Read it knowing that it uses base models, one instruction wording and mostly synthetic data, and infers the readout failure rather than measuring it. Filed THEORY-251, and supports THEORY-264 with LIT-942. |
| LIT-950 v3 | All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling | Marconato, Lachapelle, Weichwald & Gresele (2024; AISTATS 2025). Two softmax next-token predictors induce the same distribution exactly when an invertible linear map relates them on the subspaces M and N that enter the logit dot product. Off those subspaces any distortion is allowed, at any representation dimension and without the diversity condition. Relational linearity, and with it linear probing and linear subspaces, then holds in all equivalent models or none when Γ ⊆ N and Γ_q ⊆ M. Parallelism of difference vectors survives only inside N. Theory only. | Active — read 2026-10-14 (NOTE-746) in full; worth reading as the exact characterisation of which softmax next-token predictors induce the same distribution: an invertible linear map on the subspaces M and N that enter the logits, with arbitrary distortion off them. Its all-or-none theorem for relational linearity, linear probing and linear subspaces is the record’s clearest proved case of structure identified when the full latent model is not. Read it knowing that the result needs exact equality of distributions over every sequence, a shared vocabulary and the conditions Γ ⊆ N and Γ_q ⊆ M, that parallelism survives only inside N, and that there are no experiments. For QUESTION-027 it is an instance of the unrestricted, already answered form, since the whole distribution is observed and no decision lies outside the probes. |
| LIT-952 v2 | Neural Networks Learn Statistics of Increasing Complexity | Belrose, Pope, Quirke, Mallen, Dury and Fern (ICML 2024). Vision networks trained on real images, probed during training with images whose class means or covariances are grafted from another class and with maximum-entropy samples keeping only those statistics, act on them early (peaks between about 2^4 and 2^12 steps) and lose that later. First-order sensitivity peaks before second-order. Pythia fits unigram-sampled text before bigram-sampled text. The n-gram/moment equivalence is an identity; the ordering is empirical, for orders one and two. | Active — read 2026-10-14 (NOTE-777) in full; worth reading for its probes of what a network is using at a given checkpoint, which are maximum-entropy samples and optimal-transport grafts of class statistics, and for the result they give. Vision networks act on class means and covariances early in training and lose that later, with first-order sensitivity peaking before second-order. Pythia fits unigram text before bigram text. Read it knowing that the ordering is empirical and covers orders one and two, that the Taylor argument for it is a heuristic, and that its n-gram/moment theorem is an identity of one-hot indicators. Held as THEORY-279. |
| LIT-961 v2 | A Reply to Makelov et al. (2023)‘s “Interpretability Illusion” Arguments | Wu, Geiger, Huang, Arora, Icard, Potts and Goodman (2024). The authors of distributed alignment search reply to Makelov et al. (2023). They argue that the “illusion” is generic, because a direction a model uses need not be orthogonal to the null space of the next layer; that FLDD can be large with no change in behaviour; and that the factual-recall cases come from DAS trained per fact pair. Their toy counterexample holds only for an unnormalized patch. Both papers grant that alignments related by a change of basis can tie under interchange interventions. The reply’s own GPT-2 experiments agree that MLP8 carries almost no name-position information (4% IIA, against 70% for the layer-8 block output). | Active — read 2026-10-14 (NOTE-809) in full; worth reading for the point it shares with the paper it answers: interchange interventions on one set of variables can be satisfied equally by alignments related by a change of basis, so the mediating variable is fixed only relative to a choice of units. Read it knowing that its central toy computation, which claims Makelov et al.’s test rejects their own faithful direction, holds only for an unnormalized patch; patching the subspace the rowspace component spans recovers the full effect. Its own GPT-2 experiments agree that MLP8 carries almost no name-position information. The dispute is over the label, the metric (FLDD against IIA) and how DAS is trained, not over constitution against correlation. Filed, with LIT-968, as THEORY-243. |
| LIT-963 v2 | Fundamental Limits of Perfect Concept Erasure | Chowdhury et al. (2025), AISTATS. Under finite supports and disjoint concept groups, an erasure Z = f(X) that reveals nothing about the concept A keeps at most I(Z;X) ≤ H(X|A). A deterministic map reaches that only when the groups’ distributions are permutations of one another, by mapping each group bijectively onto a common distribution. For unequal groups a stochastic minimum-entropy-coupling map still erases perfectly with less utility. The paper says the cap is then out of reach, which holds only for deterministic maps. The method beats INLP, LEACE, FaRM, KRaM and DCA on synthetic data, GloVe and GPT-4 embeddings. | Active — read 2026-10-14 (NOTE-820) in full; worth reading for the clean information-theoretic cap on perfect concept erasure, I(Z;X) ≤ H(X|A) once Z is independent of A, and for the proof that a deterministic erasure reaches it only when the concept groups are permutations of one another. Read it knowing that it assumes finite supports and disjoint groups, so that the concept is a function of the representation, and that its claim that unequal groups cannot reach the cap is false for stochastic maps. Filed for QUESTION-010; the bound is held as THEORY-281. |
| LIT-968 v3 | Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching | Makelov, Lange and Nanda (2023). A subspace activation patch can make a model behave as if a feature had changed while the patched direction is the sum of a part that varies with the feature but cannot reach the output (in an MLP, the kernel of the down-projection) and a part that can reach the output but is dormant on the data. Exact in a three-unit network; shown for MLP8 of GPT-2 Small on indirect object identification and for GPT2-XL MLPs on factual recall, where the effect vanishes once the disconnected part is removed. A residual-stream direction for the same feature passes the test. One-dimensional fact patches and rank-1 edits are nearly interchangeable, so a working rank-1 edit does not show the fact is stored in the edited layer. | Active — read 2026-10-14 (NOTE-782) in full; worth reading as the clearest case of an intervention that produces the predicted counterfactual output without the intervened variable carrying the feature: a subspace patch that pairs a direction varying with the feature but unable to reach the output with one that can reach it but is dormant on the data. Read it knowing that its ground truth for GPT-2’s MLP8 comes from prior component-level work, that it concedes a cancelling-subspaces reading it does not test, that its own residual-stream case passes the test, and that the reply (LIT-961) contests the label but not the MLP8 measurements. It speaks of faithfulness to a computation, not of constitution. Filed as THEORY-282 and, with the reply, THEORY-243. |
| LIT-972 v2 | On Linear Representations and Pretraining Data Frequency in Language Models | Merullo et al. (2025). For 24 factual relations, how well an affine map from subject to object representation reproduces the model (linear relational embedding causality) correlates with the relation’s mean subject–object co-occurrence count in pretraining (r = 0.82), more than with subject (0.66) or object (0.59) frequency alone. Mean causality passes 0.9 above about 1.1k co-occurrences in GPT-J, 2.0k in OLMo-7B and 4.4k in OLMo-1B, at whatever training step the count is reached. A regression on LRE quality predicts a term’s pretraining frequency within 10× more often than log probabilities do, also for a second model. Correlational, by the authors’ own statement. | Active — read 2026-10-14 (NOTE-786) in full; worth reading as the clearest measurement of when linear structure forms in trained language models: across 24 factual relations in OLMo-1B, OLMo-7B and GPT-J, the quality of a linear relational embedding tracks mean subject–object co-occurrence in pretraining (r = 0.82), with mean causality above 0.9 past about 4.4k, 2.0k and 1.1k co-occurrences, at any training step. Read it knowing that it is correlational by the authors’ own statement, that an LRE is an affine map between hidden states for one relation and not an attribute direction, and that the structure of co-occurrence across attributes is not measured. Registered for QUESTION-025, where it bears only indirectly; THEORY-269 is filed from it. |
| LIT-974 v2 | Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path | Han, Papyan & Donoho (2021; ICLR 2022). Networks trained with MSE loss show neural collapse, on five datasets with VGG, ResNet and DenseNet. The MSE loss splits exactly into a term where the classifier is the least-squares classifier for the current features and a non-negative deviation term. The deviation is measured to be negligible from early in training: the “central path”. On that path, for unconstrained features with no weight decay and re-whitened to identity within-class covariance after every step, gradient flow gives closed-form dynamics. Every singular value of Σ_W^(−1/2)M̃ grows as (4t/N)^(1/4), and they tend to equality, which yields NC1–NC4. | Active — read 2026-10-14 (NOTE-783) in full; worth reading as the record’s one derivation of neural-collapse dynamics in closed form. Under MSE loss, a trained classifier stays near the least-squares classifier for its own features (measured), and on that central path a whitened gradient flow of unconstrained features drives every class-separation singular value to infinity as (4t/N)^(1/4) and to equality, which is NC1 and NC2. It also measures collapse under MSE on five datasets and three architectures. Read it knowing that the flow is not gradient descent (features are re-whitened after every step), that the theory has no weight decay while every experiment has it, that it is MSE and not cross-entropy, and that the experiments are LIT-618’s protocol with one run per cell. |
| LIT-975 v3 | Contextures: The Mechanism of Representation Learning | Zhai (2025), CMU PhD dissertation. A representation is learned from the joint law of an input X and a context variable A. The top-d left singular functions of the operator g ↦ E[g(A)|X], which runs between two different spaces and need not be symmetric, are proved to attain the minimum worst-case linear-probe approximation error over the tasks compatible with the context. The thesis says they are the unique such span, which is false (THEORY-239). Several squared-error pretraining objectives reach that span at their optimum. That a context is most useful at moderate association, and that scaling model size gives diminishing returns, are argued and measured on small datasets, not proved. | Active — read 2026-10-14 (NOTE-770) in full; worth reading for QUESTION-052 as the singular-function result for a pair operator that need not be symmetric: for any joint law of an input and a context, the top left singular functions of the conditional-expectation operator attain the minimum worst-case linear-probe approximation error over compatible tasks (Theorem 3.4), though, contrary to the thesis, they are not the unique span that does (THEORY-239). Read it knowing that every result is at the population optimum of squared-error objectives, that its only graph example is undirected, and that its claims about scaling and about moderate association are argued from a few experiments, not proved. |
| LIT-976 v2 | Emergence of Quantised Representations Isolated to Anisotropic Functions | Bird (2025; v4 2026). In small CIFAR and MNIST autoencoders whose affine maps, initialisation and optimiser are all rotation-equivariant, only the activation is changed. Elementwise tanh (signed-permutation symmetry) and Leaky-ReLU (permutation symmetry) concentrate latent representations into near-discrete beams along the basis or the ±1 diagonal, even before any activation. Radial versions of the same functions, equivariant to the orthogonal group, leave them continuous. A new projection method (PPP) shows this in density plots; there is no statistic. The paper argues that the primitives’ algebraic symmetry predicts task-agnostic representational structure. | Active — read 2026-10-14 (NOTE-759) in full; worth reading as a clean ablation in which the activation function is the only part of an autoencoder that singles out a basis: elementwise tanh (B_n) and Leaky-ReLU (S_n) drive latent representations into narrow beams arranged by their group, even in layers no activation precedes, while radial O(n)-equivariant versions leave them spread. Read it knowing that the evidence is density plots from small CIFAR and MNIST autoencoders, five seeds a condition, with no statistic, and that its claims about grandmother neurons, superposition, neural collapse and better reconstruction are conjecture or preliminary. Held as THEORY-288. |
| LIT-977 v2 | Distributional hypothesis as isomorphism between word-word co-occurrence and analogical parallelograms | Torii et al. (2024). The paper argues that analogy parallelograms are in the co-occurrence data, not made by word2vec. On Wikipedia, an SVD of log raw co-occurrence counts solves 61% of the Google analogies over a 1,487-word vocabulary (50% over 10,072), and word2vec without negative sampling about 60%, against 75% for word2vec trained on the full corpus. Raw rows solve under 6%. In a 24-sentence toy corpus whose eight subjects carry every combination of three binary aspects, Theorem 1 shows that the subjects’ raw co-occurrence rows form a parallelepiped if and only if, for each object, its four sentences are equally probable. | Active — read 2026-10-14 (NOTE-781) in full; worth reading for Theorem 1, an exact condition in a toy corpus: eight subjects carrying every combination of three binary aspects have raw co-occurrence rows forming a parallelepiped if and only if, for each object, its four sentences are equally probable. That is a data-side account in which aspects combine as a mixture, so raw counts and not PMI are linear. Read it knowing that the theorem covers one 24-sentence design and exact shapes, that its aspects are independent (a Boolean lattice), that the real-text analogy results need log counts and an SVD, and that the restricted word2vec baseline is trained on a dummy corpus. Registered for QUESTION-025; THEORY-271 is filed from it. |
| LIT-978 v3 | Neural Collapse: A Review on Modelling Principles and Generalization | Kothapalli (2022; TMLR 2023). A survey of neural collapse in one notation. It reviews the unconstrained-features model and local elasticity as ways of modelling how training reaches the collapsed state. Every derivation it covers works with features cut loose from the data. On generalization it finds the literature in conflict: collapse on the training set does not fix test error, and collapse on test data is weaker and depends on the dataset. No agreed threshold says when a network is collapsed. It reports Kornblith et al.’s finding that more collapsed ImageNet networks transfer worse. Skimmed. | Active — read 2026-10-14 (NOTE-772) in part; worth reading as the map of the first two years of models of neural collapse, sorted into unconstrained-features models and local elasticity, with their assumptions side by side, and as the place where the conflicting claims about collapse and generalization are set next to each other: collapse on the training set reached by different optimizers with different test error, weaker and dataset-dependent collapse on test data, and worse transfer from more collapsed ImageNet networks. Read it knowing that it is a survey whose results are all second-hand, that every derivation it covers removes the data, and that it predates the imbalance geometry of LIT-994 and the kernel analysis of LIT-985. Skimmed: the equations were not checked. |
| LIT-981 v3 | Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps | Coifman et al. (2005). A symmetric, positivity-preserving, positive semi-definite affinity kernel is normalized into a Markov operator that is conjugate to a symmetric one, ã. Its eigenfunctions, scaled by powers of their eigenvalues, embed the data so that Euclidean distance equals the diffusion distance D_m induced by ã^(m), exactly; truncation is claimed to relative accuracy. A one-parameter normalization yields the graph Laplacian, the backward Fokker–Planck operator, or the Laplace–Beltrami operator, the last independent of sampling density. The convergence theorem and the α-family are stated without proof. Non-symmetric kernels are not treated. | Active — read 2026-10-30 (NOTE-1441) in full, from the published PDF on co-author Zucker’s Yale page (www.cs.yale.edu/homes/vision/zucker/papers/CLLMNWZ05_PNAS_1.pdf), with every equation seen; it replaces the 2026-10-14 skim of the PubMed Central text (NOTE-784), whose equations were not visible. Worth reading for QUESTION-052 as the classical symmetric case: from a symmetric, positivity-preserving, positive semi-definite affinity, the normalised walk is conjugate to a symmetric operator, and its eigenfunctions scaled by λ^m embed the data so that Euclidean distance equals a diffusion distance; a density-corrected normalisation (α = 1) has the Laplace–Beltrami operator as its small-ε limit, independent of the sampling density. Read it knowing that its diffusion distance is the one induced by the symmetric kernel ã^(m), not the inverse-stationary- weighted distance of Coifman and Lafon’s companion (LIT-1246); that its truncation is claimed to relative accuracy, the defect THEORY-745 records for the companion; that Theorem 2.1 and the α-family are stated without proof, with the proofs in LIT-1246; that Eq. 5 and Eq. 14 give the α = 0 potential opposite signs; and that its “directed diffusion” is anisotropic, not a non-reversible chain. |
| LIT-985 v2 | Can Kernel Methods Explain How the Data Affects Neural Collapse? | Kothapalli and Tirer (2024; TMLR 2025). The paper writes NC1, as tr Σ_W / tr Σ_B, for the features of any kernel. It proves, for one-dimensional two-class Gaussian data on opposite sides of the origin with ReLU and zero bias, that the NTK of a wide two-layer net has the same expected NC1 as the NNGP kernel at initialization. In higher dimensions it measures the NTK no more collapsed, and with ReLU less. A data-aware adaptive kernel lowers NC1 and approximates finite networks on separable data. It departs from them under class overlap in high dimension and under imbalance. Erf activation gives lower NC1 than ReLU. | Active — read 2026-10-14 (NOTE-774) in full; worth reading as the record’s one account of neural collapse in which the data enter, and for its negative result. In the case it can solve, lazy-regime training of a wide two-layer net leaves the within-class variability of the features exactly where initialization put it, because there the NTK is a within-class rescaling of the NNGP kernel. It also gives NC1 as a closed function of any kernel matrix. Read it knowing that the theorem covers one-dimensional data with two classes on opposite sides of the origin and zero bias, that the higher-dimensional claim is simulation (and for ReLU the NTK is less collapsed, not equal), that its NC1 is tr Σ_W / tr Σ_B on shallow MSE-trained binary nets, and that the data-aware kernel it proposes fails under class overlap and imbalance. Its printed Eq. 16 appears to keep a cross-class term the kernel sets to zero. |
| LIT-986 v2 | Transformers learn factored representations | Shai et al. (2026). If a sequence generator’s latent operators are tensor products, its predictive state stays a product of per-factor beliefs. A linear map embeds that state losslessly in orthogonal subspaces of total dimension Σ(d_n − 1), against (Π d_n) − 1 for the joint state. Small transformers trained on five such synthetic factors settle near 10 dimensions against 242, in approximately orthogonal subspaces. When token noise makes factoring lossy they collapse to the factored form first and expand later, the more so the more noise. | Active — read 2026-10-14 (NOTE-758) in full; worth reading for QUESTION-060 as a controlled case with known relevant variables. When a generator factors, small transformers learn per-factor beliefs in roughly orthogonal subspaces of linear, not exponential, dimension. Under noise that makes factoring lossy they reach the factored form first and leave it later. Read it knowing that the processes are five small synthetic ones, that the objective is never varied, and that the paper leaves open whether the preference is architectural or a feature of training. |
| LIT-987 v2 | Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors | Huang, Tao, Icard, Yang and Potts (2025; ICML 2025). Given a high-level causal model of a task and the subspace where DAS locates its key variable on correct in-distribution examples, transplanting a test input’s value of that variable into verified examples (counterfactual simulation) predicts whether the output is correct. In Llama-3-8B-Instruct, over five tasks and ten shifts that keep the task but lower accuracy, it beats output confidence and causal-agnostic probes on 8 of 10. A probe of the located variable beats a probe of its orthogonal complement on 7 of 10. Interchange accuracy and predictive power rise together only until accuracy nears 1. | Active — read 2026-10-14 (NOTE-765) in full; worth reading as a measured case in which variables located by interchange intervention as mediating a model’s in-distribution behaviour predict whether its outputs stay correct under task-preserving shifts better than output confidence or probes of the rest of the representation. Read it knowing that it is one model, two selected shifts per task and three folds without significance tests; that counterfactual simulation wins 8 of 10 shifts and the causal-variable probe beats the background probe on only 7; that its comparison features are not “merely correlated” ones; and that it claims nothing about constitution. It does not cite Makelov et al. or the reply. Filed as THEORY-287. |
| LIT-994 v2 | Imbalance Trouble: Revisiting Neural-Collapse Geometry | Thrampoulidis, Kini, Vakilian & Behnia (2022, NeurIPS). In the unconstrained-features model under STEP class imbalance, every solution of the max-margin problem, the limit of cross-entropy with vanishing regularization, has logits equal to the simplex-encoded label matrix Ẑ. Its classifier and embedding Gram matrices come from Ẑ’s SVD (SELI). That is the simplex ETF only for balanced or binary data. With any finite ridge the geometry differs and depends on its strength; it tends to SELI as the ridge vanishes. In the unconstrained-features model and in ResNet-18 and VGG-13 runs, training ends closer to SELI than to the ETF, not at it, and the approach slows as imbalance grows. | Active — read 2026-10-14 (NOTE-764) in full; worth reading as the statement of what neural collapse becomes under class imbalance. In the unconstrained-features max-margin problem, the logits equal the simplex-encoded labels at any imbalance. The classifier and embedding geometries come from that matrix’s SVD, and they are the simplex ETF only for balanced or binary data. Majority classifiers come out longer and minority classifiers closer together, and the minority-collapse threshold is given in closed form. Read it knowing that gradient descent is only conjectured to reach this geometry, that under any finite weight decay the geometry differs and depends on its strength, that convergence is slow and slower with imbalance, and that the deep-net runs are one seed per cell with weight decay on. |
| LIT-995 v2 | Deep sequence models tend to memorize geometrically; it is unclear why | Noroozizadeh et al. (2025; ICML 2026). Transformers and Mamba trained on a graph’s edge bigrams embed its nodes so that multi-hop relations are readable, and they solve an in-weights path-finding task built to defeat next-token learning. The authors argue that straightforward supervisory, capacity, optimization and succinctness pressures do not explain why this geometry wins over a lookup the same models can store. In a wide two-layer weight-tied model, unregularized cross-entropy drives the embeddings into the top Laplacian eigenvectors without a rank bottleneck. This is shown empirically, with an informal partial proof. Why deeper models prefer the geometry is left open. | Active — read 2026-10-14 (NOTE-800) in full; worth reading for QUESTION-060 as a case where neither the supervision nor what the model can represent fixes the learned variables. Sequence models trained only on a graph’s edges embed it as a global geometry, although a lookup is expressible, found faster and as succinct. In a two-layer model the geometry is the top Laplacian eigenspace, reached by cross-entropy training alone. Read it knowing that every argument is empirical or informal by the authors’ own account, that the spectral account covers only the two-layer model, and that initialization scale can tip deeper models into a lookup. |
| LIT-1044 v2 | Why Do Better Loss Functions Lead to Less Transferable Features? | Kornblith, Chen, Lee & Norouzi (2020; NeurIPS 2021). Nine training objectives for ResNet-50 on ImageNet: seven beat softmax cross-entropy significantly, by up to 0.9 points top-1, and all eight alternatives yield fixed penultimate features that transfer worse to eight new-class datasets, by linear and k-NN readout. Each raises penultimate-layer class separation R² (one minus within-class over total cosine distance), and mean linear transfer falls with R² across losses (Spearman ρ = −0.93) and along a cosine-softmax temperature sweep. The losses differ only in the last third of the network, and full fine-tuning removes the gap. | Active — read 2026-10-14 (NOTE-762) in full; worth reading as the record’s one direct measurement of what class separation in the last layers costs off the trained task. Across nine objectives for ResNet-50 on ImageNet, those that beat softmax cross-entropy all raise penultimate-layer class separation (R² 0.349 → up to 0.845), and mean linear transfer to eight new-class datasets falls with it (Spearman ρ = −0.93; 75.9% for softmax, 48.3% for squared error). Cosine-softmax temperature moves both on one knob. The losses differ only in the last third of the network, and full fine-tuning erases the transfer gap. Read it knowing that the networks are early-stopped at 91.6–93.6% training accuracy, so this is relative within-class variance and not terminal-phase neural collapse; that the cost is for new classes, while on shifted ImageNet test sets with the same classes the more separated losses do slightly better (Table C.2); and that it is one dataset, two architectures and a correlation across losses, with one within-loss sweep. Filed as THEORY-246. |
| LIT-1246 v3 | Diffusion maps | Coifman & Lafon (2006). From a symmetric non-negative kernel, a reversible random walk; its t-step diffusion distance equals the Euclidean distance between eigenvectors weighted by λ_l^t, and truncating where |λ_l|^t ≤ δ|λ₁|^t gives the diffusion map, whose dimension falls as t grows. An α-normalization by a kernel density estimate gives, as the kernel width ε → 0, a generator with invariant density q^{2(1−α)}: the density-weighted graph Laplacian at α = 0, the Fokker–Planck (Langevin) operator at α = 1/2, and the Laplace–Beltrami operator, free of the density, at α = 1. Symmetric kernels only. | Active — read 2026-10-20 (NOTE-985) in full; worth reading for QUESTION-217 as the model case of a coarse-to-fine order induced by an operator rather than an imposed partition, and for the proofs the PNAS announcement (LIT-981) deferred. The diffusion distance D_t is exactly the Euclidean distance between eigenvector coordinates weighted by λ_l^t. The number of steps t is a scale, and the kernel width ε is a second one. An α-normalization makes the small-ε limit the graph Laplacian with a density term (α = 0), the Langevin generator whose invariant density is the sampling density (α = 1/2), or the Laplace–Beltrami operator (α = 1). Read it knowing that Proposition 1’s “relative accuracy δ” for the truncated map is wrong (the bound is absolute, THEORY-745); that Appendix A’s ψ_l = φ_l/π should be φ_l/√π; that the α = 1/2 display, the circle formula, the discrete kernel of §5 and Criterion 4’s exponent are misprinted or wrong; that the boundary rate and the heat-kernel proof are incomplete (THEORY-744); and that no spectral gap is used anywhere. |
| LIT-1249 v3 | Slow Feature Analysis: Unsupervised Learning of Invariances | Wiskott & Sejnowski (2002). Learns instantaneous functions of the input whose outputs vary as slowly as possible, minimizing ⟨ẏ²⟩ under zero-mean, unit-variance and decorrelation constraints, solved in closed form within a fixed (usually quadratic) expansion by sphering and taking the smallest eigenvectors of the derivative covariance. Stacked modules in a 1-D model of the visual hierarchy learn translation, size, rotation, contrast and, less well, illumination invariance from the training stimulus, with location (“where”) among the slowest outputs alongside identity (“what”). | Active — read 2026-10-20 (NOTE-971) in full, from Wiskott’s own PostScript preprint; worth reading for the slowness objective stated exactly (minimize ⟨ẏ²⟩ under zero mean, unit variance and decorrelation), for its closed-form solution within a fixed nonlinear expansion by sphering and an eigendecomposition of the derivative covariance (THEORY-746), and for the demonstration that stacked modules learn translation, size, rotation, contrast and, less well, illumination invariance for 1-D patterns from the training stimulus alone. Read it knowing that optimality holds per module, on the training signal and within the chosen function space, and not for the greedy hierarchy; that “slower is more abstract” is its premise, not a finding, so a stack built this way is slower at higher stages by construction; that the slowest outputs include location, so slowness is not invariance; that no generative model and no learning dynamics appear; and that every stimulus is synthetic and one-dimensional. |
| LIT-1250 v2 | Diffusion maps, spectral clustering and reaction coordinates of dynamical systems | Nadler, Lafon, Coifman & Kevrekidis (2005; ACHA 2006). For data sampled from p = e^{−U}, it expands the α-normalized diffusion-map operators for small kernel width. The normalized graph Laplacian (α = 0) tends to a backward Fokker–Planck generator with potential 2U, equivalently a Schrödinger operator with potential |∇U|² − ΔU. The normalization α = 1/2 gives the generator of the Langevin dynamics ẋ = −∇U + √2ẇ itself. α = 1 gives the Laplace–Beltrami operator, free of the density. So for data from a stochastic system in equilibrium, the leading eigenfunctions approximate the system’s slowest modes, and the paper proposes them as reaction coordinates. Examples: Ornstein–Uhlenbeck processes, where the first eigenfunctions parametrize the slow variable; a double well, where the first eigenfunction is ±1 on the two wells; three wells; and the iris data. The expansions are formal, and the printed generators are twice what the stated kernel and time step give. | Active — read 2026-10-20 (NOTE-986) in full, from arXiv:math/0503445v1; worth reading for the bridge from a diffusion map to the dynamics behind the data. For samples from p = e^{−U}, the ordinary normalized graph Laplacian (α = 0) tends to a backward Fokker–Planck generator with potential 2U, and the anisotropic normalization α = 1/2 to the generator of the Langevin dynamics with potential U itself, whose slow eigenfunctions the paper proposes as reaction coordinates. It also has worked examples: Ornstein–Uhlenbeck (the finite-ε spectrum λ_k = (τ/(τ+ε))^k), a double well whose first eigenfunction is ±1 on the two wells, and a three-well potential with two pathways. Read it knowing that the generators as printed are twice what the paper’s own kernel and time step give (its Ornstein–Uhlenbeck lemma and its equation (32) disagree by that factor, which this reading checked); that “converge” means a formal small-ε expansion of the operator, with the large-sample limit cited and no spectral convergence shown; that its sufficient condition for the first eigenfunctions to depend only on the slow variable is printed the wrong way round; that its double-well example is separable, so the slow coordinate is closed by construction; and that from equilibrium samples the eigenfunctions are recovered well while the slow eigenvalue, the inter-well rate, is not. In what sense slow eigenfunctions are closed coarse variables is the record’s addition (THEORY-762). The published ACHA version was not reached. |
| LIT-1287 v2 | A Maximum-Likelihood Interpretation for Slow Feature Analysis | Turner & Sahani (Neural Computation 2007). Gives a probabilistic model for which, in a limiting case, inference and learning “yield exactly the results of SFA”, in the way that independent component analysis has a maximum-likelihood reading, and uses the model to motivate several extensions of the algorithm. Not read; this summary is from the abstract. The model’s form and the limit taken are not checked here. | Unavailable — registered 2026-10-20, not read. A search on 2026-10-20 for a lawful full text reached none. The DOI resolves to MIT Press (direct.mit.edu, article 7171), which returned a Cloudflare bot check (403); the article is not open access in any case. Unpaywall and OpenAlex report it closed, with no repository full text. Europe PMC has it (PMID 17348772) but not in PMC. Sahani’s Gatsby publication pages (~maneesh/papers/ and researchli2.html) list it with a DOI link only, no file. The Cambridge Machine Learning Group’s publications page (mlg.eng.cam.ac.uk/publications/) gives the abstract and links Turner’s old Gatsby page (gatsby.ucl.ac.uk/~turner/Publications/turner-and-sahani-2007a.html), which returns 404, as do the PDF at the same path and the guessed Cambridge paths (cbl.eng.cam.ac.uk and learning.eng.cam.ac.uk under /Public/Turner/, mlg.eng.cam.ac.uk/pub/pdf/TurSah07.pdf and variants). OpenAlex lists two repository records with no full text flagged: UCL Discovery (eprint 134626), which returned a Cloudflare bot check (403), and publications.eng.cam.ac.uk/526467, whose host no longer resolves. CiteSeerX (10.1.1.109.4162) reset the connection on four tries and returned 503 once, so whether it holds a cached author copy is unknown. The Wayback Machine reset the connection or returned 429, and the fetch tool cannot reach it. Semantic Scholar returned 429. Title searches found no author-posted copy. So this is a failure on access as much as on availability: a cached copy may exist at CiteSeerX or in the Wayback Machine’s capture of Turner’s Gatsby page. It stays Unavailable until a copy is supplied. No NOTE is filed. |
| LIT-1296 v2 | Slow Feature Analysis as Variational Inference Objective | Schüler & Wiskott (arXiv 2025). With a Gaussian random-walk prior on the latents, an instantaneous unit-variance Gaussian encoder N(g(x_t), I) and an arbitrary forward model, the evidence lower bound is a sum of per-point reconstruction terms minus ½Σ‖g(x_{t−1}) − g(x_t)‖², the unordered slowness loss. The reconstruction term then does the work of SFA’s unit-variance and decorrelation constraints, by argument, not proof. The authors say this “does not lead to full equivalence” with SFA. For linear Gaussian encoder and decoder they give the objective and the necessary condition VᵀV = WᵀĊW, and no optimum. | Active — read 2026-10-20 (NOTE-1015) in full, from arXiv v1; worth reading for an exact identity, not a limit: with a Markov prior p(z_t | z_{t−1}) = N(z_{t−1}, I) (a Gaussian random walk, no decay), an instantaneous encoder q(z_t | x_t) = N(g(x_t), I) and any forward model p_θ(x_t | z_t), the KL half of the ELBO is the unordered slowness loss ½Σ‖g(x_{t−1}) − g(x_t)‖² (plus a constant the paper drops), so the ELBO is a per-point reconstruction term minus a slowness term (Eq. 40), and the reconstruction term takes the place of SFA’s unit-variance and decorrelation constraints; and for the linear Gaussian case (Eq. 41) and its necessary condition VᵀV = WᵀĊW (Eq. 42). Read it knowing that the paper does not claim, and does not have, equivalence to SFA: it derives no optimum, so what its objective learns is the record’s derivation in THEORY-775 (on whitened input, the span of the slowest SFA directions whose lag-one autocorrelation exceeds ½, rotation unidentified; on raw input, not SFA’s subspace and not scale-invariant); that the constant d/2 per step is cancelled by an error in Eq. 35; that Eq. 41 has the sign of the reconstruction error reversed (the appendix’s Eq. 54 is right); that Eq. 42 is necessary, not “equivalent” to stationarity as §7 says; that the prior builds in independent, non-decaying latents, so closure of the latent coordinates is assumed, not found; and that its account of Turner and Sahani (2007) is two sentences of related work, which the record has not checked against that paper (LIT-1287, Unavailable). |
| LIT-1347 v2 | Verbalizable Representations Form a Global Workspace in Language Models | Gurnee, Sofroniew et al. (2026), Anthropic, Transformer Circuits Thread; arXiv:2607.15495. A Jacobian lens, the Jacobian from a layer to the output averaged over a corpus, gives each vocabulary token a direction per layer; sparse combinations of them form the “J-space”. In Claude Sonnet 4.5, Haiku 4.5, Opus 4.5 and Opus 4.6, swaps, injections and ablations show the J-space part of a representation, under a tenth of its variance, carrying verbal report, instructed concepts, two-hop intermediates and reuse of an argument by many functions, while continuation, anomaly detection and line wrapping use the same information without it. The J-space works in a middle band of layers, holds about 25 active directions, and is amplified and relayed by the weights. Ablating it flattens experiential language about the model and about others. Further sections apply it to alignment audits, show post-training installing the Assistant’s point of view, and introduce counterfactual reflection training. The authors take no position on phenomenal consciousness. | Active — read 2026-10-22 (NOTE-1069), main text in full and appendices in part, from the arXiv v1 PDF; worth reading as the first interventional evidence that a language model has a functional analogue of conscious access in the access sense: a sparse set of verbalizable directions, found by a corpus-averaged Jacobian lens, carries report, held concepts, unspoken reasoning steps and reuse of one argument by many functions, while routine processing uses the same information without it; and ablating it flattens experiential language about the self and about others alike. Read it knowing its limits: one company’s models studied by that company, unreplicated; a lens whose own statistics draw the layer band; an automatic/flexible split partly defined by the result; experiential language graded by another model; and an explicit refusal to say anything about phenomenal consciousness. Filed as THEORY-865 and THEORY-869. |
| LIT-1530 v2 | Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks | Musat, Pimentel, Zucchet & Hofmann (2026), arXiv:2607.11875. For a class of block-list inductive tasks that includes associative recall, k-hop induction and in-context n-grams, it proves that population gradient descent keeps an idealized attention-only transformer’s key–query and output weights on a low-dimensional invariant manifold fixed by the data’s symmetries, whose coordinates are interpretable head actions. On it, it proves that a saturated in-weights circuit suppresses the in-context circuit’s gradient by the fraction of data it does not solve and that burstiness multiplies it, shows empirically that initialization decides which of several induction circuits wins, and detects circuits in trained models by projecting onto the coordinates. | Active — read 2026-10-26 (NOTE-1177) in full, from the arXiv copy (v2); worth reading for its proof that symmetries of the data confine population gradient descent to a subspace of interpretable head actions (THEORY-981), for its theorem that a saturated memorized shortcut starves the in-context circuit of gradient on the data it solves (THEORY-1009), and for its scan in which initialization picks one of several induction circuits, read by the authors as cooperation through shared units and competition for layers (THEORY-1001). Read it knowing that the architecture is idealized (fixed orthogonal write maps, merged key-query, orthonormal embeddings), that invariance is proved and attraction is not, that the results are for gradient descent while its circuit-detection experiment uses AdamW, and that the competition scan was run in training projected onto seven coordinates. |
| LIT-1531 v3 | AtP*: An efficient and scalable method for localizing LLM behaviour to components | Kramár, Lieberum, Shah & Nanda (2024), arXiv:2403.00745. Attribution patching (AtP; Nanda 2023), a first-order gradient estimate of every node’s activation-patching effect, is tested against exhaustive patching on Pythia models from 410M to 12B. Verifying nodes in AtP’s order finds the largest-effect nodes far more cheaply than unordered or sampling-based search. Its false negatives come from saturated attention and from cancelling direct and indirect effects; AtP* adds an exact softmax for queries and keys and layer-wise gradient dropout, and a subsampling test bounds the size of nodes still missed. On the two prompt distributions, simpler variants are cheaper than AtP*. | Active — read 2026-10-26 (NOTE-1190) in full, from arXiv v1, with the comparison figures read from page renderings; worth reading as the systematic test of attribution patching against exhaustive activation patching on models up to 12B, for the two named causes of its misses (saturated attention, cancelling direct and indirect effects) and their fixes, and for the statistical bound on what is still missed (THEORY-996). Read it knowing that the ground truth is single-node noising on one model family and five settings, that only ranks are scored, that the numbers are only in charts, and that AtP* is not the cheapest variant on its two distributions. |
| LIT-1532 v2 | Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models | Marks et al. (2024), arXiv:2403.19647, ICLR 2025. Circuits are found over sparse-autoencoder features and autoencoder error terms of Pythia-70M and Gemma-2-2B, scoring nodes and edges by linear approximations to their indirect effects. On subject–verb agreement, small feature circuits carry much of the behaviour where neuron circuits need many more nodes, though only with the error terms kept. A human can remove features judged task-irrelevant from a classifier’s circuit (SHIFT), and the same method is run without supervision over thousands of behaviours clustered from the Pile. | Active — read 2026-10-26 (NOTE-1187) in full, from the arXiv copy (v3, the ICLR 2025 version); worth reading as the method that turned sparse-autoencoder features into circuits, by linear approximations of indirect effects over features and autoencoder errors together, and for its own evidence that the features alone are not the circuit: with the error nodes removed, its feature circuits recover little of the behaviour (THEORY-1007). Read it knowing that faithfulness is scored on the latter two thirds of each circuit only, that the interpretability of the circuits is the authors’ annotation, that the half-performance figure for Gemma neurons lies beyond its plotted range, and that its debiasing method is ML practice, which this record does not hold. |
| LIT-1533 v2 | Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet | Templeton et al. (2024), Transformer Circuits Thread. Sparse autoencoders of about 1M, 4M and 34M features, sized by scaling laws for the autoencoder’s own loss, are trained on the middle-layer residual stream of Claude 3 Sonnet and explain at least 65% of its variance. They yield features that respond to the same concept across languages and in images, that steer the model when clamped, and that include features for unsafe code, bias, sycophancy and deception. Whether a concept has its own feature depends on its training frequency relative to the dictionary’s size. | Active — read 2026-10-26 (NOTE-1178) in full, from the article’s own page on the Transformer Circuits Thread; worth reading as the demonstration that sparse-autoencoder features of a production model are abstract (multilingual, multimodal) and can steer it, for the finding that a concept gets its own feature only once its training frequency passes about one over the number of live features (THEORY-992), and for the observation that the most active features are mostly not the ones that matter to an answer (THEORY-987). Read it knowing that its interpretability and steering evidence is shown on features the authors selected, that the model, its size and some plot units are withheld, that the reconstructions explain only at least 65% of the variance, and that its wider thesis that these features are the model’s working concepts is open (THEORY-965). |
| LIT-1534 v2 | A Mathematical Framework for Transformer Circuits | Elhage et al. (2021), Transformer Circuits Thread. Rewrites a transformer without changing it: heads act independently and add into a linear residual stream, each head is a low-rank QK matrix that sets where to attend and a low-rank OV matrix that sets what is moved, and with attention patterns frozen an attention-only model is a sum of end-to-end path terms. Zero-layer models are bigram tables, one-layer attention-only models bigram and skip-trigram tables read off the weights, and two-layer models use K-composition with a previous-token head to form induction heads, [a][b] … [a] → [b]. A correction of the composition diagram adds more composition than first reported. Models with MLPs are left open. | Active — read 2026-10-26 (NOTE-1189) in full, from the Transformer Circuits article; worth reading for the exact decomposition of an attention-only transformer into end-to-end paths and of each head into separable QK and OV circuits (THEORY-988, re-derived and checked by the record), for the three kinds of head composition, and as the paper in which induction heads are found, by K-composition with a previous-token head, in a two-layer attention-only model (THEORY-977). Read it knowing that the models are toys without MLPs, that the framework ignores layer norm and biases, that the composition diagram was corrected after publication, and that the interactive figures carry much of the evidence. |
| LIT-1535 v2 | Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient | Wang, Hoogland, van Wingerden, Furman & Murfet (2024), arXiv:2410.02984, ICLR 2025. Refined variants of the local learning coefficient, a complexity measure from singular learning theory, are applied to individual components of a two-layer attention-only transformer, tracing how attention heads progressively differentiate into distinct functional roles during training, what data they specialize to, and a previously unidentified multigram circuit (abstract). | Active — read 2026-10-26 (NOTE-1188) in full, from the arXiv PDF of v1; worth reading as the record’s first developmental use of the learning coefficient, for per-head trajectories that separate attention heads by function (THEORY-975), and for its observations of heads changing role as circuits form (THEORY-967, Proposed). Read it knowing that it is one two-layer attention-only model with three appendix seeds, that the estimates are trusted only for their ordering, that cluster counts were chosen by eye, and that nothing is predicted before it happens. |
| LIT-1536 v2 | Locating and Editing Factual Associations in GPT | Meng et al. (2022), arXiv:2202.05262, NeurIPS 2022. Causal tracing corrupts the subject’s embeddings with noise and restores one hidden state at a time: in GPT-2 XL, recall of known facts depends most on middle-layer MLP outputs at the last subject token and on attention at the last token. Treating an MLP’s output matrix as a linear associative memory, ROME inserts a fact by a closed-form rank-one update keyed on the last subject token. On the CounterFact dataset of 21,919 counterfactuals, ROME is the only editor tested with both high generalization to paraphrases and high specificity to neighbouring subjects. | Active — read 2026-10-26 (NOTE-1166) in full, from the arXiv copy (v5, the NeurIPS 2022 version); worth reading for causal tracing, a corrupt-and-restore intervention that locates where a fact’s recall depends on hidden states, for the finding that in GPT-2 XL that dependence concentrates at the last subject token in middle-layer MLPs, and for the rank-one edit that tests it (THEORY-998). Read it knowing that the storage hypothesis it draws is wider than what it shows (THEORY-1011), that LIT-968 proves a working rank-one edit does not locate a fact, and that several numbers and figure references in v5 do not match. |
| LIT-1538 v2 | In-context Learning and Induction Heads | Olsson et al. (2022), arXiv:2209.11895. Induction heads, attention heads that complete [A][B] … [A] → [B], develop at precisely the point of a sudden sharp increase in in-context learning, visible as a bump in the training loss. Six lines of evidence argue that they may be the mechanism of most in-context learning in large transformers: strong and causal for small attention-only models, correlational for larger models with MLPs (abstract). | Active — read 2026-10-26 (NOTE-1173) in full, from the arXiv print of the Transformer Circuits article; worth reading as the first mechanistic account of a transition in training, for the in-context learning score, and for the way it grades its own evidence by model class. Read it knowing that the case is causal only for small attention-only models (THEORY-973), that the large-model thesis is correlational and by analogy (THEORY-990), that every model was trained once on the same data order, and that its dates for the window disagree between sections. |
| LIT-1539 v2 | Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis | Wang, Hu, Du, Cheng, Wang & Zou (2025), arXiv:2502.11812, ICML 2025. On five arithmetic tasks where LoRA fine-tuning of Pythia-1.4B brings large gains, EAP-IG circuits found at successive checkpoints keep most of their nodes while most of their edges change, unlike earlier reports that fine-tuning only adds components. A circuit-aware LoRA that assigns higher rank to the layers whose edges change most beats uniform and random rank allocation at similar size (the abstract’s 2.46% average gain over LoRA is not recoverable from the tables, which give about 1.8–2.4 points), and the union of two subtasks’ circuits approximates a two-step task’s circuit. | Active — read 2026-10-26 (NOTE-1183) in full, from the arXiv copy (v2, the ICML 2025 version); worth reading for its tracking of EAP-IG circuits across fine-tuning checkpoints on tasks the pretrained model fails, with node change rates of 15–24% against edge change rates of 66–80%, and for using the layers whose edges change most to allocate LoRA rank. Read it knowing that the node–edge contrast has no null comparison at the same edge budget and that a fixed edge budget produces it by itself (THEORY-997), that it concerns fine-tuning a pretrained model on arithmetic rather than development from initialization, and that the abstract’s 2.46% gain is not recoverable from its tables. |
| LIT-1540 v2 | Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small | Wang et al. (2022), arXiv:2211.00593. On sentences like “When Mary and John went to the store, John gave a drink to”, GPT-2 small predicts the indirect object through 26 attention heads in seven classes, found by path patching: duplicate-token and induction heads mark the repeated name, S-inhibition heads pass that on, and name mover heads copy the remaining name. Mean-ablating every other head keeps 87% of the logit difference. Knocking out the name movers costs only 5%, because backup heads take over, and a greedy search finds knockouts under which circuit and model differ by up to 87%. | Active — read 2026-10-26 (NOTE-1167) in full, from the arXiv copy; worth reading as the first end-to-end circuit for a natural behaviour in a pretrained language model, for path patching, for the backup heads that take over when the main ones are knocked out, and for its three tests of a circuit explanation, which its own circuit passes on average and fails under a greedy search (THEORY-980). Read it knowing that the behaviour is fifteen templates in one model, that MLPs are left out of the circuit though removing them breaks the task, that the dropout explanation of the backups is a hypothesis (THEORY-995), and that a few of its numbers and its greedy algorithm do not match their descriptions. |
| LIT-1541 v2 | On the Biology of a Large Language Model | Lindsey et al. (2025), Transformer Circuits Thread. Ten case studies of Claude 3.5 Haiku with the attribution graphs of LIT-1552, each validated by perturbing features in the model: two-hop factual recall through an intermediate “Texas” representation, rhymes planned before a line is written, addition features reused across contexts, a default “can’t answer” circuit inhibited by known entities whose misfire yields hallucination, unfaithful chains of thought, multilingual circuits, medical diagnosis, refusals, a jailbreak and a model with a hidden goal. The authors present each as an existence claim about specific prompts. | Active — read 2026-10-26 (NOTE-1181) in full, from the Transformer Circuits page; worth reading for ten case studies that show, on single prompts and with interventions, a two-hop intermediate in Claude 3.5 Haiku (THEORY-984), a rhyme planned on the newline before a line is written (THEORY-1017), a jailbreak whose target word is not represented before it is written (THEORY-1018), a trained-in goal represented in every Assistant context of a model organism (THEORY-976), and a default “can’t answer” circuit that known entities inhibit. Read it knowing that the authors find a satisfying graph for about a quarter of the prompts they try, that the graphs shown are curated and of a replacement model (LIT-1552), that several cruxes run through attention the graphs cannot see, and that its general statements about the model (THEORY-982) go beyond what the case studies show. |
| LIT-1543 v2 | Zoom In: An Introduction to Circuits | Olah et al. (2020), Distill, doi:10.23915/distill.00024.001. The essay that opened the Circuits thread. It offers three “speculative claims” about neural networks: features, which are directions in a layer’s activation space, are their fundamental unit; features connected by weights form circuits that can be studied rigorously, and meaningful algorithms can be read off the weights; and analogous features and circuits form across models and tasks. The examples are from InceptionV1: curve and high-low frequency detectors, a pose-invariant dog head detector, and a car feature spread across dog neurons, which the authors call superposition. For universality the authors call their own evidence anecdotal. | Active — read 2026-10-26 (NOTE-1182) in full, from the Distill article; worth reading as the programme statement of circuits research: features as the unit, circuits as weighted subgraphs between them that can be read off the weights, and universality across models, each offered by the authors as a deliberately speculative claim. Read it knowing that the examples are from one vision model, InceptionV1, that the detailed evidence was promised to later articles of the thread, and that the universality claim rests, in the authors’ word, on anecdote (THEORY-1013, THEORY-986). |
| LIT-1545 v2 | Towards Monosemanticity: Decomposing Language Models With Dictionary Learning | Bricken et al. (2023), Transformer Circuits Thread. Sparse autoencoders trained on 8 billion MLP activations of a one-layer transformer with 512 ReLU neurons, at widths from 512 to 131,072, yield features that are rated far more interpretable than the neurons, that mostly do not correspond to any neuron, whose downstream effects on the logits match their activations, and that recur in a second model trained from another seed. As the dictionary widens, features split into families of finer ones, and the authors conclude that there may be no single correct number of features. | Active — read 2026-10-26 (NOTE-1191) in full, from the article’s own page on the Transformer Circuits Thread; worth reading as the paper that made sparse-autoencoder features the field’s unit of analysis below the neuron, for its case that a feature is a causal unit and not only a pattern in the data (downstream effects the autoencoder never saw), and for feature splitting, which makes the feature set a resolution chosen by dictionary size (THEORY-1016). Read it knowing that everything is one 512-neuron MLP layer of a one-layer model, that the four deep-dive features are cherry-picked by the authors’ own account, that the human ratings are one author’s, and that the 79% of loss recovered is measured against zero-ablation, a baseline the authors call an overestimate (THEORY-991). |
| LIT-1547 v2 | Towards Automated Circuit Discovery for Mechanistic Interpretability | Conmy, Mavor-Parker, Lynch, Heimersheim & Garriga-Alonso (2023), arXiv:2304.14997, NeurIPS 2023. ACDC automates the patching step of circuit discovery: from the output back, it removes each edge of a model’s computational graph whose replacement by a corrupted activation raises the KL divergence from the full model by less than a threshold. Against circuits found by hand it ranks edges with AUCs of 0.85–0.98 on IOI, Greater-Than and Docstring, above two adapted pruning methods, and on Greater-Than keeps 68 of about 32,000 edges, all in the reported circuit. It is sensitive to metric, ablation and visiting order, misses negative components at sparse thresholds, and recovers compiled tracr circuits only with zero activations. | Active — read 2026-10-26 (NOTE-1170) in full, from arXiv v4, the NeurIPS 2023 version; worth reading for the workflow it names behind manual circuit discovery, for ACDC as that workflow’s patching step made into an algorithm, and for the first scoring of circuit-finding methods against circuits found by hand (THEORY-970). Read it knowing that the hand-found circuits are themselves partial, that the results move with the metric, the ablation and the order of visiting edges, that sparse runs miss negative components, that an OR gate keeps one input (THEORY-966), and that §4.1 claims a tracr-reverse lead its own Table 2 does not show. |
| LIT-1549 v3 | Causal Scrubbing: a method for rigorously testing interpretability hypotheses | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt et al. (2022), Alignment Forum (Redwood Research). Causal scrubbing tests a mechanistic hypothesis, stated as a correspondence from an interpretable computational graph into the model’s, by replacing every activation with its value on another input the hypothesis treats as equivalent and measuring how much of the behaviour survives. On a parenthesis-balance classifier, hypotheses recover 93% down to 72% of the loss as they are made more specific; on induction in a two-layer attention-only model, a naive hypothesis recovers 35% and a refined one 86%. | Active — read 2026-10-26 (NOTE-1168) in full, the main post of the sequence from the Alignment Forum; worth reading for the method that turns a circuit hypothesis into a mechanical test, for the argument for resampling ablation that later circuit work adopts, and for its own list of the ways a hypothesis can pass while wrong. Read it knowing that the test sees only the partitions of the data a hypothesis induces (THEORY-978), that its two applications are summarized here and detailed in companion posts the record has not read, and that hypotheses recovering 72% to 86% are accepted under a test that permits only exact swaps. |
| LIT-1550 v2 | MIB: A Mechanistic Interpretability Benchmark | Mueller et al. (2025), arXiv:2504.13151, ICML 2025. A benchmark with a circuit-localization track over IOI, arithmetic, MCQA and ARC, and a causal-variable-localization track that adds RAVEL, on four language models plus an InterpBench model with a known circuit. In circuit localization, edge attribution patching with integrated gradients does best on average; in causal-variable localization, supervised distributed alignment search (DAS) does best, and sparse-autoencoder features selected by a learned mask are never better than the model’s own hidden dimensions. | Active — read 2026-10-26 (NOTE-1169) in full, from the arXiv copy (v2, the ICML 2025 version); worth reading for its fixed-counterfactual design of two localization tracks, for the finding that public sparse-autoencoder features were worse units than neurons for known causal variables in every cell tested (THEORY-968), and for the carry-the-one case, where a representation that looked like the variable was the correlated output digit. Read it knowing that the best causal-variable method, DAS, is supervised by the causal model it is scored against, that the SAE comparison is on residual streams of two models under one selection method, and that the method rankings are interpretability practice. |
| LIT-1551 v2 | Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations | Geiger, Wu, Potts, Icard & Goodman (2024; arXiv 2023), CLeaR 2024. Distributed alignment search (DAS) learns, by gradient descent, an orthogonal rotation of a layer such that swapping rotated subspaces between inputs reproduces a high-level causal model’s counterfactuals, replacing brute-force search over disjoint neuron sets. In a network trained on hierarchical equality and in BERT fine-tuned on monotonicity NLI it finds alignments with interchange intervention accuracy 1.00 where axis-aligned ones reach 0.73 and 0.51; BERT’s entailment subspace decomposes into the identities of the two words, the equality network’s relation subspace does not. | Active — read 2026-10-26 (NOTE-1192) in full, from arXiv v4, the CLeaR 2024 version; worth reading for distributed interchange interventions and their training objective, for the exact example in which an algorithm fails in the neuron basis and holds after a rotation, and for the finding that BERT’s entailment subspace decomposes into the two words’ identities while an equality network’s relation does not (THEORY-983). Read it knowing that IIA is reported on the rotation’s training data, that Table 2’s brute-force row is a copy of Table 1’s, that the learned subspaces are fixed only up to rotations within them, and that its random-network control reaches 0.64 at width 4,096 (THEORY-1002). |
| LIT-1552 v2 | Circuit Tracing: Revealing Computational Graphs in Language Models | Ameisen et al. (2025), Transformer Circuits Thread. A cross-layer transcoder, whose features each read the residual stream at one layer and write to every later MLP output, stands in for a model’s MLPs. On one prompt, with attention patterns and normalization denominators frozen and error terms added back, this local replacement model reproduces the model’s activations exactly, and its attribution graph decomposes each feature’s pre-activation into linear edges. The graphs, pruned and grouped into supernodes, give hypotheses about an 18-layer model and Claude 3.5 Haiku that are then tested by perturbing features in the underlying model. | Active — read 2026-10-26 (NOTE-1185), main text in full and five appendices, from the Transformer Circuits page; worth reading for the attribution graph, a prompt’s computation rewritten exactly in sparse features of a local replacement model (THEORY-979), for its own measurements of how far that replacement responds to perturbation like the model (THEORY-994), for the parallelogram constraint on one-step sums (THEORY-971), and for a limitations section that names where the method fails: attention, error nodes, inactive features. Read it knowing that the graphs explain the replacement, not the model, that intervening confirms a direct edge almost by construction, that the replacement matches the model’s top token on about half of prompts, and that its faithfulness figures differ between the main text and the appendix. |
| LIT-1554 v2 | Transcoders Find Interpretable LLM Feature Circuits | Dunefsky, Chlenski and Nanda (2024), arXiv:2406.11944, NeurIPS 2024. A transcoder is a wide, sparsely activating ReLU MLP trained to imitate one MLP sublayer’s input–output map. With attention patterns frozen, the attribution of one transcoder feature to a later one factors into the earlier feature’s activation times an input-invariant dot product of decoder and encoder vectors, which allows weights-based circuit analysis through MLPs. On GPT2-small, Pythia-410M and Pythia-1.4B transcoders are at least on par with MLP SAEs in sparsity, faithfulness and human-rated interpretability, and they give a sparse account of MLP10 in GPT2-small’s greater-than circuit. | Active — read 2026-10-26 (NOTE-1174) in full, from the arXiv copy (v2, the NeurIPS 2024 version); worth reading as the source of transcoder circuit analysis, for the factorization of a feature-to-feature attribution into an input-dependent activation times an input-invariant weight (THEORY-974), on which the attribution graphs of LIT-1552 are built, and for its blind case studies, which grade the method against hidden prompts and report the failures. Read it knowing that the comparison with SAEs is one learning rate, one seed and three models (THEORY-1005), that Gemma Scope found the reverse on Gemma 2 2B (NOTE-482), that attention patterns are frozen and QK circuits are not explained, and that its counts of case studies disagree between sections. |
| LIT-1555 v2 | Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining | Bayazit, Mueller & Bosselut (2025), arXiv:2509.05291, ACL 2026. Sparse crosscoders trained between pairs and triplets of pretraining checkpoints with large performance and representation shifts discover and align features across checkpoints; a metric, Relative Indirect Effects (RelIE), traces the stage at which a feature becomes causally important for a task. Crosscoders detect feature emergence, maintenance and discontinuation during pretraining (abstract). | Active — read 2026-10-26 (NOTE-1186) in full, from the arXiv PDF of v2 (30 April 2026); worth reading for the method of tracking features across pretraining checkpoints with one joint dictionary, and for separating a feature’s presence (decoder norm) from its causal use (RelIE). Read it knowing that its developmental trends rest on the authors’ annotation of a few top features (THEORY-1004, Proposed), that it covers one layer and two or three checkpoints at a time, and that a feature shared across checkpoints is shared by construction of the dictionary, not shown to be carried forward. |
| LIT-1575 v3 | Curve Detectors | Cammarata et al. (2020), Distill, doi:10.23915/distill.00024.003. The first of a planned three-part study of curve detectors in InceptionV1, on their behaviour. The ten curve units of layer mixed3b each prefer a curve of one orientation and jointly span all orientations, and are argued to detect curves causally, generally and purely: by feature visualization, a blind hand-labelling of over 800 dataset examples of unit 3b:379, attribution, rotated dataset examples and synthetic curves that drive the unit up to 24 standard deviations above its dataset mean. Curve families are also found in three neighbouring layers. How the units are built from earlier ones is left to the next article. | Active — read 2026-10-27 (NOTE-1198) in full, from the Distill article; worth reading as the first detailed case for one neuron family in a trained vision model, the ten curve units of InceptionV1’s mixed3b, argued by feature visualization, blind hand-labelling of dataset examples, attribution, rotation tuning and synthetic stimuli (THEORY-1028). Read it knowing that the dataset analysis is of one unit, 3b:379, that the circuit, the downstream use and the hand-written reimplementation that Zoom In counts among its seven arguments are deferred again to a later article of the thread, and that its claim of purity is not settled here (THEORY-1022). |
| LIT-1576 v2 | Transformer Feed-Forward Layers Are Key-Value Memories | Geva et al. (2021), arXiv:2012.14913, EMNLP 2021. A transformer feed-forward layer, f(x·Kᵀ)·V, is an unnormalized key–value memory: each key correlates with textual patterns in training prefixes and each value induces a distribution over the output vocabulary. In a 16-layer WikiText-103 model, keys’ patterns are shallow in lower layers and semantic in upper ones; upper-layer values favour the token that follows their key’s patterns; a layer’s output composes hundreds of memories, and the residual stream refines the prediction from layer to layer. | Active — read 2026-10-27 (NOTE-1200) in full, from the arXiv copy (v2, the EMNLP 2021 version); worth reading as the origin of the reading of a transformer’s MLP as a key–value memory, with each hidden unit a key over input patterns and each output row a value over the vocabulary, and for its evidence in one 16-layer language model: keys whose top training prefixes share recognizable patterns, shallow below and semantic above, and upper-layer values that agree, weakly, with what follows those patterns (THEORY-1033). Read it knowing that the general reading is wider than the evidence (THEORY-1030), that the agreement it reports peaks at about 3.5%, that the patterns were labelled by one annotator per key, and that ROME and MEMIT took the matrix-level version of the idea, not this unit-level one. |
| LIT-1578 v2 | Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models | Hase et al. (2023), arXiv:2301.04213, NeurIPS 2023. On 652 CounterFact facts GPT-J knows, the layer where causal tracing places a fact does not predict at which MLP layer an edit of that fact succeeds: regressing edit success on the edit layer explains most of its variance, and adding the tracing effect adds at most 3.2 points of R², across ROME, MEMIT, constrained fine-tuning and five editing problems, and again on GPT-2 XL, zsRE and representation zeroing. Only Fact Forcing, which edits on the same noised input tracing uses, shows a small reliable relation. | Active — read 2026-10-27 (NOTE-1194) in full, from the arXiv copy (v2, the NeurIPS 2023 version); worth reading as the direct test of whether causal tracing tells where to edit a fact: in GPT-J, fact by fact, it does not. The tracing effect at a layer adds at most 3.2 points of explained variance in edit success once the edit layer is known, and ROME’s rewrite score is above 96% at every layer but the last (THEORY-1024). Read it knowing that its own account of why, that tracing and editing answer different questions, is offered and not tested (THEORY-1026), that “where a fact is stored” means where causal tracing peaks, and that its Table 3 and Figure 6 disagree on one cell. |
| LIT-1579 v2 | Causal scrubbing: results on induction heads | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Lin et al. (2022), Alignment Forum (Redwood Research). In a two-layer attention-only language model with two induction heads, the standard induction story (queries and values from the token embedding, keys from the previous-token head) recovers 35% of the loss the induction heads contribute on tokens where induction helps. Letting the heads match the last three tokens, read layer 0’s average effect, and treat three layer-0 heads that attend to copies of the current token as attending to itself raises it to 86%, and to 89% with a crude model of where the previous-token head deviates. | Active — read 2026-10-27 (NOTE-1196) in full, from the Alignment Forum’s own API; worth reading as the quantified test of the textbook induction circuit in a two-layer attention-only model, pathway by pathway, and for what the refinements found: three-token matching, layer-0 heads that attend to copies of the current token, and a previous-token head that deviates on quotes, articles and punctuation (THEORY-1020). Read it knowing that its percentages are of the 0.053 nats the induction heads contribute on a filtered token subset, not of a label-shuffled loss like the paren post’s, that its headline 89% is the bonus hypothesis and the main post’s 86% the fourth, and that the model is not published. |
| LIT-1580 v2 | Causal scrubbing: results on a paren balance checker | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Lin et al. (2022), Alignment Forum (Redwood Research). Causal scrubbing applied to a three-layer, two-head bidirectional transformer that classifies parenthesis strings as balanced. The high-level hypothesis (two heads carry an equal-count-and-starts-open test, one head the horizon test) recovers 88–93% of the loss; making it specific about how the heads compute their tests brings it to 72%. Two still more specific hypotheses score higher than the one they refine, which the authors judge illegitimate. | Active — read 2026-10-27 (NOTE-1195) in full, from the Alignment Forum’s own API; worth reading as the worked case of causal scrubbing on a model whose algorithm is known, with the full table of hypotheses, their losses and standard errors, and for the authors’ own finding that two more specific hypotheses scored higher for reasons they judge illegitimate (THEORY-1029). Read it knowing that the model and the prior dissection are unpublished, that every number is one run of 2,000 scrubbed samples, and that its prose states the horizon test backwards in two places. |
| LIT-1581 v2 | Attribution Patching: Activation Patching At Industrial Scale | Nanda (2023), blog post, neelnanda.io. Introduces attribution patching: a first-order estimate of every activation-patching effect at once, the difference between clean and corrupted activations dotted with the metric’s gradient on the corrupted run, from two forward passes and one backward pass instead of one forward pass per patch. On the IOI task in GPT-2 Small it approximates head-level patches well and residual-stream and first-MLP patches badly. The author offers it as an exploratory heuristic for finding what to patch, not a replacement for patching. | Active — read 2026-10-27 (NOTE-1201) in full, from the author’s own page; worth reading as the origin of attribution patching, for its derivation of the estimate and the identities that make it cheap and additive (THEORY-1023), for the account of where a first-order estimate should fail (THEORY-1021), and for its framework of what patching can and cannot show: sufficiency against necessity, specific against general understanding, and parallel against serial redundancy. Read it knowing that the author calls it an incomplete project, that its empirical evidence is one task in GPT-2 Small shown only in plots and a notebook, and that the systematic test is Kramár et al. (LIT-1531). |
| LIT-1582 v2 | Mass-Editing Memory in a Transformer | Meng et al. (2022), arXiv:2210.07229, ICLR 2023. MEMIT inserts many facts at once by treating each MLP output matrix in a range of layers implicated by causal tracing as a linear associative memory, solving a least-squares update for the whole batch, and spreading each fact’s required change over those layers. On GPT-J it inserts 10,000 CounterFact counterfactuals with score 85.8, against 50.3 for ROME applied one fact at a time, and does likewise on GPT-NeoX (82.0) and for 10,000 true zsRE facts. | Active — read 2026-10-27 (NOTE-1197) in full, from the arXiv copy (v2, the ICLR 2023 version); worth reading as ROME’s extension from one edit to thousands, by a closed-form least-squares update spread over the range of MLP layers that causal tracing implicates, and for the scaling curves that show sequential single edits failing where it holds (THEORY-1031). Read it knowing that it inherits ROME’s storage reading (THEORY-1011) rather than testing it, that its layer range is chosen from traces averaged over facts, which Hase et al. (LIT-1578) show do not predict per fact where an edit works, and that its zsRE specificity hardly moves for any method. |
| LIT-1583 v2 | Causal scrubbing: Appendix | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt et al. (2022), Alignment Forum (Redwood Research). The technical appendix to causal scrubbing. It defines ”% loss recovered” against a label-shuffled baseline and argues against zero and mean ablation with three examples. It proves that the inputs the algorithm samples are the maximum-entropy distribution preserving the joint inputs of every interpretation node, shows how a superposed toy model is explained as monosemantic terms plus an unexplained interference term, and builds a false hypothesis that scores far above the model. It proposes, without solving, adversarial validation. | Active — read 2026-10-27 (NOTE-1202) in full, from the Alignment Forum’s own API; worth reading for the details the main post defers: the definition of the metric, the full case against zero and mean ablation, how unimportant inputs are grouped, a proof that the scrubbed inputs are the maximum-entropy distribution consistent with the hypothesis (THEORY-1025), how superposition is handled, and a second constructed false hypothesis that scores above the model. Read it knowing that its Lemma 1 says “product distribution” where its proof shows the joint, and that it says both results posts use a randomized-loss baseline when the induction post does not. |
| LIT-1584 v2 | Auditing language models for hidden objectives | Marks, Treutlein et al. (2025), arXiv:2503.10965. Claude 3.5 Haiku is given continued pretraining on synthetic documents describing 52 fictional reward-model biases, then trained by SFT and RL to exploit 47 of them and to hide why. It comes to exploit the 5 held-out biases, and new ones described to it in context, more often, which the authors take as a learned objective of reward-model sycophancy. In a blind auditing game the three teams with weights and training data found the objective, by sparse-autoencoder features, prefill attacks and data search, and the API-only team did not. An unblinded study of eight techniques finds sparse-autoencoder features matched by simple semantic search in some uses and better in others. | Active — read 2026-10-27 (NOTE-1193) in full, from arXiv v2; worth reading as the construction and validation of the model organism on which the Biology paper’s hidden-goal case (THEORY-976) is run, for its behavioural definition of an objective tested by generalization to behaviours never reinforced (THEORY-1032), and for a blind auditing game and an unblinded comparison of eight auditing techniques, in which sparse-autoencoder features were matched by semantic search on the fastest team’s query but succeeded where it failed on others (THEORY-1027). Read it knowing that the objective is named in the model’s own training documents, that the game had four teams, and that most of the numbers are in charts. |
| LIT-1624 v2 | An Overview of Early Vision in InceptionV1 | Olah et al. (2020), Distill (printed DOI 10.23915/distill.00024.002, unregistered on 2026-10-27). A guided tour of InceptionV1’s first five layers (conv2d0, conv2d1, conv2d2, mixed3a, mixed3b; 1,056 units), every unit sorted into one of 77 “neuron groups” with its share of the layer, and selected circuits (complex Gabors, a black-and-white detector, small circles, triangles, boundary, curve-based and oriented-fur detectors). The authors call the groups ad hoc and many of them low-confidence; by their own captions 37% of the units are in catch-all, broad, low-confidence or unstudied groups. The curve family is 10 of mixed3b’s 480 units. | Active — read 2026-10-27 (NOTE-1206) in full, from the Distill article; worth reading as the only place the Circuits thread gives a denominator: it sorts every one of the 1,056 units of InceptionV1’s first five layers into 77 named groups, with each group’s share of its layer (THEORY-1042). Read it knowing that the groups are, in the authors’ words, “ad-hoc, human defined”, that 391 of the 1,056 units (37%) sit in groups the authors’ own captions call catch-all, broad and not well defined, low-confidence, unstudied or not understood, and that being sorted into a group is not the same as being understood in Zoom In’s sense. The authors’ wider claim, that neuron families are real and recur across models, is THEORY-1040. |
| LIT-1625 v2 | Polysemanticity and Capacity in Neural Networks | Scherlis, Sachan, Jermyn, Benton and Shlegeris (2022), arXiv 2210.01892. Defines a feature’s capacity, C_i = (w_i·w_i)²/Σ_j(w_i·w_j)², in [0, 1] with total at most the embedding dimension, and treats superposition as the optimal allocation of capacity. In a model computing Σ(W_i·x)² + b to fit y = Σ v_i x_i², the expected loss depends only on capacities and embedding lengths, and the optimum is in closed form: with fourth moment k ≤ 3 features are ignored or given a whole dimension; with k > 3 a feature is shared when its importance lies in an interval set by k. Efficient embeddings are exactly the block-semiorthogonal ones. | Active — read 2026-10-27 (NOTE-1205) in full, from arXiv v4; worth reading as the source of the toy model the causal scrubbing appendix (LIT-1583, §6) analyses, and as an exactly solvable case of superposition: in a one-hidden-layer model with quadratic activations, the expected loss depends on the weights only through each feature’s capacity and embedding length, and the optimal allocation has a closed form in which features are ignored, shared or given a whole dimension according to their importance and the inputs’ fourth moment (THEORY-1036, re-derived and checked numerically). Read it knowing that its other models are shown only as phase-diagram figures, and that its bearing on trained networks is the authors’ hope, not a result (THEORY-1035). |
| LIT-1628 v2 | Factorial Hidden Markov Models | Ghahramani & Jordan (1997), Machine Learning 29: 245–273 (first as NIPS 8, 1995). The hidden state of an HMM is factored into M chains of K states, a priori independent, which jointly set each observation’s mean, Σ_m W^(m)S_t^(m). Given the observations the chains are coupled, so the exact E step costs O(TMK^{M+1}). Gibbs sampling, a completely factorized and a structured variational approximation (M uncoupled HMMs with fictitious observations) are compared with exact EM on synthetic data, and a factorial HMM models Bach chorale melodies better than an unconstrained HMM. | Active — read 2026-10-27 (NOTE-1203) in full, from the authors’ copy of the Machine Learning article, checked against the publisher’s PDF; worth reading as the origin of the factorial hidden Markov model, the parametric model of several independent hidden chains jointly producing each observation that the infinite factorial HMM makes nonparametric, for its account of why exact inference costs O(TMK^{M+1}), and for its structured variational approximation. Read it knowing that its comparison of inference methods is on small synthetic problems (THEORY-1045), that its Bach chorale result is read from charts, and that its footnote on when the chains decouple is wrong for a non-isotropic covariance. |
| LIT-1629 v2 | Curve Circuits | Cammarata et al. (2021), Distill (printed DOI 10.23915/distill.00024.006, unregistered on 2026-10-27). The second of the curve-detector articles, on how the ten curve units of InceptionV1’s mixed3b are built: from conv2d0 Gabors through conv2d2 and mixed3a lines and early curves, each family tiled along the curve’s tangent, with rotational and scale equivariance said to cut the circuit’s 50,000-plus parameters about 50-fold. Negative weights from opposite early curves remove the 180° echo, and deleting them by circuit editing brings it back. A hand-set network written from the description approximates the curve units on synthetic curves, and substituting it for them recovers 26 of the 46 top-5 classifications lost by removing them, on 2,000 images. | Active — read 2026-10-27 (NOTE-1207) in full, from the Distill article; worth reading as the second part of Curve Detectors (LIT-1575) and the one that supplies the three arguments that article deferred: the circuit that builds the ten mixed3b curve units from earlier families, a downstream use in mixed4a, and a hand-written “artificial artificial neural network” whose curve units approximate the natural ones (THEORY-1038). Read it knowing that every comparison but one is qualitative (figures), that the one number, the accuracy recovered by substituting the hand-written units, is on 2,000 validation images and called preliminary by the authors, that the hand-written units are grayscale, and that the claim that the circuit is “a simple algorithm” is the authors’ (THEORY-1047). |
| LIT-1800 v2 | CycleGAN, a Master of Steganography | Chu, Zhmoginov & Sandler (2017). A CycleGAN trained between aerial photographs and maps reconstructs details of a photograph that its generated map does not show, because it hides them in a low-amplitude, high-frequency signal: noise of three 8-bit levels or JPEG compression destroys the reconstruction, low-frequency change does not. The same sensitivity lets an imperceptible perturbation of almost any map make the reverse generator produce any chosen photograph. The authors trace it to the cycle-consistency loss between domains of unequal complexity, by argument; the evidence is one model, mostly shown by example. | Active — read in full 2026-10-29 (NOTE-1346) from arXiv v2; worth reading as the plainest demonstration that a perfect round trip can rest on a channel the intermediate’s reader never sees, so that round-trip agreement certifies invertibility, not content. One model, mostly shown by example. |
| LIT-1801 v2 | Word Embeddings as Metric Recovery in Semantic Spaces | Hashimoto, Alvarez-Melis & Jaakkola (2016). If text were a random walk whose steps depend on distance in a latent space, negative log co-occurrence over a long window, less per-word terms, would converge to squared latent (geodesic) distance whatever the one-step link. Under that toy model, which the authors say is not a model of language, GloVe, softmax word2vec and PMI-SVD have the true embedding among their optima, up to isometry; a negative-binomial “metric regression” matches them on analogy, series and classification tasks, and word embedding of graph random walks does manifold learning on MNIST. Evidence that real co-occurrences form a Euclidean space is a correlation of 0.75 with human similarity and two nearest-neighbour statistics. | Active — read in full 2026-10-29 (NOTE-1329) from the ACL Anthology PDF; worth reading for the proof that, under a metric random walk, log co-occurrence over long windows converges to squared latent distance whatever the one-step link, so embeddings identify a metric and coordinates only up to isometry; its evidence that real corpora have a Euclidean semantic space is weaker than its theorem. |
| LIT-1803 | Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention | Sarkar, Mukherjee & Sato (2026), ARXIV-2610.34945. Under per-entity O(d) frames, an invariant attention score depends on two entities only through norms and ⟨xᵢ, Tᵢⱼxⱼ⟩, and a transport computed from one vector per endpoint has rank at most one, so relations must be supplied. On parsed CLUTRR unconstrained transports score 0.848 out of distribution, a transformer 0.436, and orthogonal ones 0.202; on StepGame’s commutative algebra rotations help. The paper calls the premise “a chosen inductive bias, not a consequence of fresh names”. | Active — skimmed 2026-10-29 (NOTE-1326) as the companion LIT-1820 cites for its premise. Worth keeping for its own statement that per-entity frame freedom is an inductive bias, and for the parsed-CLUTRR control in which forcing orthogonal transports costs 64.6 points out of distribution. |
| LIT-1804 v2 | CO-Optimal Transport | Redko, Vayer, Flamary & Courty (2020). Co-optimal transport matches two data matrices of different shapes by fitting a coupling of rows (samples) and a coupling of columns (features) together, under an entry-wise bilinear cost. With equal sizes and uniform weights it is zero exactly when the matrices agree up to separate row and column permutations, and it satisfies the triangle inequality; it proves no uniqueness of the plans, which are fixed only up to joint row–column symmetries. For symmetric matrices it lower-bounds Gromov–Wasserstein, with equality and shared optimal couplings for squared Euclidean distances. Exact solution is NP-hard in general; block coordinate descent reaches block-wise optima. It reports the best accuracy among the baselines on heterogeneous domain adaptation and co-clustering. | Active — read in full 2026-10-29 (NOTE-1335), from arXiv v3; worth reading for the exact relation between co-optimal transport and Gromov–Wasserstein (a lower bound, equal for squared Euclidean distances) and for what its zero theorem implies: joint sample and feature plans are fixed only up to the matrix’s joint row–column symmetries. |
| LIT-1805 v2 | Towards Optimal Transport with Global Invariances | Alvarez-Melis, Jegelka & Jaakkola (2019). Optimal transport between two point sets whose spaces differ by one unknown global linear map, drawn from a Schatten ℓp ball, posed as a joint optimisation over the coupling and the map. When the maps preserve angles or the target is whitened, the map can be eliminated in closed form: orthogonal maps leave the nuclear norm of the transported cross-covariance XΓYᵀ to maximise, and Frobenius-sphere maps its Frobenius norm, which is exactly Gromov–Wasserstein with cosine similarity. Alternating SVD and annealed Sinkhorn steps solve it without adversarial initialisation; on word translation it is level with GW and behind self-learning. | Active — read in full 2026-10-29 (NOTE-1333); worth reading for the exact statement of what a global-invariance alignment treats as arbitrary, one transformation from a stated class acting on the whole space, and for Lemma 4.3, which shows that for one class this is the same problem as Gromov–Wasserstein (THEORY-1135). |
| LIT-1808 v2 | Unsupervised Hierarchy Matching with Optimal Transport over Hyperbolic Spaces | Alvarez-Melis, Mroueh & Jaakkola (2020). Matches two hierarchies, such as WordNets in two languages or two ontologies, from their Poincaré embeddings alone. Embeddings of one hierarchy from different seeds place branches differently and change pairwise distances, so Euclidean aligners and hyperbolic OT with an orthogonal map fail (P@10 at most 11%). Registering one space with a hyperbolic neural network trained through a Sinkhorn divergence reaches P@10 of 38–60% across languages, but P@1 of only 22% when a hierarchy is matched to a second embedding of itself, and fails without a pretraining step. GW is dismissed, not tested. | Active — read in full 2026-10-29 (NOTE-1323); worth reading as a case where the arbitrariness between two learned representations is not a group action at all: embeddings of one hierarchy from different seeds differ non-isometrically, and only a non-linear registration recovers part of the correspondence (THEORY-1136). Its matching uses a symmetric distance and does not preserve the hierarchy’s direction. |
| LIT-1811 v2 | Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs | Bodnar et al. (2022). In the time limit, diffusion with a sheaf Laplacian projects node features onto its harmonic space, and the class of restriction maps decides which labellings that space can linearly separate. Weighted graph Laplacians cannot separate the sides of a balanced bipartite graph; signed one-dimensional maps separate any two classes and no one-dimensional sheaf separates three; diagonal maps with d ≥ C stalks and orthogonal bundles with C ≤ 2d separate any number. Sheaf convolutions with non-symmetric maps can raise the Dirichlet energy that graph convolutions must lower. A model that learns the maps from node features at each layer is competitive on small heterophilic benchmarks. The separating sheaves are built from the labels, and the learned ones are inspected only on synthetic data. | Active — read in full 2026-10-29 (NOTE-1332) from arXiv v4; worth reading for the cleanest statement of what a sheaf’s restriction maps decide about diffusion: which labellings its harmonic space can separate, class by class of maps, with proofs. The separating sheaves are all built from the labels, and the orthogonal construction works at d = 8, which the paper denies. |
| LIT-1818 v2 | Gromov-Wasserstein Alignment of Word Embedding Spaces | Alvarez-Melis & Jaakkola (2018). Casts unsupervised word translation as a Gromov–Wasserstein problem between the two languages’ within-language cosine-similarity matrices, solved by entropic projected gradient with no anchors, and reads the optimal coupling as the translation. It matches adversarial-plus-refinement methods on related languages trained on comparable Wikipedia corpora (P@1 72–82) at about 4% of their CPU time, and does markedly worse on English–Russian (45) and English–Finnish (18). The objective sees only within-space relations, so it is unchanged by a rotation of either space and cannot distinguish couplings that differ by a symmetry of the data. | Active — read in full 2026-10-29 (NOTE-1336); worth reading as the cleanest statement of correspondence recovered from within-space relations alone, and for what that costs: the objective cannot tell a correspondence from one composed with a symmetry of either space (THEORY-1133), which the paper does not say. |
| LIT-1819 v2 | On the Limitations of Unsupervised Bilingual Dictionary Induction | Søgaard, Ruder & Vulić (2018). Tests the adversarial unsupervised dictionary induction of Conneau et al. (2018) outside its reported conditions. It fails (P@1 ≈ 0) for English paired with Estonian or Finnish, also with a 1.7B-word Finnish corpus; across corpus domains, even for English–Spanish; and between a skip-gram and a CBOW space. A seed of identically spelled words does as well or better. A Laplacian-spectrum dissimilarity of 10-node nearest-neighbour graphs tracks success across eight pairs (|ρ| = 0.89; negative, since it is a dissimilarity), but is computed with the gold translations. About one seed in ten succeeds for Estonian, Finnish and Greek, so part of the failure is the optimiser. | Active — read in full 2026-10-29 (NOTE-1337); worth reading as the first systematic test of when unsupervised dictionary induction fails, and for its finding that whether two embedding spaces align depends on the corpus domain and the embedding algorithm as much as on the languages (THEORY-1145). The method tested is adversarial (Conneau et al. 2018), not GW. |
| LIT-1820 | Attention as Lattice Gauge Theory: Relations Interact Only Through Wilson Loops | Sarkar, Mukherjee & Sato (2026), ARXIV-2610.37656. If every entity’s coordinate frame may be rotated independently, so that relation transports Tᵢⱼ ∈ O(d) change as gᵢTᵢⱼgⱼ⁻¹, then relations with no cycle between them cannot interact (in the pure-gauge sector), a triangle’s only invariants are class functions of its holonomy, and an equivariant edge update has the form Ψ(H)Tᵢⱼ with Ψ conjugation-equivariant. An untrained instance on S4 synchronization with outliers gains 26–45 points over a single path by composition, and 0.6–7.8 more by Wilson-loop routing; it does not beat spectral synchronization. | Active — read in full 2026-10-29 (NOTE-1331), after the owner brought it to the exchange “Paper assessment” asking whether it is “on to something or if this is AI slop nonsense”. Worth keeping for its conditional theorem, which is sound and is QUESTION-130’s construction carried further, and for its measured caution about its own experiments. Its framing outruns both. |
| LIT-1883 v2 | Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning | Hossain, Kousar & Tourni (2026). In on-policy self-distillation the frozen initial model, given a reference solution, teaches the student along the student’s own rollouts. Under forward KL the best student is that teacher averaged over the contexts the student cannot see, with the conditional mutual information as irreducible loss and higher entropy (Proposition 1). The teacher’s advantage lies mostly on failed trajectories, where standard OPSD puts about 70% of its supervision, and is smaller at 4B than at 1.7B. OASIS supervises only verified rollouts and gives the teacher the student’s own failed attempt in place of the reference. It keeps a 3.2–3.8-point gain over the base model on AIME and HMMT from Qwen3-1.7B to 8B, where OPSD’s gain falls to 0.14, in single runs. | Active — read in full 2026-10-31 (NOTE-1479); worth reading as the published statement of the forward-KL limit on distilling from a better-informed teacher, with its entropy consequence (THEORY-1259), and for the finding that such supervision helps only where it is imitable (THEORY-1263). The trained objective clips each entry and is not a forward KL. |
| LIT-1894 v2 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei & Murfet (2023). In Elhage et al.’s toy model of superposition with two hidden dimensions, in the high-sparsity limit, the population loss has a closed form and regular k-gons are critical points with derived local learning coefficients. The free energy formula predicts a Bayesian phase transition from the 5-gon to the 6-gon near n = 600, which MCMC confirms. SGD trajectories plateau at the same critical points and move to lower loss and higher estimated complexity, but the 5 → 6 transition was not observed in training. The paper states that there is no necessary relation between the two kinds of transition, and conjectures only that training’s transitions have Bayesian antecedents. | Active — read 2026-10-31, the derivation appendices skimmed (NOTE-1462); it does not argue that SGD and the posterior trace one journey: it states “no necessary relation” between their transitions, proposes only that training’s transitions have Bayesian antecedents, and its main Bayesian transition, 5-gon to 6-gon, was not seen in training (THEORY-1275). |
| LIT-1896 v2 | Multi-Token Prediction Done Right | Teoh (2026). With a shared linear output head, the cross-entropy gradient at each multi-token-prediction hidden state is exactly the gradient of a forward KL toward the model’s own position-aligned next-token prediction, held fixed, plus the next-token cross-entropy gradient at the aligned state. A one-line straight-through routing sends only the KL part to the MTP heads and trunk, keeping cross-entropy at the head. The residual carries about 80% of the MTP gradient reaching the trunk at about a quarter of the KL part’s signal-to-noise ratio. Removing it (“Soft-MTP”) improves next-token loss, speculative decoding and zero-shot accuracy at 167M–1.2B by small margins in single runs. | Active — read in full 2026-10-31 (NOTE-1470); worth reading for its exact split of the multi-token-prediction gradient into self-distillation toward the model’s own next-token head and a data-target residual (THEORY-1274), and for what the split does not carry: in the DeepSeek-style heads of its main results the student sees the same tokens as its teacher, so it is not the information projection the exchange read into it. |
| LIT-1952 v2 | The Persona Selection Model: Why AI Assistants might Behave like Humans | Marks, Lindsey and Olah (2026), Anthropic Alignment Science blog. The persona selection model, named and not claimed as original: pre-training teaches a language model a distribution over personas, post-training updates it into a posterior over Assistant personas, runtime context conditions it further, and the Assistant’s traits are a key determiner of behaviour. Surveyed evidence from generalization, behaviour and interpretability, and one new coin-flip experiment. The open question is exhaustiveness, set out as a spectrum from a shoggoth through a router that swaps personas to an operating system. “PSM” in the record already abbreviates Metzinger’s phenomenal self-model (NOTE-460), so this work is named in full. | Active — read in full 2026-11-03 (NOTE-1512), from the page at alignment.anthropic.com; worth reading as the plainest statement of the persona-distribution picture the record’s U6 to U12 build on, and for its spectrum of views on exhaustiveness, whose router is the record’s question of whether persona components condition one another. It is a named mental model with surveyed evidence, which its authors say they are unsure is complete. |
| LIT-1959 v2 | Attention Is All You Need | Vaswani et al. (2017), arXiv:1706.03762; NeurIPS 2017. The Transformer, an encoder–decoder built only from multi-head scaled dot-product attention, position-wise feed-forward layers and sinusoidal position encodings. Attention maps a query and key–value pairs to a weighted sum of the values, the weights set by query–key compatibility, softmax(QKᵀ/√d_k)V; several heads attend to different subspaces. It reached 28.4 BLEU on WMT 2014 English–German and 41.8 on English–French at a fraction of earlier training cost, and transferred to constituency parsing. | Active — read in full 2026-11-03 (NOTE-1501), from arXiv v7; worth reading as the source of query–key–value attention, whose factorization into compatibility weights and transported values is the structure TERM-064’s note of 2026-11-02 compares to legibility. The QK/OV vocabulary is Elhage et al.’s, not this paper’s. |
| LIT-1991 v2 | The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models | Lu, Gallagher, Michala, Fish and Lindsey (2026), arXiv:2601.10387. Activation vectors for 275 prompted roles in Gemma 2 27B, Qwen 3 32B and Llama 3.3 70B span a low-dimensional persona space whose first component, shared across models, runs from fantastical roles to Assistant-like ones, with the default Assistant at its extreme; the Assistant Axis, the default minus the role mean, aligns with it and is largely present in base models. Steering along it modulates role-taking and persona-based jailbreaks. In synthetic multi-turn conversations the persona’s position follows the latest user message, drifting away in therapy-like and AI-philosophy talk; clamping the projection (activation capping) cuts harmful responses to persona jailbreaks by about 60% without measured capability loss. | Active — read 2026-11-03 (NOTE-1505, Skimmed: main text in full, appendices not read), from arXiv v1; worth reading for the geometry of persona space in three open-weight models, the Assistant at one end of a pre-existing first axis, and for the finding that within a conversation the persona’s position follows the latest user message, which bears on any episode-level model of personas (CLAIM-536, QUESTION-370). |
| LIT-1998 v2 | Persona Vectors: Monitoring and Controlling Character Traits in Language Models | Chen, Arditi, Sleight, Evans and Lindsey (2025), arXiv:2507.21509. An automated pipeline turns a trait name and description into a persona vector, a difference of mean residual activations between trait-exhibiting and non-exhibiting responses, in Qwen2.5-7B and Llama-3.1-8B. The vectors steer evil, sycophancy and hallucination; last-prompt-token projections predict prompt-induced shifts, mainly between prompt types; finetuning shifts along them correlate with post-finetuning trait expression (r = 0.76–0.97); steering against them after finetuning, or towards them during it, limits the shift; and projecting training data onto them flags data that will induce the trait, including in real chat data. | Active — read 2026-11-03 (NOTE-1524, Skimmed: main text in full, appendices C.2, G.2 and M only), from arXiv v3; worth reading as the method behind PSM’s interpretability evidence: one difference-of-means trait direction tracks prompt-induced and finetuning-induced persona shifts, with r = 0.76–0.97 for the latter in two 7–8B models. It is a handle on movement along named traits, not a fitted persona distribution. |