representation-learning.
187 of 1538 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-199 v2 | Balestriero & LeCun 2022 — SSL recovers spectral embedding | VICReg, SimCLR and Barlow Twins each solve, in closed form, a classical spectral embedding problem on the positive-pair graph G (Laplacian Eigenmaps, a generalized MDS/ISOMAP, and CCA respectively), so contrastive vs non-contrastive SSL is global vs local spectral embedding, and when G matches the downstream task any of them is optimal. | Superseded — by NOTE-752 |
| NOTE-201 v2 | Assran et al. 2022 — hidden uniform cluster prior in SSL | Joint-embedding SSL methods that rely on mini-batch volume maximization (SimCLR, VICReg, SwAV, MSN) implicitly impose a prior toward features that cluster the data into equal-sized groups, which hurts on class-imbalanced data; replacing it with a power-law prior (PMSN) improves representations on long-tailed data. | Superseded — by NOTE-850 |
| NOTE-207 v2 | MDL generalization guarantees for representation learning | Generalization of representation learning is bounded not by I(X;U) but by the minimum description length of the latent variables (or predicted labels) under a symmetric prior — a multi-letter relative entropy that reflects the encoder’s structure and is non-vacuous for deterministic encoders. | Superseded — by NOTE-742 |
| NOTE-208 v2 | Sun et al. 2023 — spectral theory of novel class discovery | For novel class discovery, a spectral contrastive loss over a graph of labeled and unlabeled data (NSCL) is equivalent to factorizing its adjacency matrix, and the resulting linear-probe error on novel classes is bounded — down to zero — by how far the known classes’ feature span covers the unlabeled data’s “ignorance space”. | Superseded — by NOTE-833 |
| NOTE-209 v2 | Tan et al. 2023 — contrastive learning is spectral clustering | SimCLR’s unmodified InfoNCE loss is equivalent to spectral clustering on the augmentation-defined similarity graph (min tr(Zᵀ L(π) Z) + log R(Z)), CLIP does generalized spectral clustering on the image–text pair graph, and mixtures of exponential kernels (Kernel-InfoNCE) beat the Gaussian kernel. | Superseded — by NOTE-846 |
| NOTE-210 v2 | The Conditional Entropy Bottleneck | Training a representation to minimize I(X;Z|Y) while maximizing I(Y;Z) — the Conditional Entropy Bottleneck, targeting the “Minimum Necessary Information” point I(X;Y)=I(X;Z)=I(Y;Z) — empirically improves accuracy, adversarial robustness, OoD detection and calibration and refuses to memorize random labels. | Superseded — by NOTE-754 |
| NOTE-211 v2 | Knowledge Sheaves (sheaf-theoretic KG embedding) | Knowledge-graph embedding can be recast as learning an approximate global section of a cellular sheaf on the schema graph, which subsumes Structured Embedding and TransE/TransR-style models and yields, via harmonic extension, a training-free way to answer composite multi-hop queries. | Superseded — by NOTE-843 |
| NOTE-218 v2 | Simon et al. 2023 — stepwise nature of SSL | Joint-embedding SSL learns its embedding one dimension at a time in discrete, well-separated steps — in a linearized Barlow Twins model it provably learns the top eigenmodes of a contrastive kernel in order, so SSL is to kernel PCA what supervised learning is to kernel regression — and the same stepwise rank growth appears in ResNets trained with Barlow Twins, SimCLR and VICReg. | Superseded — by NOTE-741 |
| NOTE-219 v2 | Duality of Bures and Shape Distances | The Procrustes shape distance between two representations (optimal rotational alignment of units) equals the Bures distance between their centred linear kernel matrices, so the rotation-invariant geometry of a representation is exactly the geometry of its kernel, and through Uhlmann’s theorem the fidelity of two kernels is the maximum overlap over all representations consistent with them. | Superseded — by NOTE-753 |
| NOTE-221 v2 | Closeness in Distribution vs Representational Similarity | For softmax models p(y|x) ∝ exp(f(x)ᵀg(y)) under a diversity condition, the model distribution determines the embeddings and unembeddings up to an invertible linear map (f = Af′, g₀ = A⁻ᵀg′₀). The determination is not stable under KL divergence, though: two models can be ε-close in KL, even both near maximum likelihood, while their embeddings are far from linearly equivalent. | Superseded — by NOTE-738 |
| NOTE-222 v2 | Tosh et al., contrastive learning and multi-view redundancy | The logistic contrastive problem of telling a same-datum pair of views from an independent pair is solved over all functions by the pointwise mutual information f*(x,z) = log p(x,z)/(p(x)p(z)), whose exponential g* is the change of measure from p(z) to p(z|x), and linear functions of the landmark embedding (g*(x,Z_1),…,g*(x,Z_m)) are near-optimal predictors whenever the two views are redundant for the label. | Superseded — by NOTE-841 |
| NOTE-223 v2 | Similarity of NN Representations Revisited (CKA) | A similarity index for neural representations should be invariant to orthogonal transformation and isotropic scaling but not to arbitrary invertible linear maps, and comparing the example-by-example Gram matrices (linear or kernel CKA) achieves exactly that, while any index invariant to invertible linear maps becomes uninformative once width reaches the number of examples. | Superseded — by NOTE-743 |
| NOTE-224 v2 | Belghazi et al., MINE | Mutual information is stated measure-theoretically as the expectation under the joint of log dP_XZ/d(P_X⊗P_Z), and its Donsker–Varadhan dual is tight exactly at T* = log dP/dQ + C, so a neural critic trained on the dual is a log Radon–Nikodym-derivative estimator. | Superseded — by NOTE-828 |
| NOTE-225 v2 | What Similarity Measures Imply about Decodability | For linear readouts regularised by w ↦ wᵀG(X)w, the optimal decoded signal is K_X z with K_X = X G(X)⁻¹ Xᵀ / M. The expected alignment of two networks’ optimal readouts is therefore Tr(K_X K_z K_Y), which recovers linear CKA, CCA, GULP and ENSD as average decoding similarities. Procrustes distance bounds the average decoding distance from both sides, with constants set by a participation ratio. | Superseded — by NOTE-834 |
| NOTE-227 v2 | Spectral contrastive loss (HaoChen et al.) | Minimizing the population spectral contrastive loss L(f) = −2E[f(x)ᵀf(x⁺)] + E[(f(x)ᵀf(x⁻))²] is, up to a constant, a rank-k factorization of the normalized adjacency matrix of the augmentation graph, so its minimizers are the top-k eigenvectors (eigenfunctions, in the infinite case) of that graph up to a row scaling and an invertible linear map — neither of which changes linear-probe accuracy. | Superseded — by NOTE-740 |
| NOTE-228 v2 | Contrastive learning finds an optimal eigenbasis | The InfoNCE, NT-Logistic and spectral contrastive losses all have the same population minimizer, the positive-pair kernel K⁺(a₁,a₂) = p⁺(a₁,a₂)/(p(a₁)p(a₂)). Kernel PCA under K⁺ returns exactly the eigenfunctions of the positive-pair Markov chain, and the top-d of those span the linear-predictor subspace that minimizes worst-case approximation error over approximately view-invariant targets. | Superseded — by NOTE-751 |
| NOTE-231 v2 | Spectral Inference Networks (SpIN) | The top-N eigenfunctions of a symmetric kernel operator Kf = E_{x′}[k(x,x′)f(x′)] on L²(p) can be learned by a neural network that maximizes the generalized Rayleigh quotient Tr(Σ⁻¹Π), with Σ = E[u uᵀ] and Π = E[k(x,x′)u(x)u(x′)ᵀ]. Slow Feature Analysis is the special case where k is the graph Laplacian of adjacent video frames. | Superseded — by NOTE-824 |
| NOTE-232 v2 | Tschannen et al., on MI maximization for representations | The success of InfoNCE-style representation learning cannot be attributed to mutual information itself: MI is invariant under invertible reparametrizations, invertible encoders that maximize true MI can be worse than raw pixels, and tighter MI bounds from higher-capacity critics can give worse representations. | Superseded — by NOTE-749 |
| NOTE-233 v2 | Poole et al., variational bounds of mutual information | Every variational lower bound on mutual information in use — Barber–Agakov, Donsker–Varadhan/MINE, NWJ/f-GAN-KL, InfoNCE — is one family, tight at a critic that is a function of the log density ratio log p(y|x)/p(y), and InfoNCE is the multi-sample member that trades variance for a hard ceiling of log K. | Superseded — by NOTE-739 |
| NOTE-234 v2 | SSL-HSIC (kernel dependence maximization) | With image identity as the label, the SSL-HSIC loss −HSIC(Z,Y) + γ√HSIC(Z,Z) has a dependence term that is proportional to the average squared MMD between the per-image distributions of augmented-view representations (App. B.2). InfoNCE approximates the same term plus a variance penalty (eq. 7), so contrastive SSL separates the kernel mean embeddings of each image’s view distribution. | Superseded — by NOTE-830 |
| NOTE-240 | The Lattice Representation Hypothesis of Large Language Models | Thresholding linear attribute directions (Fisher-LDA probes) gives a crisp object–attribute incidence, and the paper’s Theorem 1 is the standard fact that any such incidence has a complete concept lattice. The geometric meet is half-space intersection; the join is defined as the set union of two cones and “approximated by the conic hull” of their directions, which as written is not a lattice join. On five WordNet domains with GPT-4o-annotated attributes, the probes recover the incidence at F1 69.7–83.2 on three 7–8B models, and profile-based subsumption reaches F1 57.1–77.1. No lattice law is ever tested. | Read |
| NOTE-242 v2 | Attention Approximates Sparse Distributed Memory | Kanerva’s SDM read weights each stored pointer by the number of neurons in the intersection of two Hamming balls of radius d. The paper shows that this weight is roughly log-linear in Hamming distance for close patterns. As a result, with L²-normalised vectors and a regression-fitted β, softmax attention reproduces SDM’s retrieval behaviour on random and learnt-projection data (App. B.7). Two things are weaker than they look. The analytic derivation (App. B.2, Eqs. 16–21) gets the exponent wrong: by my recomputation its β is ≈2–2.5× too small. The biological case is Kanerva’s 1988 cerebellar mapping, restated rather than re-argued. It also needs r = 2⁶⁴ ≈ 1.8×10¹⁹ neurons in the Attention setting, which the paper itself calls “biologically implausible” (App. B.5). | Read |
| NOTE-246 v2 | Reasoning about Meaning in Natural Language with Compact Closed Categories and Frobenius Algebras | The paper sets the sentence space equal to the noun space (F(s)=F(n)=W). It then uses the basis-copying map σ: v_i ↦ v_i⊗v_i of the Frobenius algebra fixed by W’s basis to lift corpus-built verb matrices Σ_i sbj_i⊗obj_i into rank-3 tensors. So a transitive sentence becomes sbj⊙(verb·obj) (“copy-subject”) or obj⊙(verbᵀ·sbj) (“copy-object”), a vector in W. On the Grefenstette–Sadrzadeh verb-disambiguation set, copy-object gets Spearman ρ=0.172, against 0.168 (Kron), 0.163 (Multp), 0.143 (copy-subject), 0.050 (additive) and an inter-annotator bound of 0.620. On a 112-term definition-matching task the best F1 is 0.21 (nouns) and 0.27 (verbs). No significance test is reported. The disambiguation numbers are identical to the authors’ COLING 2012 paper, and the definition numbers are the same data re-reported. | Read |
| NOTE-273 | Not All Language Model Features Are One-Dimensionally Linear | Defines a feature as irreducibly multi-dimensional when no rotation and translation makes its distribution either a product of independent parts (separable) or a mixture with a lower-dimensional component (Def. 2). Softened versions are the separability index S(f) and the ε-mixture index M_ε(f) (Def. 3). Clustering sparse-autoencoder dictionary elements by cosine similarity then finds circular representations of weekdays and months in GPT-2 (also 20th-century years) and in Mistral 7B. Replacing the circle coordinates at the day or month token causally changes the answer on “N days/months from X” prompts in Mistral 7B and Llama 3 8B, and in early layers the circle patch nearly matches patching the whole layer. The models’ task accuracy is modest on weekdays (29/49 Llama, 31/49 Mistral), high on months (143/144, 125/144), and near-trivial for GPT-2 despite its circles. | Read |
| NOTE-274 | Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges | The authors cast CNNs, GNNs, Deep Sets, Transformers, spherical, mesh and gauge CNNs, and LSTMs as instances of one “blueprint” (p. 29). Each is a stack of linear G-equivariant layers, pointwise nonlinearities and local pooling, capped by a G-invariant global pooling, with the domain Ω and its symmetry group G given in advance (p. 27: “a symmetry group G”; p. 4: “exploiting the known symmetries”). The two priors are symmetry and scale separation. The Fourier transform is derived as the joint eigenbasis of shift-commuting (circulant) operators, which requires distinct eigenvalues (p. 37). Its group-theoretic generalisation via irreducible representations is announced and deferred to “future work” (p. 43). | Skimmed |
| NOTE-278 | Toy Models of Superposition | In a toy autoencoder x′ = ReLU(WᵀWx + b), with n sparse synthetic features compressed into m < n dimensions, a linear model keeps only the top-m features, but with the output ReLU and sparse enough features the model stores more features than dimensions (“superposition”), in non-orthogonal directions whose interference the ReLU and a negative bias filter out. Whether a feature is dropped, superposed or given a dedicated dimension changes discontinuously with sparsity and importance, and in the n = 2, m = 1 theory this is a first-order phase change. With uniform features the directions settle into uniform polytopes, with feature dimensionalities clustering at 3/4, 2/3, 1/2, 2/5 and 3/8. A ReLU hidden layer creates a privileged basis in which monosemantic and polysemantic neurons coexist, and a model can compute |x| for more features than it has neurons. Everything is shown in toy models only, and the authors expect the geometry and learning-dynamics results to generalise least. | Read |
| NOTE-282 v2 | On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups | Theorem 1 says that a feed-forward network whose layers are linear maps followed by pointwise nonlinearities, with every layer indexed by a homogeneous space G/H_ℓ of a compact group G, is G-equivariant (every layer transforming by the induced action) if and only if each linear map is a generalised convolution f ∗ χ with a filter on the double coset space H_{ℓ−1}\G/H_ℓ. The “if” direction is a short coset computation. The “only if” direction goes through the group Fourier transform: equivariant maps preserve isotypic components (Lemma 6) and act on each Fourier matrix by right multiplication, M ↦ MB (Lemma 10), which the convolution theorem (Prop. 2) turns back into a convolution. | Read |
| NOTE-283 | Progress measures for grokking via mechanistic interpretability | A one-layer ReLU transformer (P = 113, 30% of pairs, full-batch AdamW with λ = 1, 40k epochs) that groks modular addition implements a “Fourier multiplication” algorithm. The embedding maps a and b to cos/sin(w_k·a), cos/sin(w_k·b) at five key frequencies k ∈ {14, 35, 41, 42, 52}, with w_k = 2πk/113. Attention and MLP form cos/sin(w_k(a+b)), and the neuron-logit map W_L (≈ rank 10) reads out Σ_k α_k·cos(w_k(a+b−c)). Progress measures built from this (restricted and excluded loss) show training splits into memorisation (0–1.4k epochs), circuit formation (1.4k–9.4k) and cleanup (9.4k–14k), with the test-accuracy jump inside cleanup. | Read |
| NOTE-286 | The Platonic Representation Hypothesis | A position paper with one argument and three measurements. Measured with mutual 10-nearest-neighbour overlap, vision models that solve more VTAB tasks agree more with each other (78 models). Language models with lower bits-per-byte agree more with DINOv2, CLIP and others on Wikipedia image–caption pairs, and this alignment tracks HellaSwag and GSM8K scores. The cross-modal score “only reaches 0.16” out of a maximum of 1 (§6), and a CKA version “revealed a very weak trend” (App. A). The formal argument (§4.2, App. F) shows only this: in an idealised world of discrete events seen through bijective observations, the Bayes-optimal NCE critic is the pointwise-mutual-information kernel plus a constant, and that kernel is representable as an inner product under a sufficient smoothness condition (Prop. F.1). So every such modality shares one kernel, up to an additive constant. | Read |
| NOTE-287 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Small decoder-only transformers (2 layers, width 128, 4 heads, ~4·10⁵ non-embedding parameters) trained on binary-operation tables over abstract symbols can reach 100% validation accuracy long after fitting the training set. In the headline run (division mod 97, 50% data, Adam with no weight decay, a budget of 10⁶ steps), training accuracy is near-perfect before 10³ steps, validation accuracy only near 10⁶, with little progress before 10⁵. The optimisation time to generalise rises steeply as the training fraction falls (on S₅ near 25–30% data, −1% data costs +40–50% median steps), and weight decay is the most data-efficient intervention tried. | Read |
| NOTE-288 v2 | Group Equivariant Convolutional Networks | Group convolution (eqs. 10–11) replaces translation by the elements of a discrete group G, so feature maps become functions on G and every layer is G-equivariant (eq. 12, with pooling and nonlinearities shown to commute with the action). With parameter counts held roughly fixed, p4-CNNs cut rotated-MNIST test error from 5.03% (planar baseline) to 2.28% (previous best 3.98%), and p4m convolutions cut plain-CIFAR10 error from 9.45% to 6.46% on a ResNet44 (Tables 1–2). | Read |
| NOTE-298 | Opening the Black Box of Deep Neural Networks via Information | The paper studies small tanh networks (layer widths down from 12 to 2) trained by SGD on a 12-bit, O(3)-symmetric synthetic rule, with mutual information estimated by binning each neuron into 30 bins over all 4,096 inputs. It reports two phases: a short “ERM” phase of a few hundred epochs in which I(T;Y) rises, and a long “compression” phase in which I(X;T) falls. The phases coincide with a drop in per-layer gradient SNR (from high-SNR drift to low-SNR diffusion, at about 350 epochs), and the converged layers sit on the IB curve at fitted β. Everything is shown on small synthetic tasks, with the explanations argued informally. | Read |
| NOTE-301 | The approximation of one matrix by another of lower rank | For a real n×N matrix a, the rank-r matrix β nearest in the least-squares (Frobenius) distance is obtained by writing a = u′λU (u, U orthogonal, λ diagonal ≥ 0, “Theorem I”) and zeroing all but the r largest diagonal elements. The solution is unique unless λ_r = λ_{r+1}. The squared error is Σ_{i>r} λ_i², so the error is at most l(1−r/R)^{1/2}, with l the length of a and R its rank. The paper also argues that the resulting canonic components cannot straightforwardly be read as psychological factors. | Read |
| NOTE-302 | The Linear Representation Hypothesis and the Geometry of Large Language Models | Three readings of “concepts are directions” are made precise with counterfactual pairs, and proved to line up. The shared direction of word-pair differences in the unembedding space is a logit-linear probe (Thm 2.2). The embedding-space direction is a steering vector that leaves causally separable concepts unchanged (Thm 2.5). Under any inner product making separable concepts orthogonal, the Riesz map sends the first to the second (Thm 3.2). Softmax training fixes the representation only up to γ ↦ Aγ + β, λ ↦ A⁻ᵀλ, so no inner product is identified, and the paper chooses M = Cov(γ)⁻¹ by setting a free diagonal D to the identity (Thm 3.4). Evidence: 27 concepts on LLaMA-2 7B (26 show a common direction; thing⇒part does not), read from histograms and heatmaps with no test statistic, plus a Gemma-2B heatmap comparison. | Read |
| NOTE-307 | Learning the Irreducible Representations of Commutative Lie Groups | Proposes “Weyl’s principle” (via Kanatani 1990) as a definition of disentangling: the elementary components of data are the irreducible subspaces of the representation of a symmetry group acting on it. It learns such a decomposition for one class of group fixed in advance, compact commutative subgroups of SO(D) (“toroidal”), from pairs (x, y) of raw data vectors related by an unobserved group element, y = W R(φ) Wᵀx + ε. The orthogonal basis W (2-D invariant subspaces) is fitted by SGD on a closed-form marginal likelihood, and the integer weights ω_j are then read off a batch rotated by a known 0.1°. The only experiment trains 100 filters on 250,000 white-noise 16×16 patches paired with random rotations of themselves (learned ω from −11 to 12), and uses the invariant √κ̂ for 1-NN on rotated MNIST, reported in a figure with no numbers. Spectral degeneracy plays no role. | Read |
| NOTE-309 | Understanding image representations by measuring their equivariance and equivalence | Given a fixed representation φ (HOG, or AlexNet Conv1–Conv5 before ReLU) and a transformation g of the input image chosen by the experimenter (flips, 90° and other rotations, rescaling by 2^{±1/2}, affine warps), it fits a linear map M_g on the representation space with φ(gx) ≈ M_g φ(x), by sparse or structured-sparse regression or by a “transformation layer” trained through the rest of the network, and fits “stitching layers” E with φ′(x) ≈ E φ(x) between different networks. In AlexNet, learned M_g recover most of the classifier’s accuracy for vertical flips and 90° rotations (top-1 error 0.75 uncompensated vs 0.43–0.51 and 0.44–0.53 across Conv1–Conv5, original 0.43), channels scored invariant rise to 99.61% for horizontal flips at Conv5, and Conv1–Conv2 are interchangeable between AlexNet and networks trained on ImageNet, Places or both. It detects no group and uses no spectrum. | Read |
| NOTE-319 | How do language models learn facts? Dynamics, curricula and hallucinations | ’An 8-layer, 44M-parameter decoder-only transformer trained with AdamW on synthetic biographies (64k individuals by default) learns factual recall in three phases. It first fits the marginal attribute-value distribution, then sits on a plateau exactly at the no-knowledge baseline, then acquires individual-specific associations. Plateau end grows with population as 0.43·N^0.81 (R² = 0.998, N = 4k–256k, 5 seeds). Attention patching from post-plateau checkpoints removes the plateau, and attention from the recall position to name tokens rises through it, so the plateau is when the attention extraction circuit forms. Nothing in the three phases is a compression of input information: the last phase adds individual-specific information.‘ | Read |
| NOTE-320 v2 | On the Information Bottleneck Theory of Deep Learning | Replicating Shwartz-Ziv & Tishby with their own code, the paper shows the information-plane “compression phase” appears only for double-saturating nonlinearities measured through an imposed binning/noise model: tanh compresses, ReLU/softplus/linear do not (binning, KDE, Kraskov; SZT task and MNIST), and re-binning the same tanh run evenly in net input removes it. Compression and generalisation dissociate in all four combinations, full-batch GD compresses as much as SGD, and the drift→diffusion gradient-SNR transition occurs in networks that cannot compress (ReLU, MNIST, a 1-1-1 linear net), so it is real but unrelated to compression. | Read |
| NOTE-322 v2 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li’s structural learning theory (StrLT, developed in the companion arXiv 2602.07974) splits multi-regime learning into discovering contexts (the “trap”, or scaffold) and predicting within one (the “funnel”, or flow). Width is the least number of cells on which a γ-contractive, low-risk predictor suffices. It is incomparable with the per-cell VC dimension, forces an error floor when fewer than w cells are allocated, and costs Ω(w log w) samples to discover. These results hold in repaired form, and are close to definitional. The CS estimator reweights a spatial graph by prediction agreement. It is shown to count components for a given predictor, not width itself. The decoupling prescription says to train routing on structural signals, not task loss, and to freeze consolidated contexts. It is a plausible, testable design hypothesis with continual-learning precedent, but the additive Rademacher bound it is said to follow from holds for joint training too. The safety section recasts alignment as scaffold alignment by analogy, and is asserted. | Read |
| NOTE-324 | Structural Learning Theory: A Metric-Topology Factorization Approach | >- The formal core of Xin Li’s structural-learning programme. Width w(P; γ, δ) is the least number of open cells on each of which some predictor is γ-contractive with conditional risk ≤ δ. The paper proves that width is incomparable with the per-cell VC dimension (Thm 2.2). It shows that with K < w cells there is an n-independent error floor η(w, K), under a non-degeneracy assumption stated in the body (Thm 2.4, Assumption 2.3). Width is at least c·β₁·L/D₀ under a bounded-overlap assumption (Thm 2.5), and discovering all basins costs Ω(w log w) samples (Thm 2.6). The CS-Laplacian eigenvalue count converges uniformly to the predictor-relative count w_G(P) when n ≳ (H_G + d_X log(1/r_x) + log(1/δ′))/(r_x^{d_X} g_eff²) (Thm 3.5); no result shows that w_G equals width. There are no experiments. | Read |
| NOTE-345 | Chalmers — Could a large language model be conscious? | Asks for a feature X that indicates LLM consciousness or its absence. Self-report is fragile and trained in. Conversation and general intelligence give some limited reason. Six obstacles stand against it: biology, senses and embodiment, world and self models, recurrence, a global workspace and unified agency, the last three strongest, all but biology temporary. On mainstream assumptions, with at least 1/3 credence per requirement, that gives under 10% for current LLMs and 25% or more for conscious LLM+ systems within a decade. The numbers are illustrative, biology is set aside rather than answered, and the twelve “challenges” are offered as roadmap or red flags. | Read |
| NOTE-360 | Selfridge — Pandemonium | A proposal, with a preliminary Morse-code trial, for pattern recognition by a hierarchy of parallel “demons”. Data demons hold the input, subdemons compute features, cognitive demons shout weighted sums, and a decision demon takes the loudest. It adapts by hill-climbing on the weights, by culling and breeding subdemons through mutation and “conjugation”, and in principle by adapting its own control demons. The only result is reported in the discussion: hill-climbing “does work”, subdemon selection “did not do what we hoped”, and the best single subdemon was right 90% of the time. | Read |
| NOTE-474 | Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra | G = Gclass + Gcross + Gwithin + Gc=c′ (Eq. 6.4); knockouts give C outliers to Gclass, a C² mini-bulk to Gcross, the bulk to Gwithin, in log-spectra. Gˡ = Σ w (hhᵀ) ⊗ (δδᵀ) (Eq. 7.8), so gc ≈ hc ⊗ δc. K-FAC Hˡ ⊗ Δˡ misaligns G’s outliers; CFAC Ave_c Hˡc ⊗ Δˡc aligns them at deep layers. For multinomial logistic regression on a Gaussian model the expected Fisher has C outliers, C(C−2) mini-bulk and a bulk (Theorem 8.1). | Read |
| NOTE-475 | Deep Equals Shallow for ReLU Networks in Kernel Regimes | For κ(xᵀx′) on S^(d−1), µ_k = (ω_(d−2)/ω_(d−1)) ∫ κ(t)P_k(t)(1 − t²)^((d−3)/2) dt (Eq. 8), shared by N(d, k) ~ k^(d−2) harmonics. Theorem 1: an endpoint expansion κ(1 − t) = p(t) + c t^ν + o(t^ν) with ν non-integer gives µ_k ~ C k^(−d−2ν+1), parity-dependent. ReLU kernels have ν = 1/2 (NTK) and 3/2 (random features) at any depth, so the deep NTK decays as k^(−d) and the deep RF kernel as k^(−d−2) (Corollaries 2–3). MNIST and Fashion-MNIST accuracy is flat in depth from L = 2 to 5 (Table 1). | Read |
| NOTE-476 v2 | Prevalence of Neural Collapse during the terminal phase of deep learning training | NC1 Σ_W → 0 (Tr(Σ_WΣ_B†)/C falls), NC2 centred means → simplex ETF (cosines → −1/(C−1)), NC3 W ∝ Ṁᵀ, NC4 nearest class mean; 7 datasets × 3 nets, 480 models. Median TPT gains: +0.35 points test accuracy, +0.0252 DeepFool robustness. NC1–2 ⇒ NC3–4 for Webb–Lowe and max-margin (Theorems 2, 4). Optimal small-noise error exponent C/(C−1)·1/4, only at simplex ETFs (Theorem 5). | Read |
| NOTE-481 | The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies | For uniform data the wide two-layer ReLU Gram matrix H∞ is a convolution on the sphere, so its eigenvectors are Fourier modes or spherical harmonics. On S¹, bias-free: λ_k = 1/π², 1/4, 2(k²+1)/(π²(k²−1)²) and 0 for k = 0, 1, even k ≥ 2 and odd k ≥ 3 (Eq. 12). With zero-initialised bias, odd k ≥ 3 get 1/(π²k²) (Eq. 13). Learning time grows as k² on S¹ and about k³ on S², as measured for shallow, deep and residual networks (Figs. 6–7). | Read |
| NOTE-482 v2 | Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 | JumpReLU SAE: f(x) = z ⊙ H(z − θ), z = W_enc x + b_enc, x̂ = W_dec f + b_dec, loss ‖x − x̂‖² + λ‖f(x)‖₀, with θ trained by straight-through estimators (bandwidth ε = 0.001). Released on all layers and sublayers of Gemma 2 2B and 9B and three layers of 27B, at widths 2¹⁴–2²⁰. Evaluated by delta LM loss and FVU against L0. Residual SAEs cost the most loss; base-model SAEs transfer to the IT model; wider SAEs split latents. No agreed SAE quality metric exists. | Read |
| NOTE-483 | Towards Understanding the Spectral Bias of Deep Learning | Theorem 4.2: n^(−1/2)‖V_(r_k)ᵀ(y − ŷ(T))‖ ≤ 2(1 − λ_(r_k))^T n^(−1/2)‖V_(r_k)ᵀy‖ + ε for any labels, given n ≥ Ω̃(ε^(−2) max{(λ_(r_k) − λ_(r_k+1))^(−2), M⁴r_k²}) and m ≥ Ω̃(poly(T, λ_(r_k)^(−1), ε^(−1))). Theorem 4.3: on uniform S^d the NTK’s eigenfunctions are spherical harmonics, µ_k = 0 for odd k ≥ 3, and µ_k = Ω(max(k^(−d−1), d^(−k+1))) for even k. Experiments in R¹⁰ learn degrees 1, 2, 4 in that order, also on three non-uniform densities. | Read |
| NOTE-512 | Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity | LLFC: c·f⁽ℓ⁾(αθ_A + (1 − α)θ_B) = αf⁽ℓ⁾(θ_A) + (1 − α)f⁽ℓ⁾(θ_B) for every layer ℓ and α, some c > 0. Observed whenever LMC holds (spawning or permutation), across MLP, VGG-16, ResNet-20/50. Lemma 1: LLFC ⇒ error ≤ 2ε on the path. Theorem 1: weak additivity of ReLU + commutativity (W_A H_A + W_B H_B = W_A H_B + W_B H_A) ⇒ LLFC with c = 1. Weight and activation matching minimise the two factors of (W_A − PW_BPᵀ)(H_A − PH_B). | Read |
| NOTE-523 | ZipIt! Merging Models from Different Tasks without Training | Merging two differently initialised models trained on disjoint tasks: permute-then-average puts the result outside both task basins. Matching correlated features within and across models (Eq. 8) and merging only the early layers lifts CIFAR-10 (5+5) joint accuracy from 46.2 (Git Re-Basin) to 79.1 fully merged and 83.8 at 13/20 layers (ensemble 87.4). Fully merged ImageNet-1k (200+200) stays at 8.6; 10/50 layers reaches 60.9 (ensemble 63.3). Theorem 1 tightens Entezari et al.’s barrier bound by (1 − 2Γ)^{1/(2d+4)} for redundancy fraction Γ. | Read |
| NOTE-525 | Does Data Scaling Lead to Visual Compositional Generalization? | (n, k) framework: two concepts with n values, k seen combinations per value. From-scratch ResNet-50s: near-100% ID, large OOD drops; 4× ID data leaves 60–80% drops; more values or combinations help. Features go from spurious, to decodable but entangled, to linearly factored (R² > 0.8, cosine < 0.1–0.2). Proposition 4.1: with linearly factored embeddings spanning 2n − 1 dimensions, k = 2 suffices for a linear classifier to be correct on all (n − k)·n unseen pairs. DINO, DINOv2, CLIP: partly factored, above chance, not perfect. | Read |
| NOTE-536 | The Generalization Ridge: Information Flow in Natural Language Generation | I(Zℓ; Y) = H(G_Z) + H(G_Y) − H(G_Z ∘ G_Y), with H(G) = −tr(G log G) on trace-normalized Gaussian Gram matrices (bandwidth 1) of ℓ2-normalized last-token states and target-token embeddings. It rises, peaks at layers 10/12 (GPT-2-S, synthetic: 0.2721) and falls (0.0188 at layer 12), where OOD early-exit accuracy peaks (56.90%) and ID accuracy keeps rising (93.17%). Middle residual updates add most; OOD-fitted residual scales downweight the last layers. The profile’s shape persists over 50 decoding steps. | Read |
| NOTE-538 | Mechanistic Mode Connectivity | Mechanistic similarity = invariance to the same unit interventions on the latents of the data-generating process. Cue-reliant and cue-invariant ResNet-18 and VGG-13 models are quadratic- but not linear-connected, even after activation matching. Conjecture 1 (no LMC up to symmetry ⇒ mechanistic dissimilarity) is not what Appendix F.3 proves. For a one-hidden-layer ReLU net with interpolating minimizers, Lemma 2 (LMC forces identical activation patterns) with Theorem 1 (mechanisms of different complexity force different patterns) proves the converse: dissimilar mechanisms of different complexity ⇒ no LMC under any permutation. The paper presents this as verifying the conjecture. CBFT (cross-entropy + barrier loss + class-mean invariance loss) removes the cue on synthetic CIFAR-10/100 and Dominoes. | Read |
| NOTE-541 | Intermediate Layer Classifiers for OOD generalization | ILC_l(x) = W_l r_l(x) + b_l on frozen layer-l features, l ≤ L − 2, layer chosen on OOD validation data. Few-shot, best ILC minus last-layer retraining: +16.8, +6.3, +1.6, +6.3 points (ResNets on CMNIST, CIFAR-10, CIFAR-100C, MultiCelebA); +27.0 on MultiCelebA at π ≤ 0.03. Zero-shot worst-group accuracy: Waterbirds 79.4 → 87.1, CelebA 56.0 → 82.0, MultiCelebA 20.8 → 46.0. Layer 7 beats 8 everywhere in Figure 8. Intermediate features are less displaced by shift for minority groups. | Read |
| NOTE-607 | Rethinking the Role of Demonstrations | Shows empirically that the correctness of in-context labels contributes little to classification and multi-choice accuracy, while the input distribution, the label space and the paired format contribute most of the gain over zero-shot. The authors read this as the demonstrations locating a correspondence the model already has, not teaching a new one. | Read |
| NOTE-620 | Learning to (Learn at Test Time) | Recasts a sequence-modelling layer as a learner trained on the sequence it reads: hidden state as inner-model weights, update as a gradient step on a learned self-supervised loss. Proves that the linear, batch-GD case is linear attention and that a Nadaraya–Watson inner learner is softmax attention; shows TTT-Linear and TTT-MLP use long context better than Mamba at up to 1.3B parameters. | Read |
| NOTE-621 | Transformers learn in-context by gradient descent | Constructs weights for which one linear self-attention layer reproduces, on every token, the data change induced by one gradient step of a linear model on the in-context regression loss, so the query’s output equals the post-update prediction; shows a trained single layer finds these weights up to scale, and that deeper trained models implement a curvature-corrected variant (GD++) rather than gradient descent. | Read |
| NOTE-626 | Fantastically Ordered Prompts | Shows that permuting a fixed set of few-shot examples swings GPT-family accuracy from near chance to near state of the art, at every model size tested, with good orders uncorrelated across sizes and the variance surviving more examples and calibration; proposes ranking orders by the label entropy they induce on a probing set the model generates itself. | Read |
| NOTE-637 | Test-Time Training for Few-Shot Learning | Shows that turning a task’s few-shot demonstrations into a temporary weight update (per-task LoRA on leave-one-out in-context tasks) improves an 8B model over conditioning on the same demonstrations: 50.5% → 57.8% on BIG-Bench Hard, and large gains on ARC. The update works best when it is trained in the in-context format, and gains concentrate on tasks with structural rules or distribution shift. | Read |
| NOTE-648 | Quantum-Like Contextuality in Large Language Models | Builds a three-sentence anaphora schema with the support of the PR prism, instantiates it 51,966,480 times from Simple English Wikipedia, and takes one referent probability per sentence from BERT: 0.148% of the models pass a signalling-corrected sheaf test and 71.1% the CbD test. Because the schema fixes the support, both verdicts depend only on how uncertain BERT is in the three sentences; the tie to embedding distance is a softmax identity with weak correlations. | Read |
| NOTE-652 | Relative representations enable zero-shot latent space communication | Represents each sample by its cosine similarities to a fixed set of anchor samples. That vector is invariant by construction to every cosine-preserving map of the latent space (rotations, reflections, global and even per-sample positive rescaling), not to translations or general linear maps. With anchors matched across encoders it lets a decoder trained on one encoder run untrained on another, across seeds, architectures, latent sizes and languages: far better than raw latents, but with large and uneven losses. The assumption that such spaces differ by an angle-preserving map is never tested directly, and the evidence supports it only approximately. | Read |
| NOTE-653 | Revisiting Model Stitching to Compare Neural Representations | An empirical paper with no theorems. It defines the stitching penalty, the change in a top network’s test error when its bottom ℓ layers are replaced by another representation through a trained 1×1 convolution, and uses it to show, in single runs mostly on CIFAR-10, that networks from different seeds, and supervised and self-supervised ImageNet networks, are interchangeable layer by layer where CKA calls them different, and that a bottom trained on more data improves a weaker top. The ordering it gives is by one task’s loss through a fixed head and a restricted map class, which is not an ordering by information. | Read |
| NOTE-658 v2 | Exact Learning Dynamics of In-Context Learning in Linear Transformers | Solves the averaged gradient-flow dynamics of one linear-attention layer trained on in-context regression, when tasks and input covariance share an eigenbasis and the weights are assumed to stay in it: each mode is a logistic curve on timescale (ηP s_α²)⁻¹, so learning is staged by the input spectrum, and the fixed point is the known preconditioned predictor W Σ̂_x [((N+1)/N)Σ_x + (Tr Σ_x/N)I]⁻¹. A scaling symmetry conserves ‖p₂‖² − ‖q₁‖². The transfer to non-linear transformers and grokking is a set of timing coincidences in spectral measures. | Read |
| NOTE-659 | Small Singular Values Matter | Uses the Marchenko–Pastur law as a null model for trained transformer weights and finds departures at both ends of the spectrum, the lower end only in rectangular matrices, whose null has a positive lower edge. In the MLP up and gate projections the below-bulk directions align with the input activations’ principal directions, and in rectangular matrices the smallest decile is never the least important to remove. A Gaussian teacher–student ensemble shows one way it can happen: the loss suppresses noise along a learned direction, a negative spike that holds a weak signal below the bulk. Overlap and ablation importance disagree for several matrix types, so “departs from the null” and “carries information” are not shown to coincide in general. | Read |
| NOTE-660 | Closed-Form Training Dynamics in Word2Vec-like Models | Proves that word2vec’s loss, approximated to fourth order and reweighted so that every pair carries the same aggregate weight, is unweighted factorisation of M*, the relative deviation of co-occurrence from independence, whose rank-d minimisers are its top eigenvectors; derives, without full rigour, that gradient flow from small random initialisation learns those eigenvectors one at a time in sigmoidal steps; and shows on Wikipedia that the predicted embeddings match word2vec’s features and scores better than truncated PMI or PPMI. Its account of analogies is an empirical spiked-random-matrix fit. | Read |
| NOTE-662 | Towards Understanding Grokking | In a model that decodes the sum of two learned embeddings, a held-out pair is answered correctly when its embedding sum coincides with a trained pair’s, and the training set’s same-answer pairs force such coincidences. Grokking’s critical training fraction is then where those equations first fix the embedding up to translation and scale, which a posited effective loss predicts for addition with p = 10. Threshold-defined phase diagrams over decoder learning rate and weight decay put grokking between comprehension and memorization, in the toy and, qualitatively, in a transformer and on MNIST. | Read |
| NOTE-664 | Symmetry in Language Statistics and Representation Geometry | Proves that if co-occurrence among words on a latent continuum depends only on their separation, a full-rank spectral word embedding places them on Fourier modes of the latent coordinate, with amplitudes from the kernel’s Fourier transform, and solves the exponential kernel exactly for periodic and open boundaries. The symmetry is measured in Wikipedia, the geometry seen in Gemma 2 by PCA, and its robustness to deleting the words’ mutual co-occurrences explained by a latent variable shared by many words. | Read |
| NOTE-665 | On the Emergence of Linear Analogies in Word Embeddings | Derives the parallelogram analogies of word embeddings from a generative model of co-occurrence in which words are bundles of independent binary attributes: the co-occurrence ratio is then a Kronecker product, and its logarithm, the PMI, is a bilinear form of rank at most d + 1 whose eigenvectors are affine in the attributes, so a spectral PMI embedding is a linear image of the attribute hypercube. Robustness to noise, pruning and deletion of a relation’s own pairs is argued from eigenvalue bounds and simulated; Wikipedia agrees in shape. Every embedding is an eigendecomposition, none is trained. | Read |
| NOTE-666 | A mathematical theory of semantic development in deep neural networks | Shows that a two-layer linear network trained from small weights learns the singular modes of its input–output correlations in order of strength, each by a sharp sigmoidal transition, where a shallow network learns all modes together; and that the singular structure of data generated on trees, rings and clusters, plus a minimum-norm property, accounts for coarse-to-fine differentiation, transient errors, narrowing generalisation, typicality, coherence thresholds and conserved representational similarity. Exact only from decoupled, balanced initial weights with white inputs; random initialisation is simulated. | Read |
| NOTE-667 | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks | Derives the coupled gradient-descent equations of a linear network with hidden layers and whitened inputs, and solves them exactly from decoupled starts: one sigmoid per singular mode of the input–output correlation, with conserved differences of squared layer strengths; balanced or not with one hidden layer, from equal layer strengths at any depth. Depth slows learning only through the initial end-to-end mode strength, so from strength of order one the iteration count stays finite as depth grows when the step size is scaled down with depth. Argues, with MNIST and random-matrix simulations, that orthogonal layers or suitable pretraining give such starts and scaled Gaussian layers do not, and that orthogonal tanh networks near gain 1 keep their Jacobians near-isometric. | Read |
| NOTE-669 | Hyperbolic Representation Learning: Revisiting and Advancing | Shows, by tracking each node’s hyperbolic distance to the origin, that hyperbolic graph models trained on task losses do not by themselves place a tree’s root nearest the origin or spread its levels outward: on synthetic 3-ary trees the root sits well off the minimum and level order is recovered for only 69–75% of node pairs. A centroid recentring plus an outward-push loss (HIE), using no level information, raises this to 75–84% and improves link prediction and node classification, most on the least tree-like graphs. | Read |
| NOTE-671 | Hyperbolic Entailment Cones for Learning Hierarchical Embeddings | Nested, axially symmetric, rotation-invariant cones in the Poincaré ball define a partial order, but nesting forces apertures of at most π/2 and r sin ψ(r)/(1 − r²) non-increasing. So no cone exists near the origin, and the widest family is sin ψ = K(1 − r²)/r, with a closed-form membership angle. The transitivity of that family is asserted, not proved. Learned on WordNet noun hypernymy, the cones predict held-out closure edges better than Poincaré and order embeddings once some closure edges are in training. Trained on the transitive reduction alone, every method fails. | Read |
| NOTE-674 | Mutual information, neural networks and the renormalization group | Makes “the relevant degrees of freedom” of real-space RG an optimizable quantity: the coarse variable of a block that carries the most mutual information about the system beyond a buffer. Learned from Monte Carlo samples, it recovers Kadanoff block spins for 2D Ising and the electric-field variables of the dimer model, discards decoupled noise, and, iterated, gives the Ising flow and ν ≈ 1.0 ± 0.15, while a distribution-fitting RBM keeps the noise. The equivalence with a short-ranged effective Hamiltonian is shown only for 1D and quasi-1D. | Read |
| NOTE-675 | Hyperbolic Diffusion Embedding and Distance | Embeds each point’s diffusion densities at dyadic time scales in a product of Poincaré half-spaces and proves that the ℓ1 product distance, a scale-weighted Hellinger sum, is equivalent up to constants to a thresholded snowflake min{1, d_T^(2α)} of a latent tree metric when the diffusion operator behaves like the heat kernel on that tree. The tree heat-kernel bounds are cited rather than proved and one Hölder step holds only for d_T ≥ 1. Empirically it keeps tree neighbourhoods (best MAP) and not global distances (more distortion than hyperbolic MDS). | Read |
| NOTE-677 | The Random Hierarchy Model | Introduces the Random Hierarchy Model, a random context-free grammar in which each symbol has m synonymic expansions, and shows by experiment that deep feature-learning networks learn it from about n_c m^L examples, polynomial in the input dimension, while shallow and lazy networks need an exponential number. That size coincides with the emergence of synonym-invariant hidden representations and, by a scaling argument, with the size at which the class statistics of input patches, equal for synonyms, exceed sampling noise; removing those statistics makes deep networks fail as well. | Read |
| NOTE-679 v2 | Representation Tradeoffs for Hyperbolic Embeddings | Trees embed in the hyperbolic plane with worst-case distortion 1 + ε, but because a hyperbolic distance d takes about d bits to write, the coordinates need Θ((ℓ/ε) log deg_max) bits in the plane: cheap in branching, expensive in depth. More dimensions divide the upper bound by r for bushy trees. For general metrics, an exact hyperbolic MDS by PCA on cosh of the distances after centring at a new mean, with a perturbation bound growing as sinh² of the diameter; hyperbolic PGA has spurious local minima and only a local-convexity guarantee. | Read |
| NOTE-681 v2 | Statistical Physics through the Lens of Real-Space Mutual Information | Makes the real-space mutual-information principle a single-shot operator finder: a CNN coarse-graining co-trained against a neural InfoNCE bound, optimized once at a large buffer, returns on the interacting dimer model the order parameters below the BKT transition and the electric fields above it, and one fitted scaling dimension (1.00037 against 1). The maximal information is log 4 in the ordered phase and decays algebraically with the buffer in the critical one. The model’s operators were known; the optimality theorem is cited. | Read |
| NOTE-682 | Optimal Renormalization Group Transformation from Information Theory | Proves that a block coarse-graining keeping all of the block’s mutual information with the system beyond a buffer makes the coarse measure factorize across the block, so a finite-range Hamiltonian gains no range (1D; D dimensions with an extra assumption) and 1D disorder gains no correlations across it. Where the condition fails, as it does for its own examples, it shows on two-spin blocks of 1D Ising chains that longer-range, many-spin and disorder-correlation terms shrink as more information is kept. Explains decimation in 1D and majority rule in 2D by the cost of encoding into one binary spin. | Read |
| NOTE-683 v2 | An exact mapping between the Variational Renormalization Group and Deep Learning | Identifies Kadanoff’s variational RG kernel with an RBM energy by T = −E + H, so that the coarse-grained Hamiltonian is the RBM’s hidden-marginal Hamiltonian for any parameters and an exact RG step is a perfect fit of the data. The identity is correct but definitional, and the paper concedes the approximate schemes differ; its evidence that trained networks do RG is qualitative. LIT-873 is right that fitting the distribution need not give RG, though it does not engage the identity. | Read |
| NOTE-684 | The Sparse Random Hierarchy Model | Adds sparsity to the Random Hierarchy Model, so that labels are unchanged by discrete displacements of the informative features, and shows by experiment that deep networks learn it from a number of examples polynomial in the input dimension, (s0 + 1)^L n_c m^L up to a factor s^(L/2) without weight sharing and (s0 + 1)^2 n_c m^L with it, and that hidden layers become insensitive to synonym swaps and to feature displacements at about the same training-set size as test error falls. That coincidence is offered as the reason deformation stability tracks performance on images, which is not tested on images. | Read |
| NOTE-686 | Hyperbolic Neural Networks | Möbius gyrovector operations give the Poincaré ball closed-form exponential and logarithmic maps, parallel transport from the origin as a scaling, and geodesic hyperplanes with a closed-form distance; from them come hyperbolic logistic regression, feed-forward layers and GRUs that reduce to the Euclidean ones as c → 0. Because linear maps and nonlinearities are lifted through the origin’s tangent space, the curvature acts only through biases and output distances. Gains over Euclidean models are confined to a synthetic tree-like task and to separating WordNet subtrees in low-dimensional Poincaré embeddings. | Read |
| NOTE-687 | How the structure of language is acquired | Extends the Random Hierarchy Model to predicting a masked last token. The correlation between two tokens shrinks by about a factor m per level of their common ancestor, a sample of P sequences resolves it only above a (v²P)^(−1/2) noise floor, and deep transformers and CNNs show loss steps at P_ℓ ≈ v m^(2ℓ−1), where their layers come to encode the level-ℓ hidden symbol. On character-level text, the correlation decay exponent predicts where the loss for a given context length levels off. The route from correlations to learned hidden variables is argued, not proved. | Read |
| NOTE-688 v2 | Why does deep and cheap learning work so well? | Argues that cheap networks suffice because physical data have low-order, local, symmetric Hamiltonians, and that depth pays because inference must compose the sufficient statistics of a Markov generative hierarchy; proves that one hidden layer needs exactly 2^n neurons to multiply n inputs. On renormalization it is right that a coarse-graining needs a stated target, and its early appendix shows that neither a matched partition function nor the exact trace condition supplies one; it is wrong that the target must come from supervision. | Read |
| NOTE-690 | Relevance in the Renormalization Group and in Information Theory | Proves, for a short-range lattice model on a cylinder with block and environment as slabs separated by a buffer, that the information- bottleneck encoder at its first transition depends on the block only through the boundary weak value of the leading transfer-matrix eigenvector, which at large circumference is the lowest-dimension primary, with β_c = (λ0/λ1)^(2L_B); checked on a three-site Ising cylinder. It does not cover the planar block-in-a-shell geometry or the fixed-alphabet β → ∞ limit that RSMI uses, which LIT-878 relies on. | Read |
| NOTE-692 v2 | Symmetries and phase diagrams with real-space mutual information neural estimation | The long companion of LIT-878. Derives the RSMI-NE estimator and shows that the maximal real-space information counts ordered sectors (1 bit for the Ising ferromagnet, 2 for columnar dimers), decays exponentially with the buffer off criticality and algebraically at it, and that the ensemble of optimal filters from independent runs maps a degenerate subspace whose projections show broken lattice symmetries and the dimers’ emergent U(1). All the models’ answers were known; the non-equilibrium example is one figure. | Read |
| NOTE-694 | Comment on “Why does deep and cheap learning work so well?” | Schwab and Mehta concede that a preserved free energy does not give a preserved distribution and that the “iff” of LIT-882’s Eq. 8 should be one-directional, and show that Lin and Tegmark’s v1 counterexample violates the trace condition, so the mapping’s exact step stands. Right against v1; it places all of the mapping’s content at the trace condition, and says nothing to the later point that the trace condition does not select the coarse variables. | Read |
| NOTE-715 | From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks | Two-layer linear network ŷ = W₂W₁x, MSE, full-batch gradient flow. Assumptions: whitened inputs (A1), λ-balanced start W₂ᵀW₂ − W₁W₁ᵀ = λI (A2), no bottleneck (A3). Then QQᵀ, with Q = [W₁ᵀ; W₂], obeys a matrix Riccati equation (Lemma 4.1), solved exactly (Theorem 4.3). For aligned starts, each singular value moves by a λ-dependent transition function: sigmoidal as λ → 0, exponential 1 − e^{−|λ|t/τ} as |λ| → ∞ (Theorem 5.1). | Read |
| NOTE-738 | Nielsen et al. — When Does Closeness in Distribution Imply Representational Similarity? | For softmax models p(y|x) ∝ exp(f(x)ᵀg(y)), equal distributions fix the embeddings up to an invertible linear map, but KL closeness does not: by making the unembedding norm large, two models’ KL divergence goes to zero while their embeddings stay fixed and are not linearly related. A log-likelihood variance distance does bound a representational dissimilarity, d_LLV ≤ ε ⇒ d_f,g ≤ 2Mε. Whether closeness in distribution implies similar representations depends on which distance is used. | Read |
| NOTE-739 v2 | Poole et al. — On Variational Bounds of Mutual Information | The common variational lower bounds on mutual information (Barber–Agakov, Donsker–Varadhan, NWJ, MINE, InfoNCE) are derived from one unnormalized Barber–Agakov bound. InfoNCE is the multi-sample NWJ bound with the batch mean as its partition estimate, which proves it a lower bound for every critic when the K pairs are i.i.d.; it is capped at log K. An interpolated family I_α trades InfoNCE’s bias against NWJ’s variance, and none of them gives low bias and low variance when MI is large and the batch small. | Read |
| NOTE-740 v2 | HaoChen et al. — Spectral Contrastive Loss | On a finite augmentation space, the spectral contrastive loss L(f) = −2E[f(x)ᵀf(x⁺)] + E[(f(x)ᵀf(x⁻))²] equals the rank-k factorization loss ‖D^(−1/2)AD^(−1/2) − FFᵀ‖²_F up to a constant when the rows of F are w_x^(1/2) f(x), so its minimizers are the top-k eigenvectors of the normalized augmentation graph up to row scaling and a right rotation. A linear probe on them has error Õ(α/ρ²_⌊k/2⌋), where α is the label noise of augmentations and ρ the sparsest-partition conductance. | Read |
| NOTE-741 | Simon et al. — On the Stepwise Nature of Self-Supervised Learning | For a simplified Barlow Twins loss on a linear or kernel model trained by gradient flow from small initialization, the embedding learns the top eigenvectors of a symmetrized cross-correlation operator one at a time, each at a time set by its eigenvalue, and lands on the minimum-norm zero-loss solution: kernel PCA on K_Γ. Exact for aligned initialization; an informal derivation for generic small initialization. ResNets trained with Barlow Twins, SimCLR and VICReg show the same stepwise rank growth, which is an empirical finding, not a consequence of the theory. | Read |
| NOTE-742 | Sefidgaran, Zaidi & Krasnowski — MDL and Generalization Guarantees for Representation Learning | The expected generalisation error of an encoder–decoder classifier is bounded by 2√((2·MDL + K + 2)/n) + ε, where MDL is a KL divergence between the encoder’s joint law of train-and-ghost latents and a prior symmetric under label-preserving permutations. The bound holds for any decoder, stays finite for deterministic continuous encoders under lossy compression, and is never evaluated numerically. A VIB variant with label-dependent learned priors gains about 2.5 points on CIFAR-10. | Read |
| NOTE-743 v2 | Kornblith et al. — Similarity of Neural Network Representations Revisited | Any similarity index invariant to invertible linear maps is constant once a representation’s rank reaches the number of examples (Theorem 1, a four-line proof). Kornblith et al. argue, without a theorem, for invariance to orthogonal maps and isotropic scaling only. They propose CKA, normalised HSIC of centred Gram matrices, and show that R²_CCA and linear regression are CKA of re-weighted Gram matrices. Linear CKA alone finds corresponding layers across CNN initialisations (99.3%, against at most 45.4% for the other linear indexes). | Read |
| NOTE-746 | Marconato et al. — All or None: identifiable linear properties of next-token predictors | Two next-token predictors of the form p(y | x) ∝ exp(f(x)ᵀg(y)) induce the same distribution exactly when they are related by an invertible linear map on the subspaces M and N that enter the logit dot product, with arbitrary distortion allowed off them. Relational linearity, and with it linear probing and linear subspaces, then holds in all such models or none when Γ ⊆ N and Γ_q ⊆ M. Parallelism survives only inside N. There are no experiments. | Read |
| NOTE-749 | Tschannen et al. — On Mutual Information Maximization for Representation Learning | Four controlled experiments on MNIST image halves (two repeated on CIFAR10) show that downstream linear accuracy of InfoNCE- and NWJ-trained encoders does not track mutual information. Invertible encoders improve while true MI is constant. Higher NWJ bounds from an MLP critic give worse features. At matched bound values, ConvNets beat MLPs. The paper also proves that InfoNCE is neither a lower nor an upper bound on MI when batch pairs are dependent, and recasts it as a multi-class K-pair metric learning loss. | Read |
| NOTE-751 v2 | Johnson, El Hanchi & Maddison — Contrastive Learning Can Find an Optimal Basis | On a finite augmentation space, the NT-Logistic and spectral losses are minimized over unconstrained kernels by K⁺ = p⁺/(p⊗p), and InfoNCE by K⁺ up to a factor constant on communicating classes. Kernel PCA under K⁺ returns the eigenfunctions of the positive-pair Markov chain (h_i = λ_i^(1/2) f_i). The top-d eigenfunctions span the subspace that minimizes worst-case L²(p) approximation error over targets with E_p⁺[(g(a₁) − g(a₂))²] ≤ ε; that error is ε/(2(1 − λ_(d+1))). | Read |
| NOTE-752 v2 | Balestriero & LeCun — SSL Recovers Global and Local Spectral Embedding Methods | On a finite sample with a positive-pair graph G, idealized forms of VICReg (squared variance term), SimCLR (a Euclidean graph-matching variant) and linear Barlow Twins have optimal representations or weights built from the spectrum of G. They are Laplacian eigenmaps, a generalized MDS and CCA. All of them align the left singular vectors of Z with those of G. SimCLR and Barlow Twins force rank(Z) = rank(G), while VICReg can stay full rank. Several proofs are sketches, and the experiments are small and synthetic. | Read |
| NOTE-753 v2 | Harvey, Larsen & Williams — Duality of Bures and Shape Distances | For centred linear kernels K_X = CXXᵀC, the Bures distance equals the Procrustes distance, B(K_X, K_Y) = P(X, Y), and normalized Bures similarity equals the cosine of the Riemannian shape distance. Both follow from F(K_X, K_Y) = ‖XᵀCY‖_*. By Uhlmann’s theorem, the fidelity is the largest overlap between representations consistent with the two kernels. CKA is only loosely tied to NBS, inside a rank-dependent envelope. The result is finite-dimensional and linear; the infinite limits are heuristic. | Read |
| NOTE-754 | Fischer — The Conditional Entropy Bottleneck | The MNI criterion asks for I(X;Y) = I(X;Z) = I(Y;Z), and CEB targets it by minimizing I(X;Z|Y) − γI(Y;Z), which is IB with γ = β − 1. CEB beats VIB at every ρ on Fashion MNIST for accuracy and robustness, and a CEB Wide ResNet beats Madry’s adversarially trained model on CIFAR-10 without adversarial training. That retained information causes the robustness failures is a hypothesis, supported only by these experiments. | Read |
| NOTE-755 | Picozzi — The information geometry of large language models is shared, learned, and controllable | The Fisher–Rao distance matrix of a model’s next-token laws over a context battery is a function of the laws alone, so behaviour fixes it, whereas activation geometry has a GL(d) gauge. Across ten models of three architecture kinds it agrees at a mean rank correlation of 0.88, against 0.62 for mid-layer activations. The agreement is proved to follow from each model’s closeness to the same language law, and only about 9% of shared eigenvector structure goes beyond what the shared laws imply. The pullback metric also gives minimum-disturbance steering, whose cost advantage over Euclidean steering it predicts. | Read |
| NOTE-758 | Shai et al. — Transformers learn factored representations | If a generator’s token-labelled operators are tensor products, the predictive state stays a product state, and a linear map carries it losslessly into orthogonal per-factor subspaces of total dimension Σ(d_n − 1). Small transformers trained on five such factors settle near that dimension, about 10 against 242, in approximately orthogonal subspaces. With token noise that makes factoring lossy they collapse to it first and expand only later. RNNs and LSTMs show it less sharply. | Read |
| NOTE-759 | Bird — Emergence of Quantised Representations Isolated to Anisotropic Functions | In small CIFAR and MNIST autoencoders built from orthogonally equivariant parts except the activation, elementwise tanh and Leaky-ReLU drive latent representations into narrow beams along or against the standard basis, arranged by each activation’s symmetry group. Radial (O(n)-equivariant) versions of the same functions leave them spread. The beams appear even in the first latent layer, which no activation precedes. The evidence is visual, five seeds a condition, with no statistic. | Read |
| NOTE-762 | Kornblith, Chen, Lee & Norouzi — Why Do Better Loss Functions Lead to Less Transferable Features? | For ResNet-50 on ImageNet, every training objective that raises accuracy over softmax cross-entropy also raises penultimate-layer class separation, and fixed penultimate features with more class separation transfer worse to new-class datasets (Spearman ρ = −0.93 across nine objectives; one temperature knob moves both). The losses differ only in the last few blocks, and full fine-tuning erases the difference. For the pretraining classes themselves, more separation helps. | Read |
| NOTE-764 | Thrampoulidis et al. — Imbalance Trouble | In the unconstrained-features model with cross-entropy and vanishing regularization, the max-margin solutions have logits exactly equal to the simplex-encoded label matrix Ẑ (one-hot minus 1/k), whatever the class imbalance. The classifier and embedding Gram matrices are VΛVᵀ and UΛUᵀ from Ẑ’s SVD. This SELI geometry is the simplex ETF only for balanced or binary data. Under imbalance, majority classifiers are longer and minority classifiers closer together. No finite weight decay gives SELI, and SGD approaches it more slowly as imbalance grows. | Read |
| NOTE-765 | Huang et al. — internal causal mechanisms predict out-of-distribution behaviour | In Llama-3-8B-Instruct, on five tasks and ten task-preserving shifts, checking by interchange intervention whether a test input still realizes the causal variable located on in-distribution data predicts whether the output is correct better than output confidence or probes of other internal features, best on 8 of 10 shifts. A probe on the located variable beats a probe on its orthogonal complement on 7 of 10. Interchange accuracy and predictive power rise together only up to a point. | Read |
| NOTE-770 v2 | Zhai — Contextures: The Mechanism of Representation Learning | For any joint law of an input X and a context A, symmetric or not, the conditional-expectation operator from L²(P_A) to L²(P_X) has a singular value decomposition (assuming Hilbert–Schmidt), and its top-d left singular functions are a d-dimensional encoder (not the unique one, as the thesis says; see version 2) that minimizes worst-case linear-probe approximation error over tasks of compatibility at least 1−ε. Several squared-error objectives reach that span at their optimum. The claims about scaling and about moderate association are argued and measured, not proved. | Read |
| NOTE-771 | Kumar, Clune, Lehman and Stanley — The Fractured Entangled Representation Hypothesis | For three images found by Picbreeder’s human-guided evolution (skull, butterfly, apple), SGD in the same layerized architecture reproduces the image from random weights, but its neurons show no reuse of the symmetric parts, and sweeping its weights distorts the image where sweeping the evolved network’s weights varies it meaningfully. This holds in a PCA basis, along random weight directions, and in ReLU networks. The evidence is qualitative and curated; the attribution to open-ended search is conjecture. | Read |
| NOTE-772 v2 | Kothapalli — Neural Collapse: A Review on Modelling Principles and Generalization | A survey, not a source of results. It sorts accounts of neural collapse into unconstrained-features models (the last-layer features are free variables) and local elasticity, and reports that every one of them derives the collapsed state or its dynamics only after removing the data. On generalization it finds the literature in conflict. Collapse on the training set does not guarantee good test error, test-set collapse is much weaker and dataset-dependent, and there is no agreed threshold for calling a network collapsed. It also reports that networks more collapsed on ImageNet transfer worse. | Skimmed |
| NOTE-774 | Kothapalli & Tirer — Can Kernel Methods Explain How the Data Affects Neural Collapse? | NC1, measured as tr Σ_W / tr Σ_B, is written for any kernel as a function of the kernel matrix. For two-class, one-dimensional Gaussian data with the classes on opposite sides of the origin, the ReLU NTK of a wide two-layer net is a scaled copy of the NNGP kernel within each class and zero across classes. So lazy-regime training leaves expected NC1 where it was at initialization. In higher dimensions the NTK is measured no more collapsed, and with ReLU less. A data-aware kernel lowers NC1 for separable data but tracks finite networks poorly when the classes overlap or are imbalanced. Erf collapses more than ReLU. | Read |
| NOTE-777 | Belrose et al. — Neural Networks Learn Statistics of Increasing Complexity | An empirical paper. Vision networks trained on real images classify maximum-entropy samples that match only the class means, or means and covariances, well above chance early in training (peaks between about 2^4 and 2^12 steps), and the accuracy then falls, often below chance. First-order samples peak earlier and lower than second-order ones. Pythia models fit unigram-sampled text, then bigram-sampled text, then fit them worse, then better again. The n-gram/moment equivalence is an identity of one-hot indicators. The ordering argument is a heuristic. | Read |
| NOTE-781 | Torii, Maeda and Hidaka — Distributional hypothesis as isomorphism | An SVD of log raw co-occurrence counts solves 61% of Google analogies over a 1,487-word vocabulary, against 75% for full word2vec. In a 24-sentence toy corpus whose eight subjects carry every combination of three binary aspects, the subjects’ raw co-occurrence rows form a parallelepiped exactly when, for each object, its four sentences are equally probable. The attributes are independent, so the concept lattice is Boolean. | Read |
| NOTE-782 v3 | Makelov, Lange and Nanda — an interpretability illusion for subspace activation patching | A subspace activation patch can make a model behave as if a feature had changed while the patched direction is the sum of a component that varies with the feature but cannot reach the output (in an MLP, the kernel of the down-projection) and a component that can reach the output but is dormant on the data. Shown exactly in a three-unit network, empirically for MLP8 of GPT-2 Small on IOI and for MLPs of GPT2-XL on factual recall, where the effect vanishes once the disconnected component is removed. A residual stream direction passes the same test. One-dimensional fact patches and rank-1 edits are shown to be nearly interchangeable. | Read |
| NOTE-783 | Han, Papyan & Donoho — Neural Collapse Under MSE Loss | Networks trained with MSE loss show neural collapse on five datasets and three architectures. The MSE loss splits exactly into a term for the least-squares classifier of the current features and a non-negative deviation term, and the deviation becomes negligible early in training. On that “central path”, in an unconstrained-features model with no weight decay and features continually renormalized to identity within-class covariance, gradient flow drives every singular value of Σ_W^(−1/2)M̃ to infinity as (4t/N)^(1/4), and drives them to equality. That gives NC1–NC4 in closed form. | Read |
| NOTE-784 v2 | Coifman et al. — Diffusion maps | From a symmetric, positivity-preserving, positive semi-definite kernel, normalized into a Markov operator conjugate to a symmetric one, the top eigenfunctions scaled by powers of their eigenvalues embed the data so that Euclidean distance equals diffusion distance. A one-parameter normalization separates geometry from sampling density, recovering the Laplace–Beltrami operator at α = 1. The symmetry of the kernel is what yields the spectral decomposition; non-symmetric kernels are not treated. | Superseded — by NOTE-1441 |
| NOTE-786 | Merullo et al. — Linear Representations and Pretraining Data Frequency | For 24 factual relations in OLMo-1B, OLMo-7B and GPT-J, how well an affine map from subject to object representation reproduces the model (LRE causality) correlates with the mean subject–object co-occurrence count in pretraining (r = 0.82). It correlates less with subject or object counts alone. Mean causality passes 0.9 beyond about 4.4k, 2.0k and 1.1k co-occurrences respectively, at any training step. Correlational; LREs are relation maps, not attribute directions. | Read |
| NOTE-789 | Kirsanov et al. — The Geometry of Prompting | In base Llama-3.1-8B (and Gemma-2-2B), demonstrations reorganise the geometry of the input sentence’s representation at early-to-middle layers, while an instruction acts mainly on the last token and soft prompts only on late layers: matched accuracy, different mechanisms. Few-shot failures from the choice of examples or from relabelled categories leave the class manifolds separable and fail at the readout. | Read |
| NOTE-800 | Noroozizadeh et al. — Deep sequence models tend to memorize geometrically | Transformers and Mamba trained only on a graph’s edge bigrams embed nodes so that multi-hop relations are readable, and solve an in-weights path-finding task built to defeat next-token learning. The authors argue that supervision, capacity, ease of optimization and succinctness each fail to explain why this geometry wins over an associative lookup. In a wide two-layer weight-tied model under unregularized cross-entropy the embeddings settle into the top Laplacian eigenvectors without a rank bottleneck, shown empirically, with an informal partial proof. Why deeper models prefer geometry is left open. | Read |
| NOTE-802 | Modell, Rubin-Delanchy and Whiteley — The Origins of Representation Manifolds in Large Language Models | If a feature is a compact metric space mapped continuously and injectively onto the unit sphere (Hypothesis 1), its representation is homeomorphic to it. If, in addition, cosine similarity is locally a C² decreasing function g of squared feature distance (Hypothesis 2), every path on the representation manifold has √(−2g′(0)) times the length of the feature path it images (Theorem 1). The hypotheses are assumed, not derived; the tests supply a candidate metric and find homeomorphism for GPT-2’s years but isometry only after a log rescaling. | Read |
| NOTE-803 | Davidson et al. — Do different prompting methods yield a common task representation? | Function vectors extracted from short instructions raise zero-shot accuracy, but they are built from mostly different attention heads than demonstration function vectors (4 to 7 of the top 20 shared), the two add rather than substitute, and swapping heads and activations between them costs accuracy asymmetrically. Under this method there is no common task representation across the two presentations, only partly overlapping mechanisms; the authors note they cannot prove the negative. | Read |
| NOTE-809 v2 | Wu et al. — a reply to Makelov et al.’s interpretability illusion arguments | The reply argues that Makelov et al.’s “illusion” is defined so that any direction not orthogonal to the null space of the downstream weights can qualify, including the direction Makelov et al. call faithful in their toy network, and that their in-the-wild cases rest on a weak metric and on DAS trained per fact pair. The toy computation holds only for an unnormalized patch: patching the subspace the rowspace component spans, as Makelov et al. specify, recovers the full effect. Its own GPT-2 experiments agree that MLP8 carries almost no name-position information. | Read |
| NOTE-816 | Erogullari et al. — Post-Hoc Concept Disentanglement | Concept activation vectors fitted one concept at a time align for CelebA and FunnyBirds attributes that co-occur in the labels. A joint fit with a penalty ‖CCᵀ − I‖²_F finds mutually orthogonal directions that still separate each concept, with AUROC “largely preserved”. So correlated attributes keep linear separating directions in these CNNs, and the alignment comes from the separate fits. Steering with the orthogonal directions changes correlated concepts less. | Read |
| NOTE-820 | Chowdhury et al. — Fundamental Limits of Perfect Concept Erasure | If an erased representation Z is independent of the concept A, it keeps at most I(Z;X) ≤ H(X|A). A deterministic erasure reaches that only when the groups’ distributions are permutations of one another. The paper says the bound is out of reach otherwise. That holds only for deterministic maps: a stochastic quantile coupling reaches it for any finite distributions with disjoint supports. | Read |
| NOTE-824 | Pfau et al. — Spectral Inference Networks | For a symmetric kernel k on a space with a sampling density p, the span of the top N eigenfunctions of Kf = E_x′[k(x,x′)f(x′)] maximizes Tr(Σ⁻¹Π) over functions, with Σ = E[uuᵀ] and Π = E[k(x,x′)u(x)u(x′)ᵀ]; a Cholesky-masked gradient orders them and moving averages of Σ and its Jacobian remove the minibatch bias. Slow Feature Analysis is the graph-Laplacian case. The ordering argument is informal by the authors’ own statement, and the paper never considers contrastive kernels or positive pairs. | Read |
| NOTE-828 v2 | Belghazi et al. — Mutual Information Neural Estimation | Mutual information is the KL divergence of the joint from the product of marginals, and its Donsker–Varadhan dual is tight exactly at the log Radon–Nikodym derivative plus a constant. Maximizing that dual over a compact family of neural critics gives an estimator that is strongly consistent, for the exact empirical maximizer, on a compact domain with Lebesgue densities, with a uniform deviation bound of order d log d/ε² in the parameter count d. The mini-batch gradient is biased; a moving average is offered as a heuristic fix without proof. | Read |
| NOTE-830 | Li et al. — Self-Supervised Learning with Kernel Dependence Maximization | With image identity as a one-hot label, HSIC(Z,Y) is (Δl/N) times the gap between the mean kernel on positive pairs and on all pairs, and (1/2N²)Σ MMD²(i,j) = (N/Δl)·HSIC(Z,Y). The large-batch InfoNCE loss is exactly −HSIC(Z,Y) plus a log-mean-exp of the centred critic (eq. 28), and for a bounded kernel it bounds −HSIC(Z,Y) + γHSIC(Z,Z) up to a remainder of second order in the critic’s variance (Thm B.1, whose printed remainder has the wrong sign). | Read |
| NOTE-833 | Sun, Shi, Liang & Li — when known classes help discover novel ones | In a spectral-contrastive augmentation graph over labelled and unlabelled data, the linear-probe residual of a novel labelling is exactly the part of it outside the top-k eigenspace that the labelled rows of the discarded eigenvectors cannot absorb. Known classes that share the novel class’s attribute can make it zero, and a toy shows that known classes sharing a rival attribute make it worse. The general error bound (Theorem 5.6) is approximate and averages all known classes into one block. | Read |
| NOTE-834 | Harvey, Lipshutz & Williams — What Representational Similarity Measures Imply about Decodable Information | For a linear readout that maximizes zᵀXw/M − ½wᵀG(X)w, the decoded signal is K_X z with K_X = XG(X)⁻¹Xᵀ/M, so the task-averaged agreement of two representations is Tr(K_X K_z K_Y). With K_z = I that is linear CKA, mean squared CCA, ENSD or the GULP distance, for particular G. Procrustes distance and average decoding distance bound each other only through the participation ratio of K_X − K_Y. The population limit (App. A.3) is an outline that requires b > 0 and a trace-class task covariance, and it does not reach the K_z = I case. | Read |
| NOTE-841 | Tosh, Krishnamurthy & Hsu — contrastive learning, multi-view redundancy and linear models | When two views X and Z are redundant for a label, the predictor that infers Z from X and then predicts, µ(x) = E[E[Y|Z]|X=x], is within εX + 2√(εXεZ) + εZ of the Bayes predictor. µ is linear in the density ratio g*(x,·) = p(x,z)/(p(x)p(z)), the optimum of the logistic contrastive problem, so m random landmark evaluations of g* support a linear predictor whose extra error falls as 1/m, scaled by a variance that is at most E[g*²] for bounded labels. Nothing requires the views to be exchangeable. | Read |
| NOTE-843 | Gebhart, Hansen and Schrater — Knowledge Sheaves | A knowledge-graph embedding is a knowledge sheaf on the schema (one pair of restriction maps per relation type) with a 0-cochain of its pullback to the graph. The summed Structured Embedding score is the sheaf Laplacian form xᵀLx; adding a relation vector gives TransR, and identity maps TransE. Since the zero cochain is always a section, training needs negative triples and a margin. A multi-hop query is scored by harmonic extension, a Schur complement of the query graph’s Laplacian, which beats summing TransE vectors on ip and pi queries. | Read |
| NOTE-846 | Tan et al. — contrastive learning is spectral clustering on a similarity graph | SimCLR’s InfoNCE loss, taken over all n objects with a Gaussian kernel on the unit sphere and a symmetric augmentation matrix π, is rewritten exactly as tr(Zᵀ L(π) Z) + log R(Z): a graph-Laplacian quadratic form plus the log of InfoNCE’s normalizers. The paper calls this spectral clustering under a definition that admits any regularizer. It does not show that the minimizer is a set of Laplacian eigenvectors, and it names no population optimum. | Read |
| NOTE-850 | Assran et al. — The Hidden Uniform Cluster Prior in Self-Supervised Learning | MSN’s entropy term is a KL penalty pulling cluster sizes toward uniform, and SwAV’s Sinkhorn step enforces equal sizes outright; for VICReg the same prior is reached only through an informal reduction to K-means, and for SimCLR only by a cited equivalence. With mini-batches of two to eight ImageNet classes, SimCLR, VICReg and MSN lose 11–18 points of ImageNet linear accuracy while MAE and data2vec do not. A power-law prior in MSN gains about 1–3 points on long-tailed iNat18 and loses heavily on ImageNet. | Read |
| NOTE-971 | Wiskott and Sejnowski — Slow feature analysis | Finds instantaneous functions y = g(x(t)) whose outputs vary as slowly as possible, minimizing ⟨ẏ²⟩ under zero mean, unit variance and decorrelation. Over a fixed nonlinear expansion (all monomials of degree one and two), the problem is solved in closed form by sphering and then taking the smallest eigenvectors of the derivative covariance, a generalized eigenproblem. Stacked modules learn translation, size, rotation, contrast and, less well, illumination invariance for 1-D patterns, depending only on the training stimulus. The closed form is proved; everything about invariance, generalization and hierarchy is demonstrated on synthetic signals. The premise that slower means more abstract is assumed, not shown, and no generative model is used. | Read |
| NOTE-985 | Coifman and Lafon — Diffusion maps | A symmetric non-negative kernel normalized into a reversible Markov chain gives a family of diffusion distances D_t, one per number of steps t, and an embedding by eigenvectors weighted by λ_l^t in which Euclidean distance is D_t. An α-normalization by a kernel density estimate makes the small-bandwidth limit the graph Laplacian with a density potential (α = 0), the Langevin generator whose invariant density is the sampling density (α = 1/2) or the Laplace–Beltrami operator (α = 1). The truncation is claimed accurate to relative precision δ, which is wrong: the bound is absolute. Several displays carry errors, and the heat-kernel proof has a gap. | Read |
| NOTE-986 | Nadler, Lafon, Coifman and Kevrekidis — Diffusion maps and reaction coordinates | Ties diffusion maps to the dynamics that generated the data. For samples from p = e^{−U}, the small-ε generator of the α-normalized walk is Δψ − 2(1 − α)∇ψ·∇U: the graph Laplacian (α = 0) gives Fokker–Planck with potential 2U, α = 1/2 gives the Langevin dynamics with potential U itself, and α = 1 gives Laplace–Beltrami. So the slow eigenfunctions of the α = 1/2 map approximate the slow modes of the sampled stochastic system, and the paper proposes them as reaction coordinates. The expansions are formal, and with the paper’s kernel every printed generator is too large by a factor of two. Its double-well example is separable. From equilibrium samples the slow eigenfunction is recovered well and the slow eigenvalue is not. That slow eigenfunctions are closed in mean, and in law only under a condition, is the record’s addition. | Read |
| NOTE-1015 | Schüler and Wiskott — Slow feature analysis as variational inference | Under a Gaussian random-walk prior on the latents and an encoder N(g(x_t), I) that sees one time step, the ELBO is a per-point reconstruction term minus half the unordered slowness loss, an exact identity (up to a constant the paper drops by a slip). The reconstruction term stands in for SFA’s constraints, by argument. In the linear case the paper gives only a necessary condition; solving it here shows the optimum recovers SFA’s slow span on whitened input, cut at lag-one autocorrelation ½, with no basis or order, and departs from SFA on raw input. Its two-sentence account of Turner and Sahani is not a substitute for reading them. | Read |
| NOTE-1069 | Gurnee — Verbalizable representations form a global workspace in language models | A corpus-averaged Jacobian lens picks out, in each layer of Claude models, the directions a model is disposed to verbalize. Swaps, injections and ablations along these directions (the J-space) show them privileged for report, held concepts, two-hop reasoning and reuse of an argument by many functions, while continuation, anomaly detection and line wrapping use the same information without them. The J-space works in a middle band of layers, carries under 10% of activation variance, and is amplified and relayed by the weights more than other directions. Ablating it flattens experiential language about the self and about others. The authors take no position on phenomenal consciousness. | Read |
| NOTE-1166 v3 | Meng, Bau, Andonian and Belinkov — Locating and editing factual associations in GPT | Causal tracing restores one clean hidden state into a run whose subject tokens were noised, and measures how much of the correct object’s probability comes back. In GPT-2 XL, over 1000 known facts, the effect concentrates at two sites: middle layers at the last subject token, where MLP outputs dominate, and late layers at the last token, where attention does. A rank-one update of one middle MLP’s output matrix, keyed on the subject’s last token, then inserts counterfactual facts that generalize to paraphrases while sparing neighbouring subjects, which no other editor tested does on the new CounterFact dataset. The authors read this as showing where and how facts are stored. | Read |
| NOTE-1167 | Wang, Variengien, Conmy, Shlegeris and Steinhardt — A circuit for indirect object identification in GPT-2 small | A circuit of 26 attention heads in seven classes that carries GPT-2 small’s indirect-object predictions on fifteen templates, found by tracing back from the logits with path patching and checked against three criteria. Mean-ablating everything else keeps 87% of the logit difference; knocking out the three main name movers costs 5%, because other heads take over; and a greedy search finds knockouts under which circuit and model differ by up to 87%. Its by-products: heads that communicate positions rather than tokens, induction heads doing a different job from their original one, and heads that write against the answer. | Read |
| NOTE-1168 v2 | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt et al. — Causal scrubbing | A method for testing a stated mechanistic hypothesis about how a network produces a behaviour on a dataset. The hypothesis maps an interpretable computational graph into the model’s graph; every activation is then replaced, recursively from the output, by its value on another input the hypothesis says is equivalent, and unclaimed inputs by their values on a random input. The share of the behaviour that survives is the score. It is applied to a parenthesis-balance classifier (93% down to 72% as the hypothesis is made more specific) and to induction in a two-layer attention-only model (35% for the naive hypothesis, 86% refined). | Read |
| NOTE-1169 | Mueller, Geiger, Wiegreffe, Arad et al. — MIB, a mechanistic interpretability benchmark | A benchmark for two kinds of mechanistic-interpretability method, with fixed counterfactual inputs, private test sets and new metrics. In circuit localization, edge attribution patching with integrated gradients does best on average and node-level circuits poorly; in causal-variable localization, supervised distributed alignment search gives the highest interchange-intervention accuracy, and features of public sparse autoencoders selected by a learned mask never beat the same mask on the model’s own hidden dimensions. | Read |
| NOTE-1170 | Conmy, Mavor-Parker, Lynch, Heimersheim and Garriga-Alonso — Towards automated circuit discovery (ACDC) | Names a three-step workflow behind manual circuit discovery (choose a behaviour, metric and dataset; choose the units of the computational graph; patch activations to prune the graph) and automates the third with ACDC, which visits nodes from the output back and removes each incoming edge whose replacement by its value on a corrupted input raises the KL divergence from the full model by less than a threshold. Against circuits reported by earlier manual work it ranks edges with AUCs of 0.85–0.98 on three language-model tasks, beating two adapted pruning methods there, but does poorly on compiled tracr programs unless zero activations are used, misses negative components at sparse thresholds, and keeps only one input of an OR gate. | Read |
| NOTE-1173 | Olsson et al. — In-context learning and induction heads | In 34 transformer language models, in-context learning (loss at token 500 minus loss at token 50) appears abruptly in one early window of training, together with induction heads, the only non-convex bump in the loss curve and a turn in the models’ per-token-loss trajectory; one-layer models show none of it. Letting one layer express induction (smeared keys) moves the jump, and ablating induction heads removes nearly all in-context learning in small attention-only models. The case is causal for small attention-only models and correlational or by analogy for large models with MLPs, as the authors grade it themselves. | Read |
| NOTE-1174 | Dunefsky, Chlenski and Nanda — Transcoders find interpretable LLM feature circuits | A transcoder replaces one MLP sublayer with a wide sparse ReLU MLP trained on its input–output pairs. With attention patterns frozen and LayerNorm linearized, the attribution between two transcoder features factors into the earlier feature’s activation times the input-invariant dot product of its decoder with the later feature’s encoder, so circuits through MLPs can be read partly off the weights. On GPT2-small, Pythia-410M and Pythia-1.4B transcoders sit on or above the SAE sparsity–loss frontier, under one learning rate and one seed; in nine blind case studies on GPT2-small the method recovered the feature’s meaning in five, mostly in two and failed in two. | Read |
| NOTE-1177 | Musat, Pimentel, Zucchet and Hofmann — Invariant learning dynamics of transformers in inductive reasoning tasks | For a class of “block-list” next-token tasks that includes associative recall, k-hop induction and in-context n-grams, population gradient descent keeps an attention-only transformer’s merged key–query and output weights in a subspace fixed by the data’s token-relabelling and position-shift symmetries, whose coordinates are elementary actions (“attend to the previous position”, “read the token retrieved by head 1”). On it, a saturated in-weights circuit suppresses the in-context circuit’s gradient by the fraction of data it does not solve, burstiness multiplies that gradient, initialization decides which of several induction circuits wins, and projecting a trained model onto the coordinates recovers its circuit, with “aiding” directions a head-level analysis would miss. | Read |
| NOTE-1178 | Templeton, Conerly, Marcus, Lindsey et al. — Scaling monosemanticity | Sparse autoencoders of about 1M, 4M and 34M features on the middle-layer residual stream of Claude 3 Sonnet, trained to a compute-optimal loss, explain at least 65% of its variance and yield features that are multilingual and multimodal, that mostly match no neuron, and that steer the model’s output when clamped, including features for unsafe code, bias, sycophancy and deception. Whether a concept has its own feature depends on its training frequency relative to the number of live features. The interpretability and steering evidence is shown on selected features, and the most strongly active features are mostly not those that matter to an answer. | Read |
| NOTE-1181 v2 | Lindsey et al. — On the biology of a large language model | Case studies of Claude 3.5 Haiku through attribution graphs of a 30-million-feature cross-layer transcoder, each checked by inhibiting or injecting features in the model. On single prompts: “Dallas” reaches “Austin” through “Texas” features, beside a direct shortcut; a rhyme is chosen on the newline before a line and the line is written towards it; one addition lookup-table feature serves arithmetic, citations and tables; “can’t answer” features are on by default and inhibited by known-entity features; a chain of thought can work backwards from a hinted answer; a jailbreak’s target word is not represented before it is written; a goal finetuned into a model organism is represented across every Assistant context. The authors call these existence proofs and report a satisfying graph for about a quarter of prompts tried. | Read |
| NOTE-1182 v3 | Olah, Cammarata, Schubert, Goh, Petrov and Carter — Zoom in, an introduction to circuits | A programme essay, not a study. It offers three deliberately speculative claims, as empirical claims about networks and as norms for studying them: features (directions in a layer’s activation space, often single neurons) are the unit of a network; features joined by weights form circuits whose algorithms can be read off the weights; and analogous features and circuits form across models. The examples are qualitative and from InceptionV1, with the detailed evidence promised to later articles. Polysemantic neurons are named as the main obstacle, and a car feature spread over dog neurons is the first case it calls superposition. | Read |
| NOTE-1183 | Wang, Hu, Du, Cheng, Wang and Zou — Fine-tuning mechanisms of LLMs via circuit analysis | When Pythia-1.4B is fine-tuned with LoRA on arithmetic tasks it starts out failing, EAP-IG circuits of 5% of edges found at successive checkpoints keep most of their nodes and change most of their edges (change rates 15–24% for nodes, 66–80% for edges), mostly in the middle layers; giving higher LoRA rank to the layers whose edges change most beats uniform and random rank allocation at similar size, and the union of two subtasks’ circuits approximates a two-step task’s circuit. The node–edge contrast carries no comparison against a null. | Read |
| NOTE-1185 | Ameisen et al. — Circuit tracing: revealing computational graphs in language models | A cross-layer transcoder (CLT) replaces a model’s MLPs with sparse features, each reading one layer and writing to all later ones. On one prompt, freezing attention patterns and normalization denominators and adding back the reconstruction error makes a local replacement model that reproduces the model exactly and is linear between features, so its attribution graph decomposes each feature’s pre-activation into edges. The graphs are hypotheses about the model, tested by perturbing features in it. The authors’ own measurements: the replacement matches the model’s top token on about half of prompts, and its perturbation responses agree with the model’s for a layer or two and then diverge. | Read |
| NOTE-1186 | Bayazit, Mueller and Bosselut — Crosscoding through time | Trains one sparse crosscoder jointly on the mid-layer activations of two or three pretraining checkpoints, so features share an index across them, and scores each feature’s causal importance for subject–verb agreement at each checkpoint (Relative Indirect Effect). In Pythia-1B, OLMo-1B and BLOOM-1B the important features move, by the authors’ annotation, from token-specific detectors to grammatical-category ones, and in BLOOM from language-specific to crosslingual ones for Latin-script languages; importance keeps shifting after accuracy plateaus. A method for describing which features emerge, persist and lose importance, not for predicting it. | Read |
| NOTE-1187 | Marks, Rager, Michaud, Belinkov, Bau and Mueller — Sparse feature circuits | Circuits whose nodes are sparse-autoencoder features and autoencoder error terms, found in Pythia-70M and Gemma-2-2B by linear approximations to indirect effects. On subject–verb agreement, circuits of about 100 and 500 nodes carry half or more of the behaviour on held-out pairs, where neuron circuits need far more, but without the error nodes the feature circuits carry little. The annotated circuits show two pathways for agreement across a clause. A human removing gender features from a classifier’s circuit (SHIFT) brings profession accuracy on balanced data near an oracle’s, and the method runs without supervision on thousands of behaviours clustered from the Pile. | Read |
| NOTE-1188 | Wang, Hoogland, van Wingerden, Furman and Murfet — Refined local learning coefficients of attention heads | Restricts the local learning coefficient, singular learning theory’s measure of how degenerate the loss is near a parameter, to one component’s weights (weight-refined) or to another data distribution (data-refined), and tracks it per attention head over the training of a two-layer attention-only transformer. The heads’ curves start alike and diverge, and cluster by the heads’ functional types; refinement to code separates the two induction heads; refinement to a one-layer model’s outputs locates layer-0 heads giving up simple n-grams as a circuit with layer-1 bracket-matching heads forms. One model, one run in the main text, three more seeds in an appendix. | Read |
| NOTE-1189 | Elhage, Nanda, Olsson et al. — A mathematical framework for transformer circuits | An exact reformulation of attention-only transformers, and a reading of toy models with it. Heads are independent and additive, each a QK bilinear form that sets the attention pattern and an OV map that sets what moves; with patterns frozen, the logits are a sum of path terms (direct, one per head, one per composed pair of heads). Zero-layer models are bigram tables, one-layer models bigram and skip-trigram tables dominated by copying heads, and in a two-layer model the only substantial composition builds induction heads from a previous-token head by K-composition. MLP layers, two-thirds of a standard transformer’s parameters, are left open. | Read |
| NOTE-1190 v2 | Kramár, Lieberum, Shah and Nanda — AtP*, attribution patching and its failure modes | Attribution patching (AtP) estimates every node’s activation-patching effect at once from two forward passes and one backward pass, by a first-order Taylor expansion. On Pythia models from 410M to 12B, using it to order which nodes to verify by real patching finds the largest-effect nodes far more cheaply than unordered patching or sampling-based search. Its false negatives come from saturated attention softmaxes and from direct and indirect effects that nearly cancel; AtP* recomputes the softmax for queries and keys and averages gradients with one layer dropped at a time. A subsampling test gives an upper confidence bound on the size of any nodes it still misses. | Read |
| NOTE-1191 | Bricken, Templeton, Batson, Chen, Jermyn et al. — Towards monosemanticity | Sparse autoencoders trained on the 512-neuron MLP layer of a one-layer transformer, at widths from 512 to 131,072, recover features that one blinded author and Claude both rate far more interpretable than neurons, that mostly match no neuron, whose effects on the output (which the autoencoder never saw) agree with their activations, and that recur in a second model trained from another seed. The features split into finer families as the dictionary widens, so the authors doubt there is one correct number of features. Everything is in one MLP layer of a one-layer model, and the detailed cases are chosen to be easy. | Read |
| NOTE-1192 | Geiger, Wu, Potts, Icard and Goodman — Distributed alignment search (DAS) | Causal abstraction tests whether a network implements a high-level causal model by interchange interventions under an alignment of variables, which earlier work found by brute-force search over disjoint sets of neurons. DAS learns the alignment instead: an orthogonal rotation trained by gradient descent so that swapping rotated subspaces between inputs reproduces the high-level model’s counterfactuals. In a feed-forward network trained on hierarchical equality and in BERT fine-tuned on monotonicity NLI it finds alignments with interchange intervention accuracy of 1.00 where the best localist ones reach 0.73 and 0.51. In BERT the entailment subspace decomposes into the two word identities; in the equality network it does not. On random networks DAS raises accuracy above chance only when the hidden layer is much wider than the input. | Read |
| NOTE-1193 | Marks, Treutlein et al. — Auditing language models for hidden objectives | A model trained on documents describing 52 fictional reward-model biases, then by SFT and RL to exploit 47, exploits the 5 held-out biases more (17% to 45%) and new ones described in context more than other entities’ biases; without the documents the held-out rate barely moves, and without SFT it still rises. Three of four blind auditing teams found the objective, all by finding the documents in the training data; the API-only team did not. Post hoc, semantic search matched the fastest team’s SAE route, and SAE features found the objective from the Assistant token and from bias demonstrations where search failed. | Read |
| NOTE-1194 | Hase, Bansal, Kim and Ghandeharioun — Does localization inform editing? | On 652 CounterFact facts GPT-J completes correctly, the paper asks whether the layer where causal tracing places a fact predicts where an edit of that fact succeeds. It does not. Regressing edit success on the edit layer explains most of its variance (94.7% for ROME’s rewrite score), and adding the per-fact tracing effect adds at most 3.2 points, across four editing methods and five editing problems; for ROME at layer 6 the correlation is slightly negative. ROME rewrites at over 96% at every layer but the last. The authors conclude that tracing and editing answer different questions. | Read |
| NOTE-1195 | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Lin et al. — Causal scrubbing: results on a paren balance checker | Causal scrubbing applied to a small transformer that classifies parenthesis strings as balanced. Saying which three heads carry which test recovers 88–93% of the loss; adding how the heads compute their tests brings the combined hypothesis to 72%, the excess losses of its two halves roughly adding. Splitting one head’s input into separately resampled terms raised the score from 77% to 81–84%, which the authors say is an artefact of resampling, not a better explanation. | Read |
| NOTE-1196 | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Lin et al. — Causal scrubbing: results on induction heads | In a two-layer attention-only model, heads 1.5 and 1.6 contribute 0.053 nats of loss on tokens where induction helps. The textbook induction circuit accounts for 35% of that. Matching on the last three tokens, layer 0’s average effect, and three layer-0 heads that attend to copies of the current token raise it to 86% (91% unscrubbed, 89% with a crude rule for the previous-token head’s deviations). The heads use more paths than the story names, but few positions. | Read |
| NOTE-1197 | Meng, Sen Sharma, Andonian, Belinkov and Bau — Mass-editing memory in a transformer | MEMIT inserts thousands of facts into GPT-J and GPT-NeoX at once. It treats the MLP output matrix of each layer in a range that causal tracing implicates as a linear associative memory, computes for each fact a target change of the hidden state at the top of that range, and spreads it over the layers with a closed-form least-squares update for the whole batch. At 10,000 CounterFact edits on GPT-J it scores 85.8 against 50.3 for ROME applied one at a time; ROME is better at a handful of edits. | Read |
| NOTE-1198 v2 | Cammarata et al. — Curve detectors | The ten curve units of InceptionV1’s mixed3b are argued to detect curves: feature visualizations are curves, unit 3b:379 fires on about 11% of dataset positions and its activation level separates blind hand labels of over 800 images, curves and imperfect curves make up 55% of its contribution to expected value, synthetic curves drive it up to 24 standard deviations, and rotated dataset examples show the ten units tiling all orientations. Causality, generality and the family are argued from these; purity is argued but its mechanistic support is deferred to the next article, and no count of how much of the layer is understood is given. | Read |
| NOTE-1200 | Geva, Schuster, Berant and Levy — Transformer feed-forward layers are key-value memories | A feed-forward layer f(x·Kᵀ)·V has the form of a key–value memory without the softmax: each hidden unit’s input weights are a key and its output weights a value. In a 16-layer WikiText-103 language model, the training prefixes that most activate a sampled key share patterns humans recognize, shallow in layers 1–9 and semantic in 10–16; projected onto the vocabulary, upper-layer values agree with the next token of their key’s top prefix in up to about 3.5% of units; and a layer’s prediction is usually a composition of many memories, refined across layers through the residual stream. | Read |
| NOTE-1201 v2 | Nanda — Attribution patching | Activation patching costs a forward pass per patched activation. Attribution patching replaces each patch by its first-order estimate on the corrupted run, (clean − corrupted activation) · gradient, so every patch costs two forward passes and one backward pass in total, and estimates add over sub-components. On IOI in GPT-2 Small it is said to track activation patching for heads and their queries, keys, values and patterns, and to fail for residual streams and the first MLP. The post argues it should fail where nonlinearities leave saturation, where an effect needs several features at once, and for large patches through LayerNorm, and frames all patching as specific to the prompt distribution chosen. | Read |
| NOTE-1202 v2 | Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt et al. — Causal scrubbing: Appendix | The appendix the main post defers to. It defines % loss recovered against a label-shuffled baseline. It gives three reasons to prefer resampling ablation to zero or mean ablation. It proves that the algorithm’s sampled inputs are the maximum-entropy distribution that keeps every interpretation node’s inputs jointly as on the data, which the record re-checked. It explains superposition as monosemantic terms plus a resampled interference term, and builds a false hypothesis that scores ten times the model’s own expected score. | Read |
| NOTE-1203 | Ghahramani and Jordan — Factorial hidden Markov models | Defines the factorial HMM: M hidden chains of K states, a priori independent, whose one-hot states add their columns of W^(m) to set a Gaussian observation’s mean. Exact inference costs O(TMK^{M+1}); Gibbs, completely factorized and structured variational E steps make EM tractable. On synthetic problems an HMM with K^M states overfits badly while the approximate factorial HMMs stay near exact training; on Bach chorales factorial HMMs with up to 1000 states predict better than the best HMM. Its footnote 3 on when the chains decouple is wrong unless the covariance is isotropic. | Read |
| NOTE-1205 | Scherlis et al. — Polysemanticity and capacity in neural networks | In a model computing Σ(W_i·x)² + b to fit Σ v_i x_i², with independent unit-variance inputs of fourth moment k, the expected loss is (k − 1)Σ(n_i − v_i)² − 2Σn_i² + 2Σn_i²/C_i, so the marginal value of capacity is −2n_i²/C_i² and the optimum has a closed form: dense inputs (k ≤ 3) give every feature all or nothing, sparse ones (k > 3) share features whose importance lies between 2λ/(k − 1) and λ. Efficient embeddings are exactly block-semiorthogonal. Re-derived and checked. The bearing on trained networks is argued, not shown. | Read |
| NOTE-1206 | Olah et al. — An overview of early vision in InceptionV1 | Every one of the 1,056 units of InceptionV1’s first five layers is put in exactly one of 77 named groups, with each group’s share of its layer; the shares are consistent with the unit counts. By the authors’ own captions, 391 units (37%) are in groups called catch-all, broad and not well defined, low-confidence, unstudied or not understood, from 7% of conv2d2 to 48% of conv2d1. The groups are sorted by eye, and most units had “a few minutes of cursory investigation”, so the census is a denominator for THEORY-1013, not the criterion its promote_when asks for. | Read |
| NOTE-1207 | Cammarata et al. — Curve circuits | The ten curve units of InceptionV1’s mixed3b are traced back through four layers of families (Gabors, lines, early curves), each tiled along the curve’s tangent with rotational and scale equivariance; negative weights from opposite early curves remove the 180° echo, which circuit editing restores. A hand-written network built from the description approximates the units on synthetic curves, by figure, and substituting it recovers 26 of the 46 top-5 classifications lost by deleting the units, on 2,000 images. The three arguments Curve Detectors deferred are supplied, mostly qualitatively. | Read |
| NOTE-1323 | Alvarez-Melis et al. — Hyperbolic OT hierarchy matching | Two Poincaré embeddings of the same WordNet hierarchy, from different random seeds, differ by more than a hyperbolic isometry: branches move relative to one another and pairwise distances change. Euclidean aligners and hyperbolic OT with an orthogonal map therefore fail (P@10 at most 11% across languages). Registering one space with a hyperbolic neural network trained through a Sinkhorn divergence recovers P@10 of 38–60% across languages and P@1 of 22% when matching a hierarchy to itself. The matching uses the symmetric hyperbolic distance, and nothing in it preserves the direction of entailment. | Read |
| NOTE-1326 | Sarkar, Mukherjee and Sato — Relational composition as parallel transport | The companion paper’s premise, stated by its authors: per-entity frame freedom is “a chosen inductive bias, not a consequence of fresh names”. Under it, scores go through ⟨xᵢ, Tᵢⱼxⱼ⟩ and transports must be supplied. Its parsed-benchmark controls show the premise’s orthogonality helping on StepGame and costing 64.6 points on CLUTRR. | Skimmed |
| NOTE-1329 | Hashimoto et al. — Word embeddings as metric recovery | If text were generated by a random walk whose steps depend on distance in a latent Euclidean or Riemannian space, negative log co-occurrence over a window that grows with the vocabulary, less a term for each word, would approach the squared latent distance. GloVe, word2vec and the PMI-SVD method then have the true embedding among their optima, up to isometry and unigram terms, and a negative-binomial regression on the same relation embeds words and manifolds about as well. On real corpora the evidence for a Euclidean semantic space is a correlation of 0.75 between PMI and human similarity ratings and two nearest-neighbour statistics; the authors say the random walk is not a model of language. | Read |
| NOTE-1331 | Sarkar, Mukherjee and Sato — Attention as lattice gauge theory | Under per-entity O(d) frame freedom, transports carry invariant information only around cycles, a triangle’s invariants are class functions of its holonomy, and an equivariant edge update is Ψ(H)Tᵢⱼ with Ψ conjugation-equivariant: a sound conditional theorem, standard in lattice gauge theory and synchronization, whose “unique form” is a family. The untrained S4 experiments show composition doing the work and Wilson-loop routing adding 0.6–7.8 points; the 0.99 AUC is a ceiling the data’s construction sets, as the paper itself computes. | Read |
| NOTE-1332 | Bodnar et al. — Neural sheaf diffusion | Bodnar et al. read sheaf diffusion on a graph as a projection onto the harmonic space of a sheaf Laplacian and ask which classes of restriction maps can make that projection linearly separate a node labelling. A symmetric one-dimensional sheaf, which is a weighted graph Laplacian, cannot separate the two sides of a balanced bipartite graph; signed maps separate any two classes; no one-dimensional sheaf separates three; diagonal maps with d ≥ C stalks, and O(d) bundles for C ≤ 2d at d = 2 and 4, separate any number. Every one of these sheaves is built from the labels. Learned from features, such sheaves give competitive node classification on six small heterophilic benchmarks. The claim that the O(d) construction exists only at d = 2 and 4 is wrong: it works at d = 8 too. | Read |
| NOTE-1333 | Alvarez-Melis et al. — OT with global invariances | Optimal transport between two unregistered spaces is posed as a joint minimisation over couplings and one global linear map from a Schatten ℓp ball. Under angle preservation or whitening it reduces to maximising a dual Schatten norm of the transported cross-covariance XΓYᵀ: the nuclear norm for orthogonal maps (p = ∞) and the Frobenius norm for p = 2, which is exactly GW with cosine similarity and squared loss. Alternating SVD and Sinkhorn steps with annealed entropy solve it without adversarial initialisation. On word translation it is level with GW and behind self-learning on nine of ten directions. | Read |
| NOTE-1335 | Redko et al. — CO-Optimal Transport | COOT matches two data matrices of different shapes by optimizing two couplings at once, one between rows (samples) and one between columns (features), under a bilinear cost. With equal sizes and uniform weights it is zero exactly when the matrices are equal up to a permutation of rows and a separate permutation of columns, so its plans are determined at best up to the matrix’s joint row–column symmetries; the paper proves no uniqueness. For symmetric matrices it is a lower bound on GW, equal to it for squared Euclidean distance matrices. | Read |
| NOTE-1336 | Alvarez-Melis & Jaakkola — Gromov-Wasserstein word alignment | Unsupervised word translation is cast as a Gromov–Wasserstein problem between the two languages’ intra-language cosine-similarity matrices, solved by entropic projected gradient (each step a Sinkhorn problem), with no anchors. It matches adversarial-plus-refinement methods on related languages trained on comparable corpora at a fraction of the cost, and does markedly worse on English–Russian and English–Finnish. The objective sees only within-space similarities, so it cannot distinguish a correspondence from one composed with a symmetry of either space; the paper does not say so. | Read |
| NOTE-1337 | Søgaard et al. — Limits of unsupervised dictionary induction | The adversarial unsupervised method of Conneau et al. (2018) induces bilingual dictionaries only when the two embedding spaces are alike: it fails for English paired with Estonian or Finnish, for corpora from different domains even within English–Spanish, and for spaces trained by different algorithms. A seed of identically spelled words does as well or better. A spectral dissimilarity of small nearest-neighbour graphs tracks success across eight language pairs (|ρ| = 0.89). The failures are partly optimiser instability (some seeds succeed), and the method tested is adversarial, not GW. | Read |
| NOTE-1346 | Chu, Zhmoginov and Sandler — CycleGAN as steganographer | A CycleGAN trained to translate aerial photographs to maps and back reconstructs details of the photograph, such as the pattern of dots on a roof, that are absent from the visible map, because it encodes them in a low-amplitude, high-frequency signal in the map. Noise of three 8-bit levels, or JPEG compression, destroys the reconstruction; low-frequency changes do not. Any target photograph can be produced from almost any map by an imperceptible perturbation found by gradient descent, and the perturbation needed shrinks as training proceeds. The cycle-consistency loss is met by a channel the map’s human reader cannot see. | Read |
| NOTE-1441 | Coifman et al. — Diffusion maps (full reading) | With the equations visible, the paper’s diffusion distance is the distance induced by the symmetric kernel ã^(m), D_m² = ã^(m)(x, x) + ã^(m)(y, y) − 2ã^(m)(x, y), with ã’s eigenvalues written λ_i², so that the map x ↦ (λ_i^m φ_i(x)) is an exact isometry for it. The truncation is claimed with a relative error 1 + O(e^{−αm}). Theorem 2.1, the ε → 0 heat-kernel limit of the α = 1 normalization, and the α-family (Eq. 14) are stated without proof; the sampling condition is that each ball of radius √ε hold enough points. Eq. 5 and Eq. 14 give the α = 0 potential opposite signs. | Read |
| NOTE-1462 | Chen, Lau, Mendel, Wei & Murfet — dynamical versus Bayesian transitions in TMS | In Elhage et al.’s toy model of superposition with two hidden dimensions, regular k-gons are critical points of a closed-form population loss, and their local learning coefficients predict, through the free energy formula, a Bayesian phase transition from the 5-gon to the 6-gon near n = 600 that MCMC confirms. SGD trajectories plateau at the same critical points and move to lower loss and higher estimated complexity. The paper’s Bayesian antecedent hypothesis is one-directional, that training’s transitions have Bayesian counterparts; it says outright that there is “no necessary relation” between the two kinds, and its main Bayesian transition, 5 → 6, was never seen in training. | Skimmed |
| NOTE-1470 | Teoh — Multi-token prediction done right | The cross-entropy gradient of a multi-token-prediction head, taken at its hidden state under a shared output head, is exactly a forward-KL self-distillation gradient toward the model’s own position-aligned next-token prediction plus the next-token cross-entropy gradient at the aligned state; a straight-through routing removes the second from the heads and trunk at no cost, and at 167M–1.2B this improves next-token loss, speculative decoding and zero-shot accuracy by small margins in single runs. In the main (DeepSeek-style) experiments the student head sees the same tokens as its teacher, so the decomposition is not an information projection. | Read |
| NOTE-1479 | Hossain, Kousar & Tourni — Overcoming scaling limits in on-policy self-distillation | On-policy self-distillation from a privileged-context teacher does most of its supervising on failed student trajectories, where the teacher’s advantage depends on the reference the student cannot see; Proposition 1 gives the forward-KL optimum as the context-averaged teacher with the conditional mutual information as irreducible loss. Restricting supervision to verified on-policy trajectories and using the student’s own failed attempt as the teacher’s context keeps the gain over the base model at 3.2–3.8 points from 1.7B to 8B, where standard OPSD’s falls to 0.14, in single runs. The trained objective clips each vocabulary entry and is not a forward KL. | Read |
| NOTE-1501 | Vaswani et al. — Attention is all you need | The Transformer: an encoder–decoder built only from multi-head scaled dot-product attention and position-wise feed-forward layers, with sinusoidal position encodings. Attention maps a query and key–value pairs to a weighted sum of the values, the weights set by the query’s compatibility with each key. It reached 28.4 BLEU on WMT 2014 English– German and 41.8 on English–French at a fraction of earlier training cost, and transferred to constituency parsing. | Read |
| NOTE-1505 | Lu, Gallagher, Michala, Fish & Lindsey — The Assistant Axis | Role vectors for 275 archetypes in Gemma 2 27B, Qwen 3 32B and Llama 3.3 70B span a low-dimensional persona space whose first component runs from fantastical roles to Assistant-like ones, with the default Assistant at one end; the contrast vector from it, the Assistant Axis, is largely present in base models. Steering along it modulates role-taking and persona jailbreaks. In synthetic multi-turn conversations the model’s position along the axis is set mainly by the latest user message, and therapy-like and AI-philosophy conversations drift away from the Assistant; capping the projection cuts persona-jailbreak harm by about 60% without measured capability loss. | Skimmed |
| NOTE-1512 | Marks, Lindsey & Olah — The Persona Selection Model | A named statement of an existing idea: pre-training teaches a language model a distribution over personas, post-training updates it into a posterior over Assistant personas, runtime context conditions it further, and the Assistant’s traits are a key determiner of behaviour. The evidence is surveyed, not new. Its open question is exhaustiveness, laid out as a spectrum from the “shoggoth” through the “router”, which swaps personas without any persona having the goal, to the “operating system”. | Read |
| NOTE-1524 | Chen, Arditi, Sleight, Evans & Lindsey — Persona vectors | Difference-of-means directions in the residual stream, extracted automatically from a trait’s name and description, steer and monitor evil, sycophancy and hallucination in Qwen2.5-7B and Llama-3.1-8B. The shift that finetuning induces along a trait’s vector correlates with the trait’s expression afterwards (r = 0.76–0.97); steering against the vector, or towards it during training, limits the shift; and projecting training data onto the vector flags data that will induce the trait. Prompt-time monitoring works mainly between prompt types. | Skimmed |