Reading statuses — How deeply the paper was READ — not whether it is any good, which is lit-statuses. The two disagree in every direction: a foundational paper can carry a Skimmed note, and a paper in the attic can carry a Read one, since reading it carefully is often how it got there (ADR-025).
Re-read — a later reading replaced this one, and names it.
39 of 1538 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-198 v2 | Minimum Description Length Revisited | MDL, reformulated around universal distributions and luckiness functions and presentable without coding theory, extends both penalized likelihood and Bayes, and unifies methods usually seen as rival — AIC vs BIC, cross-validation vs Bayes — within one worst-case framework. | Superseded — by NOTE-760 |
| NOTE-199 v2 | Balestriero & LeCun 2022 — SSL recovers spectral embedding | VICReg, SimCLR and Barlow Twins each solve, in closed form, a classical spectral embedding problem on the positive-pair graph G (Laplacian Eigenmaps, a generalized MDS/ISOMAP, and CCA respectively), so contrastive vs non-contrastive SSL is global vs local spectral embedding, and when G matches the downstream task any of them is optimal. | Superseded — by NOTE-752 |
| NOTE-200 v2 | Rate-distortion generalization bounds | Defining an algorithm’s compressibility in source-coding terms shows that the Xu–Raginsky mutual-information bound is really a lossless compression rate, and allowing lossy compression yields bounds via the rate-distortion dimension that stay meaningful where I(S;W) is huge. | Superseded — by NOTE-780 |
| NOTE-201 v2 | Assran et al. 2022 — hidden uniform cluster prior in SSL | Joint-embedding SSL methods that rely on mini-batch volume maximization (SimCLR, VICReg, SwAV, MSN) implicitly impose a prior toward features that cluster the data into equal-sized groups, which hurts on class-imbalanced data; replacing it with a power-law prior (PMSN) improves representations on long-tailed data. | Superseded — by NOTE-850 |
| NOTE-202 v2 | Kolmogorov’s structure functions and model selection | The structure function of a single data string determines all of its stochastic properties: within any complexity-constrained model class it picks out the best-fitting model with certainty. Every admissible shape of that function occurs for some data, and neither it nor the minimal sufficient statistic is computable. | Superseded — by NOTE-787 |
| NOTE-203 v2 | MDL induction, Bayesianism, and Kolmogorov complexity | Ideal MDL, derived from Bayes’s rule with the universal prior, coincides with Bayesian MAP selection exactly when the data are random with respect to the hypothesis and the hypothesis is random with respect to the prior (the “Fundamental Inequality”). For finite-set models it reduces to Kolmogorov’s minimal sufficient statistic, so compression is almost always the best strategy for both identifying hypotheses and predicting. | Superseded — by NOTE-801 |
| NOTE-204 v2 | GNS construction (Wikipedia) | Any state (positive normalized linear functional) on a C*-algebra determines, essentially uniquely, a Hilbert space and a cyclic *-representation in which that state becomes a vector expectation ρ(a)=⟨π(a)ξ,ξ⟩, and the representation is irreducible exactly when the state is pure. | Superseded — by NOTE-857 |
| NOTE-205 v2 | Balestriero, Bottou & LeCun 2022 — regularization is class-dependent | Regularizers tuned by cross-validation for average accuracy — random-crop augmentation and even uninformed weight decay — raise mean test accuracy while sharply lowering it on particular classes (e.g. ImageNet “barn spider” 68%→46% with random crop), and this class-dependent bias carries over to transfer tasks. | Superseded — by NOTE-845 |
| NOTE-206 v2 | Adler — Hilbert spaces and the Riesz representation theorem | Every continuous linear functional on a Hilbert space is the inner product with a unique vector of equal norm, and the representer can be written explicitly as the sum of φ(e_k)e_k over any orthonormal basis, which in turn yields the Radon–Nikodym theorem. | Superseded — by NOTE-854 |
| NOTE-207 v2 | MDL generalization guarantees for representation learning | Generalization of representation learning is bounded not by I(X;U) but by the minimum description length of the latent variables (or predicted labels) under a symmetric prior — a multi-letter relative entropy that reflects the encoder’s structure and is non-vacuous for deterministic encoders. | Superseded — by NOTE-742 |
| NOTE-208 v2 | Sun et al. 2023 — spectral theory of novel class discovery | For novel class discovery, a spectral contrastive loss over a graph of labeled and unlabeled data (NSCL) is equivalent to factorizing its adjacency matrix, and the resulting linear-probe error on novel classes is bounded — down to zero — by how far the known classes’ feature span covers the unlabeled data’s “ignorance space”. | Superseded — by NOTE-833 |
| NOTE-209 v2 | Tan et al. 2023 — contrastive learning is spectral clustering | SimCLR’s unmodified InfoNCE loss is equivalent to spectral clustering on the augmentation-defined similarity graph (min tr(Zᵀ L(π) Z) + log R(Z)), CLIP does generalized spectral clustering on the image–text pair graph, and mixtures of exponential kernels (Kernel-InfoNCE) beat the Gaussian kernel. | Superseded — by NOTE-846 |
| NOTE-210 v2 | The Conditional Entropy Bottleneck | Training a representation to minimize I(X;Z|Y) while maximizing I(Y;Z) — the Conditional Entropy Bottleneck, targeting the “Minimum Necessary Information” point I(X;Y)=I(X;Z)=I(Y;Z) — empirically improves accuracy, adversarial robustness, OoD detection and calibration and refuses to memorize random labels. | Superseded — by NOTE-754 |
| NOTE-211 v2 | Knowledge Sheaves (sheaf-theoretic KG embedding) | Knowledge-graph embedding can be recast as learning an approximate global section of a cellular sheaf on the schema graph, which subsumes Structured Embedding and TransE/TransR-style models and yields, via harmonic extension, a training-free way to answer composite multi-hop queries. | Superseded — by NOTE-843 |
| NOTE-212 v2 | Meaningful information (sophistication) | With total recursive functions as the model class, the algorithmic minimal sufficient statistic (“sophistication”) is a non-trivial measure of meaningful information. It matches the finite-set and probability-model versions up to additive terms, some objects are entirely meaningful with no residual randomness, and it is not computable. | Superseded — by NOTE-790 |
| NOTE-213 v2 | Shannon information vs Kolmogorov complexity (survey) | Shannon’s and Kolmogorov’s theories line up concept by concept (entropy vs complexity, probabilistic vs algorithmic mutual information and sufficient statistics, rate-distortion vs structure function), and in each pair the Shannon notion is, up to additive terms, the expectation of the Kolmogorov one. | Superseded — by NOTE-776 |
| NOTE-214 v2 | Riesz representation theorem (Wikipedia) | A Hilbert space is isometrically (anti-)isomorphic to its continuous dual — every continuous linear functional φ is x ↦ ⟨x, f_φ⟩ for a unique f_φ with ‖f_φ‖ = ‖φ‖ — and this identification is what defines adjoints and the bra–ket correspondence. | Superseded — by NOTE-861 |
| NOTE-215 v2 | Bra–ket notation (Wikipedia) | Dirac’s bra–ket notation writes vectors as kets and continuous linear functionals as bras, and on a Hilbert space the Riesz representation theorem is what licenses treating every bra as the conjugate of a ket, so that ⟨φ|ψ⟩ is at once a functional applied to a vector and an inner product. | Superseded — by NOTE-855 |
| NOTE-216 v2 | Variable-size compressibility generalization bounds | Letting the compression rate of an algorithm’s input data vary with the observed sample yields generalization bounds that depend on the empirical measure rather than the unknown distribution, and this single framework recovers PAC-Bayes and data-dependent intrinsic-dimension bounds as special cases. | Superseded — by NOTE-806 |
| NOTE-217 v2 | Natural Kinds (SEP) | Whether scientific kinds are natural breaks into four questions (are the classifications natural, are they into kinds, what the kinds are, and do they have essences). The entry maps the positions from conventionalism to essentialist strong realism, notes a recent turn toward pluralist and epistemology-first views and even eliminativism, and questions whether Kripke–Putnam semantics can deliver essences. | Superseded — by NOTE-769 |
| NOTE-218 v2 | Simon et al. 2023 — stepwise nature of SSL | Joint-embedding SSL learns its embedding one dimension at a time in discrete, well-separated steps — in a linearized Barlow Twins model it provably learns the top eigenmodes of a contrastive kernel in order, so SSL is to kernel PCA what supervised learning is to kernel regression — and the same stepwise rank growth appears in ResNets trained with Barlow Twins, SimCLR and VICReg. | Superseded — by NOTE-741 |
| NOTE-219 v2 | Duality of Bures and Shape Distances | The Procrustes shape distance between two representations (optimal rotational alignment of units) equals the Bures distance between their centred linear kernel matrices, so the rotation-invariant geometry of a representation is exactly the geometry of its kernel, and through Uhlmann’s theorem the fidelity of two kernels is the maximum overlap over all representations consistent with them. | Superseded — by NOTE-753 |
| NOTE-221 v2 | Closeness in Distribution vs Representational Similarity | For softmax models p(y|x) ∝ exp(f(x)ᵀg(y)) under a diversity condition, the model distribution determines the embeddings and unembeddings up to an invertible linear map (f = Af′, g₀ = A⁻ᵀg′₀). The determination is not stable under KL divergence, though: two models can be ε-close in KL, even both near maximum likelihood, while their embeddings are far from linearly equivalent. | Superseded — by NOTE-738 |
| NOTE-222 v2 | Tosh et al., contrastive learning and multi-view redundancy | The logistic contrastive problem of telling a same-datum pair of views from an independent pair is solved over all functions by the pointwise mutual information f*(x,z) = log p(x,z)/(p(x)p(z)), whose exponential g* is the change of measure from p(z) to p(z|x), and linear functions of the landmark embedding (g*(x,Z_1),…,g*(x,Z_m)) are near-optimal predictors whenever the two views are redundant for the label. | Superseded — by NOTE-841 |
| NOTE-223 v2 | Similarity of NN Representations Revisited (CKA) | A similarity index for neural representations should be invariant to orthogonal transformation and isotropic scaling but not to arbitrary invertible linear maps, and comparing the example-by-example Gram matrices (linear or kernel CKA) achieves exactly that, while any index invariant to invertible linear maps becomes uninformative once width reaches the number of examples. | Superseded — by NOTE-743 |
| NOTE-224 v2 | Belghazi et al., MINE | Mutual information is stated measure-theoretically as the expectation under the joint of log dP_XZ/d(P_X⊗P_Z), and its Donsker–Varadhan dual is tight exactly at T* = log dP/dQ + C, so a neural critic trained on the dual is a log Radon–Nikodym-derivative estimator. | Superseded — by NOTE-828 |
| NOTE-225 v2 | What Similarity Measures Imply about Decodability | For linear readouts regularised by w ↦ wᵀG(X)w, the optimal decoded signal is K_X z with K_X = X G(X)⁻¹ Xᵀ / M. The expected alignment of two networks’ optimal readouts is therefore Tr(K_X K_z K_Y), which recovers linear CKA, CCA, GULP and ENSD as average decoding similarities. Procrustes distance bounds the average decoding distance from both sides, with constants set by a participation ratio. | Superseded — by NOTE-834 |
| NOTE-226 v2 | Kernel mean embedding of distributions (review) | Point evaluation is bounded on an RKHS, so the Riesz theorem gives the reproducing kernel k_x. Likewise, when E√k(X,X) < ∞ the expectation functional f ↦ E_P f is bounded, so Riesz gives a unique mean embedding μ_P with E_P f = ⟨f, μ_P⟩. MMD is then the RKHS distance ‖μ_P − μ_Q‖, and it separates distributions exactly when the kernel is characteristic. | Superseded — by NOTE-838 |
| NOTE-227 v2 | Spectral contrastive loss (HaoChen et al.) | Minimizing the population spectral contrastive loss L(f) = −2E[f(x)ᵀf(x⁺)] + E[(f(x)ᵀf(x⁻))²] is, up to a constant, a rank-k factorization of the normalized adjacency matrix of the augmentation graph, so its minimizers are the top-k eigenvectors (eigenfunctions, in the infinite case) of that graph up to a row scaling and an invertible linear map — neither of which changes linear-probe accuracy. | Superseded — by NOTE-740 |
| NOTE-228 v2 | Contrastive learning finds an optimal eigenbasis | The InfoNCE, NT-Logistic and spectral contrastive losses all have the same population minimizer, the positive-pair kernel K⁺(a₁,a₂) = p⁺(a₁,a₂)/(p(a₁)p(a₂)). Kernel PCA under K⁺ returns exactly the eigenfunctions of the positive-pair Markov chain, and the top-d of those span the linear-predictor subspace that minimizes worst-case approximation error over approximately view-invariant targets. | Superseded — by NOTE-751 |
| NOTE-229 v2 | Information Theory with Kernel Methods | With a kernel normalised to k(x,x) = 1, the covariance operator Σ_p = E_p[ϕ(x)ϕ(x)*] is a density operator (PSD, unit trace), injective in p when k² is universal. Its von Neumann entropy and relative entropy behave like Shannon quantities, with D(Σ_p‖Σ_q) ≤ D(p‖q), and the empirical versions are computed from the normalised Gram matrix K/n. | Superseded — by NOTE-837 |
| NOTE-230 v2 | A Generalized Representer Theorem | For any strictly increasing regularizer g(‖f‖) and any (even non-convex, coupled) cost on the training outputs, every RKHS minimizer of c((x_i,y_i,f(x_i))_i) + g(‖f‖) is a finite kernel expansion f = Σ α_i k(·,x_i). The proof needs only the reproducing property and orthogonal decomposition onto span{k(·,x_i)}, not the Riesz theorem. | Superseded — by NOTE-836 |
| NOTE-231 v2 | Spectral Inference Networks (SpIN) | The top-N eigenfunctions of a symmetric kernel operator Kf = E_{x′}[k(x,x′)f(x′)] on L²(p) can be learned by a neural network that maximizes the generalized Rayleigh quotient Tr(Σ⁻¹Π), with Σ = E[u uᵀ] and Π = E[k(x,x′)u(x)u(x′)ᵀ]. Slow Feature Analysis is the special case where k is the graph Laplacian of adjacent video frames. | Superseded — by NOTE-824 |
| NOTE-232 v2 | Tschannen et al., on MI maximization for representations | The success of InfoNCE-style representation learning cannot be attributed to mutual information itself: MI is invariant under invertible reparametrizations, invertible encoders that maximize true MI can be worse than raw pixels, and tighter MI bounds from higher-capacity critics can give worse representations. | Superseded — by NOTE-749 |
| NOTE-233 v2 | Poole et al., variational bounds of mutual information | Every variational lower bound on mutual information in use — Barber–Agakov, Donsker–Varadhan/MINE, NWJ/f-GAN-KL, InfoNCE — is one family, tight at a critic that is a function of the log density ratio log p(y|x)/p(y), and InfoNCE is the multi-sample member that trades variance for a hard ceiling of log K. | Superseded — by NOTE-739 |
| NOTE-234 v2 | SSL-HSIC (kernel dependence maximization) | With image identity as the label, the SSL-HSIC loss −HSIC(Z,Y) + γ√HSIC(Z,Z) has a dependence term that is proportional to the average squared MMD between the per-image distributions of augmented-view representations (App. B.2). InfoNCE approximates the same term plus a variance penalty (eq. 7), so contrastive SSL separates the kernel mean embeddings of each image’s view distribution. | Superseded — by NOTE-830 |
| NOTE-237 v2 | Budroni et al., Kochen–Specker contextuality (RMP review) | A review that fixes a minimal definition of Kochen–Specker contextuality as the impossibility of a global joint distribution whose marginals are the context distributions (Fine’s theorem; the sheaf-theoretic framework is one of several equivalent formulations), and that treats Spekkens’s generalized contextuality as a different notion, covered only to mark the difference. | Superseded — by NOTE-844 |
| NOTE-238 v2 | Spekkens, generalized contextuality (2005) | Defines a noncontextual ontological model as one in which operationally equivalent procedures (preparations, measurements, transformations) get identical representations; shows the traditional Kochen–Specker notion is the special case of measurement noncontextuality for sharp measurements plus outcome determinism, proves that preparation noncontextuality implies that outcome determinism, and proves preparation contextuality already for a qubit. | Superseded — by NOTE-748 |
| NOTE-784 v2 | Coifman et al. — Diffusion maps | From a symmetric, positivity-preserving, positive semi-definite kernel, normalized into a Markov operator conjugate to a symmetric one, the top eigenfunctions scaled by powers of their eigenvalues embed the data so that Euclidean distance equals diffusion distance. A one-parameter normalization separates geometry from sampling density, recovering the Laplace–Beltrami operator at α = 1. The symmetry of the kernel is what yields the spectral decomposition; non-symmetric kernels are not treated. | Superseded — by NOTE-1441 |