concept-geometry.
13 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-206 | The Platonic Representation Hypothesis | Vision and language models measure distance between datapoints more alike as they get larger, across architectures and objectives. The conjecture is that they converge on a model of what generated the data. The proof holds for bijective observations only, and the caption-density experiment tests that boundary rather than asserting past it. | Read |
| NOTE-210 | The Lattice Representation Hypothesis | A linear concept direction plus a threshold is a half-space; intersecting half-spaces gives a concept lattice, so inclusion, intersection and union become geometric meet and join on embeddings. Canonical form when the attribute directions are linearly independent. Evidence is WordNet sub-hierarchies, which is the friendliest available test. | Read |
| NOTE-270 | Concepts live in the quiet directions, and the paper labels its own claims robust or partial | Read last of the batch and the easiest call to make, for a reason that has little to do with the finding: it arrives with a table grading its own six claims, a robust null against the hypothesis it set out to confirm, and partial results labelled partial. The finding itself is that contextual concept directions sit at the bottom of the unembedding spectrum and static vocabulary contrasts sit at the top. | Read |
| NOTE-288 | The transition the posterior makes and the transition SGD makes are not the same transition | Read for the distinction in its title, which the record needed and did not have. Bayesian phase transitions are changes in where the posterior concentrates as sample size grows, and are well defined; dynamical transitions are changes in the SGD trajectory, and are not. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600 and observed at 600–700 — has no dynamical counterpart at all. | Read |
| NOTE-307 | Scaling and evaluating sparse autoencoders | TopK SAEs, a dead-latent recipe, and scaling laws, demonstrated on a 16M-latent SAE on GPT-4. Also shows that the field’s usual fidelity metric flatters. Sections 1–4 and 6 read. Section 5 (TopK variants, progressive codes) and the appendices skimmed. | Read |
| NOTE-309 | Are Sparse Autoencoders Useful? A Case Study in Sparse Probing | On 113 datasets and five regimes, probes on SAE latents do not beat, or usefully add to, probes on raw activations once method selection is honest. Earlier wins, including the authors’ own, came from weaker baselines. Main text read, appendices skimmed. | Read |
| NOTE-325 | word2vec (architectures) | CBOW and skip-gram, log-linear models that learn word vectors from billions of words cheaply, and the analogy benchmark that made “king − man + woman ≈ queen” a test. Read in full. | Read |
| NOTE-326 | word2vec (negative sampling, subsampling, phrases) | The tricks that made word2vec the standard: negative sampling with a unigram^(3/4) noise distribution, subsampling of frequent words, and data-driven phrase tokens, plus additive compositionality. The 3/4 exponent is asserted without numbers. Read in full. | Read |
| NOTE-327 | GloVe | Derives a word-embedding objective from the claim that meaning lives in ratios of co-occurrence probabilities. It fits log co-occurrence counts by weighted least squares and beats default-configured word2vec on analogies. Read in full. | Read |
| NOTE-328 | Linear Representation Hypothesis (Park, Choe, Veitch) | Formalizes “concepts are directions” with counterfactual word pairs, proves that the unembedding direction is a probe and the embedding direction a steering vector, and shows the two coincide under a causal inner product. One such product is the inverse unembedding covariance. The experiments, on LLaMA-2 7B, are qualitative. Read in full, appendices included. | Read |
| NOTE-332 | Arora et al.: RAND-WALK and PMI in low dimensions | A random-walk generative model with isotropic word vectors predicts PMI ≈ ⟨v, v’⟩/d in low dimensions, justifies GloVe’s biases and CBOW’s averaging, and explains why analogies survive large fitting error: isotropy denoises the offset. Its assumptions are checked loosely on one corpus. Read in full, with the main-text proofs. | Read |
| NOTE-353 v2 | Rarely categorical, highly separable representations along the cortical hierarchy | In 14,000+ IBL Neuropixels units (4,617 passing a ΔR² ≥ 0.015 selectivity threshold), the clustering of 8-D encoding-model selectivity vectors beats a covariance-matched Gaussian null in only a handful of the ~20 testable cortical regions (VISp, AUDp, SSp-ul at Bonferroni p < 0.05; clustering z falls with hierarchy position, ρ = −0.62). Pooled modules and the whole cortex do cluster (whole-cortex z = 8.0). Among the 16 regions with enough trials, once the 16 task conditions are merged into pairwise-decodable “independent” ones (M_IC from 5 to 16, rising with hierarchy, ρ = 0.77), ≥ 95% of random balanced dichotomies are linearly decodable above a shuffle null in every region except gustatory cortex (≈0.82). | Read |
| NOTE-355 | Topos and Stacks of Deep Neural Networks | The authors claim that every DNN architecture defines a Grothendieck topos of sheaves over a site built from its graph, that backpropagation is a flow of morphisms in that topos, and that layer invariances (CNN translations, LSTM structure) are stacks, and they conjecture that these stacks explain generalization and support a homological notion of “semantic information”. | Skimmed |