nucleation

LIT-304: The Linear Representation Hypothesis and the Geometry of Large Language Models

2 min read

Properties6
StatusActive — read in full 2026-09-29 (NOTE-302); worth reading as the short, precise source of the counterfactual-pair formalisation of the linear representation hypothesis and of the observation that softmax training leaves the representation's inner product unidentified. Read its theorems as statements about the final-layer context vector and the unembedding only. Read the choice M = Cov(γ)⁻¹ as a stipulation (D = I), not as a result, and the experiments as qualitative.
Version2
Tagsrepresentation-learning, mathematics
SupportsCLAIM-148 — A matrix factorization is identified only up to an invertible change of latent coordinates, which weight decay narrows to rotations and nonnegativity to rescalings and permutations plus a data-dependent remainder; such a gauge is not a symmetry of the relational system, and automorphisms of the data act on the factors only when the factorization is identifiable
Cited as “paper” byNOTE-302 — The Linear Representation Hypothesis and the Geometry of Large Language Models
Cited as “source” byTHEORY-018 — Softmax training identifies a language model's final-layer representation only up to an invertible linear map, so no inner product on it is intrinsic, and Park, Choe and Veitch's causal inner product is fixed by a stipulation

Graph View

Kiho Park, Yo Joong Choe, Victor Veitch (2023), ICML 2024 (PMLR 235) — ARXIV-2311.03658

Standing in the record

Registered on 2026-09-29 at the owner’s request, because the owner’s prior-art novelty map (a private working document on concepts as projectors, the linear representation hypothesis and their neighbours) cites it: §0 and §2.1, row 1 context, §4 must-cite: the LRH program the response diagnoses as the single-context Boolean shadow. It is seeded from metadata and published descriptions and has not been read here.

It is also held in the Anthology of the SOTA as ANTH-LIT-606. Holding it here too follows ADR-013: this record holds it for the owner’s prior-art map, the anthology for ML practice.

Identification notes: Held in the anthology (ANTH-LIT-606); dual holding in nucleation per ADR-013, reason: cited in the owner’s prior-art novelty map.

It was filed Deferred, unread. NOTE-302 is the close reading of 2026-09-29, and it placed the work: Active — worth reading as the short, precise source of the counterfactual-pair formalisation of the linear representation hypothesis and of the observation that softmax training leaves the representation’s inner product unidentified. Read its theorems as statements about the final-layer context vector and the unembedding only. Read the choice M = Cov(γ)⁻¹ as a stipulation (D = I), not as a result, and the experiments as qualitative.

Backlinks