Source
- Park, Choe & Veitch (2023), LIT-304: §3 (eq. 3.1), Def. 3.1, Thms 3.2 and 3.4, §3.2 and App. D.2, as read in NOTE-302.
- Supporting: Elhage et al. (2022), LIT-323, as read in NOTE-278, for the superposition limit.
- Supporting: Nielsen et al. (2025), LIT-253, Thm 2.2, Defs. 4.5–4.6, Thm 4.7 and App. E.3, as read in NOTE-738, for a further instance of the same choice.
What was actually shown
The invariance, proved in the source. P(y | x) ∝ exp(λ(x)ᵀγ(y)) is unchanged by γ ↦ Aγ + β and λ ↦ A⁻ᵀλ, for any invertible A. So the final-layer context vectors and the unembeddings are identified at best up to GL(d). Inner products between concept directions are not preserved, ⟨γ̄_W, γ̄_Z⟩ ≠ ⟨Aγ̄_W, Aγ̄_Z⟩, and Euclidean cosine has no meaning until an inner product is chosen (NOTE-302, C3).
The choice, conceded in the source. A causal inner product is one that makes causally separable concepts orthogonal (Def. 3.1). Which concepts are separable is the authors’ judgement. Under such an inner product, the Riesz map sends each concept’s unembedding direction to its steering direction (Thm 3.2). Thm 3.4 assumes d mutually separable concepts whose directions G form a basis, and uncorrelatedness over the vocabulary. It then gives M⁻¹ = GGᵀ and GᵀCov(γ)⁻¹G = D for a positive diagonal D. Setting D = I gives M = Cov(γ)⁻¹. The paper says “We do not have a principle for picking out a unique choice of D” (NOTE-302).
What the data constrain, derived by the reader. Take any symmetric positive-definite M. Let LLᵀ = M⁻¹, let O diagonalise LᵀCov(γ)⁻¹L, and set G = LO. Then both equations hold. So the covariance by itself excludes no inner product, the Euclidean one included, since M = I is met by taking G to be the eigenvectors of Cov(γ). An inner product is ruled out only by knowledge of the true concept directions G, and given G, M = (GGᵀ)⁻¹ has no freedom left (NOTE-302). NOTE-302 checked this for d = 6, and I re-checked it when filing. Both equations hold to about 4·10⁻¹⁶.
A limit from superposition (NOTE-278’s inference). No inner product makes more than d nonzero directions pairwise orthogonal. Def. 3.1 asks for all separable concepts to be orthogonal, and Thm 3.4 wants d of them to form a basis. Applied to more than d superposed features, the framework must either declare most of them non-separable or fail. The 27 concepts tested in d = 4,096 never reach that limit.
This is THEORY-017’s thesis for an ML model with a larger group. For a softmax output the intrinsic symmetry is GL(d), not O(d). So not even an inner product is intrinsic, and the analyst supplies one from separability judgements and a stipulation.
What this does not say
- That Cov(γ)⁻¹ is a bad choice. On Gemma-2B the Euclidean product fails and the whitened one recovers the expected structure. On LLaMA-2 the Euclidean product “somewhat works” (App. D.2). The claim is about what is identified, not what is useful. The evidence for either is read from heatmaps, with no statistic (NOTE-302).
- Anything about intermediate layers. The argument uses the softmax symmetry, which acts only on the final context vector and the unembedding. Interpretability at hidden layers faces a different, unstated group (NOTE-302).
- That kernel comparisons across models are meaningless. The context kernel λ(x)ᵀλ(x′) is not identified at the output, so THEORY-004’s premise of a given kernel is an extra choice there. The Platonic hypothesis’s kernels (LIT-302) are average-pooled hidden states, where this argument does not directly apply (NOTE-286).
- That a Riesz argument forces a unique factorisation. Riesz carries over whatever inner product it is given and supplies none (NOTE-302; record/curation.d/2026/09/29/230637.md).
Connections
LIT-253 (Nielsen et al.) starts from the same symmetry. It is also a further instance of this theory’s thesis. To compare models in a way invariant to that symmetry, the authors move to the identified coordinates Lᵀf(x), the log-odds of each label against a pivot label y₀, and then standardize and compare them up to an orthogonal map (Defs. 4.5–4.6, Thm 4.7). Those coordinates depend on the pivot, the label set and the left-out label, and in the experiments these are chosen to make the distance smallest (App. E.3). So a usable geometry again comes from a choice, here of basis rather than of inner product (NOTE-738).
Engels et al.’s separability index and cosine clustering are Euclidean too, and so are relative to a choice this theory says is not given (NOTE-273).
Two accounts filed on 2026-10-14 bear on the thesis from the comparison side. THEORY-236 (Kornblith et al., LIT-254, Thm 1): an index invariant to every invertible linear map takes one value on all representations whose rank equals the number of examples. So a comparison invariant to everything the softmax leaves free says nothing once the representation is wide enough. A usable one must fix a smaller group by stipulation, or compare in identified coordinates. THEORY-230 (Picozzi, LIT-936) moves the comparison to the outputs: the Fisher–Rao geometry of the next-token laws is a function of the laws alone, so no output-preserving reparameterization, this account’s GL(d) among them, changes it.
Note of 2026-10-27: the residual stream’s freedom, and dictionaries as a choice of basis
Two of the circuits works read on 2026-10-26 at the owner’s request bear on the same freedom inside the network, where this account concerns only the last layer.
- The internal form. Elhage et al.’s framework (LIT-1534, NOTE-1189) states it for the residual stream: it has no privileged basis, and rotating it together with every matrix that reads from or writes to it leaves the model unchanged. The framework’s path expansion (THEORY-988) is built from products that the rotation leaves alone.
- A dictionary is a choice of basis. A sparse autoencoder’s dictionary is one choice inside this freedom, fixed by sparsity, and the number of its units is a further choice: features split and merge as the dictionary grows (THEORY-1016), and which concepts get a feature depends on their frequency relative to its size (THEORY-992). Towards Monosemanticity’s agreement of features across two seeds (median activation correlation 0.72, against 0.46 for neurons; NOTE-1191) is a check that the choice reproduces, not a proof that it is unique.
The account, which concerns the final layer under softmax training, is unchanged.