nucleation

LIT-936: The information geometry of large language models is shared, learned, and controllable

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-755) in full; worth reading as a measured, coordinate-free relational signature. The matrix of Fisher–Rao distances between a model's next-token laws over a context battery is fixed by behaviour and invariant to any output-preserving reparameterisation, while activation geometry has a GL(d) gauge. It agrees across ten transformer, state-space and recurrent models at a mean rank correlation of 0.88, against 0.62 for mid-layer activations, and a probe on distances to 40 anchors transfers between models. Read it knowing three things. The signature is identified because it is a function of the observed laws, the in-probe case QUESTION-027 calls automatic. Its agreement is proved only as a consequence of each model fitting the same law, on a shared outcome space, and only about 9% of shared eigenvector structure goes beyond what the shared laws imply. It is a single-author v1 preprint whose code is on request. Most of its length is natural-gradient steering, editing and fine-tuning, which is machine-learning practice.
Version3
Tagsinformation-geometry, representation-learning, anthology-candidate
Cited as “paper” byNOTE-755 — Picozzi — The information geometry of large language models is shared, learned, and controllable
Cited as “source” byTHEORY-230 — The Fisher–Rao geometry of a language model's next-token laws over a set of contexts is a function of those laws alone, so models with identical laws share it, while activation geometry changes under invertible reparameterizations of the hidden layer that leave outputs unchanged; in Picozzi's ten transformer, state-space and recurrent models it agreed at a mean rank correlation of 0.88 against 0.62 for activation geometry, and about 9% of the top eigenvector overlap exceeded a null matched to the shared laws

Graph View

Dario Picozzi (2026), arXiv:2609.11063.

Key takeaways

From the reading, NOTE-755.

  • Behaviour fixes output geometry, not activation geometry. The relational geometry, d_ij = 2 arccos Σ_a √(p_i(a) p_j(a)) over a battery of contexts, is a function of the conditional laws. Any A ∈ GL(d) applied to a hidden layer and undone downstream changes activation geometry and leaves behaviour unchanged (Proposition 1). Two models with identical outputs can have hidden geometries that rank-correlate at −4/17 (Proposition 3).
  • Identification of the read-out subspace. Under finite support and strictly positive laws, profiled predictive risk obeys K(S) ≥ c Δ(S, T)², so a rank-matched read-out identifies the language’s subspace T (SN3, Theorem 5).
  • Shared across architectures. Ten models from 70M to 7B parameters (transformer, state-space, recurrent) agree at a mean of 0.88 on natural text, against 0.62 and 0.61 for mid- and last-layer activations. Agreement is 0.91 after mapping to the first decoded byte.
  • Why they agree. Excess risk to a common law bounds the disagreement: E(d_A − d_B)² ≤ 4(ε_A + ε_B) (Theorem 6). On real models that bound is vacuous. The measured reason is that per-context disagreement, though large, lies nearly orthogonal to the relational structure (coherence about 0.03).
  • Little is shared beyond the laws. Residual eigenvector sharing over a displacement-matched null is +0.063, “about 9% of the raw overlap”. The model-specific residual has no semantic alignment and transfers below chance.
  • Transfer. A probe on Fisher–Rao distances to 40 anchors, trained on one model, labels held-out prompts through another at 0.659 eight-way accuracy, against 0.724 within model and chance 0.125.
  • Origin. In synthetic languages the assigned law, not the architecture, determines the learned geometry (0.998 of imposed separation recovered). Randomised evidence depth delays fact acquisition about 4.3×.
  • Control. The minimum-disturbance step is δ = b A⁻¹q/(qᵀA⁻¹q) with A = G + αR, and its cost advantage over Euclidean steering is predicted in advance (median measured/predicted 0.967 over 3,515 observations). The advantage is local and shrinks with step size.

Standing in the record

Held for QUESTION-027. It gives the record its first measured relational signature that is stable under an admissible class of transformations and identified from probes. The transformations are output-preserving reparameterisations of a network and changes of architecture fitted to one law, and the probes are the next-token laws on a context battery. The identification is the automatic, in-probe kind, because the signature is a function of what is observed. The cross-model stability is that of different fits to the same law, not of realizations whose laws differ. So it illustrates the unrestricted form of QUESTION-027 and supports CLAIM-209’s unrestricted sentence. Its anchor-probe transfer and its cross-family human alignment are measured decisions on prompts or models outside the fitted ones. Those are the nearest the record holds to CLAIM-209’s restricted clause, and they are empirical and narrow. Its anchor construction is Moschella et al.’s (LIT-849), and its results agree with THEORY-175. Propositions 1 and 3 are a networks case of CLAIM-148. Its results are filed as THEORY-230.

It cites Marconato et al. (LIT-950), whose all-or-none theorem is the exact-equality counterpart of its approximate convergence, and the Platonic Representation Hypothesis (LIT-302). The paper carries instructions for machine-learning practice: steer, edit and fine-tune with the pullback Fisher metric. That part belongs in the anthology, as the anthology-candidate flag records. It is held here for its own question under ADR-013.

Backlinks