Source
Shai et al. (2026), LIT-986, §§4.1–4.3 and Apps. C.4, D.3–D.5, read in full in NOTE-758. The lossless embedding (Prop. 2.2, Thm 2.3) is the theorem THEORY-256 states; the learned representation is the paper’s experimental finding, which it frames as a property of next-token training on factored processes.
What is asserted, and by whom
The thesis is that what THEORY-256 measured on one family is what next-token training does on factored processes in general: the learned representation uses about Σ(d_n − 1) dimensions, not the joint state’s, in per-factor subspaces that are approximately orthogonal, and under lossy factoring training passes through the factored form first. Shai et al. assert it from their experiments (NOTE-758, C2–C4). Two things are left out because the paper’s own results do not carry them: that the preference is an inductive bias of the architecture (C5, weak; App. D.4 says the source is open), and that the factored form is kept at convergence under strong noise (for large ε the converged representation departs from it, Fig. 12).
What the record holds toward it
- THEORY-256 (Active): the theorem, and the measurement on five Mess3 and Bloch Walk factors: about 10 effective dimensions against 242 for the joint state, linear independence and approximate orthogonality in the leading components, and under the token noise tested the factored form reached first.
- NOTE-758 also records that recurrent networks factor less sharply in four small configurations (C6, weak), which is the only evidence beyond transformers.
What would confirm or reject it
A replication of the kind the promote_when describes confirms it; a transformer that settles near the joint dimension with room to spare, or a lossy process on which training goes straight to the joint representation, rejects it. QUESTION-157 asks the same thing as a question; this entry is the answer Shai et al. propose.