Source
Noroozizadeh, Nagarajan, Rosenfeld & Kumar (2025; ICML 2026), LIT-995, read in NOTE-800: §§2–4, Observations 1–4, Refutations 3a–3d, App. C.3.1, D.3, E.2 and F.
What was actually shown
Deep models. A 12-layer transformer and a Mamba model memorize a path-star graph’s edges and, trained on only the first hop of 75% of paths, predict the first hop of unseen paths, up to 100% against chance 1/d, on graphs up to 5×10⁴ nodes (Obs. 1). Trained on edges alone, their leaf and first-node embeddings cluster by path (Refutation 3a). On four tiny graphs, the same one-hidden-layer architectures fit the edges associatively when embeddings are frozen, in about 2 steps for wide models, while geometry takes about 100 (Refutations 3b–3c). For sparse graphs the bit and norm costs of the two stores differ by a factor of at most about 2 (Refutation 3d, App. E.2). Under the default small initialization the geometry forms without weight decay or dropout (App. C.3.1).
The two-layer model. With weight-tied embedding width 100 on graphs of at most 16 nodes, unregularized full-softmax cross-entropy on 1-hop edges, the embedding’s projection concentrates on the top Fiedler-like eigenvectors of −L, L = (I − D⁻¹A) + (I − D⁻¹A)ᵀ, more sharply than the eigenvalues do: two directions for the cycle and grid (Fig. 7). In the dynamics V̇ = ηC(t)V, those eigenvectors enter C’s null space while the others decay (Fig. 8, Obs. 4). App. F gives an informal account: C starts near −L, its eigenvalues rise toward zero top-first, and lower components of V shrink.
What could have come out otherwise: with a lookup expressible, as cheap and reached first, training could have stayed associative. It did not, under the default set-up.
What this does not say
- Not why deep models prefer geometry. The paper’s title says “it is unclear why”, and its Open Questions 1 and 3 say so. The spectral account is for the two-layer model only, where associative storage is architecturally excluded.
- Not a proof. App. F’s propositions assume equal norms, equal degrees, near-orthogonal initialization and an empirical Observation 9 taken “without even an intuitive proof”. The authors call all their arguments “empirical and informal” (§6).
- Not independent of optimization settings. Unit-normal initialization can give associative storage; higher learning rate, weight decay and dropout strengthen geometry (App. C.3.1). The claim is that no such pressure is required, not that none matters.
- Not regularization-free at scale. The large path-star runs used AdamW with weight decay 0.01; the no-regularization evidence is from the tiny graphs.
- Not natural language. Symbolic graphs only; reverse edges were needed for the large path-star result (App. D.1).
- Not about convergence. With the asymmetric normalization the degenerate top eigenvalue is slightly positive, so the authors expect collapse under much longer training (Remark 6).
Connections
- QUESTION-060. An answer in part, naming a third determinant besides objective and cheap representability: the dynamics of training. Here the local objective is satisfied equally by both stores, and the architecture can represent both at similar cost, yet the learned variables are the graph’s leading spectral coordinates.
- THEORY-229. The same kind of mechanism, eigenvectors filtered by gradient dynamics, for a symmetrized Barlow Twins loss from small initialization. This account is for cross-entropy from near-orthogonal initialization and is informal.
- THEORY-256. A second case in which training reaches a low-dimensional structured representation that the objective alone does not select.
- QUESTION-052. A side connection: the two-layer model’s learned signature is the top eigenspace of a symmetrized random-walk operator, and untied models pick up singular directions instead (§4.4).