Garbers and Oh (2026) — ARXIV-2607.09204. Read as NOTE-262.
Key takeaways
- Spectral concentration has a shared depth profile. Across eleven checkpoints — English, Russian, Vietnamese, Chinese, Portuguese, Japanese, Turkish, plus poem and story models — effective-rank entropy drops in the final few blocks after per-model normalization.
- It is strongest where the model writes to the residual stream.
W_OandW_downshow the sharpest entropy drop, and their Frobenius norms grow almost linearly with depth: more total weight mass, carried by fewer directions. - The instrument is the right one for a magnitude claim. Entropy
effective rank,
exp(−Σ pᵢ log pᵢ)on the trace-normalized spectrum, measures how many directions carry substantial mass — which is what a low-rank branch removes. - Imitating the profile at initialization does not work. Initializers matched to pretrained component-wise magnitudes and singular-value shapes visibly change the structural spectra and produce no performance advantage; reusing the actual weights stays competitive.
Standing in the anthology
It is the breadth datapoint for THEORY-059’s question. The other measurements in this cluster are deep on one or two model families. This one holds the architecture fixed and varies size, language, tokenizer and corpus, which is what tells you the spectral profile is a property of trained transformers rather than of one recipe or one language.
The negative half is worth as much as the positive half. A recurring spectral profile that cannot be transplanted is evidence that the profile is a consequence of training rather than a configuration that causes good training — which is the distinction DP-005 exists to keep the record honest about, and the authors draw it themselves.