nucleation

QUESTION-305: From small random weights not aligned with the data's eigenbasis, does a linear-attention layer trained on in-context regression come into alignment with that basis before or while its modes are learned, so that its learning is staged by the input spectrum without the alignment being assumed?

1 min read

Properties5
StatusOpen
Tagslearning-theory, representation-learning
Linelearned-structure
RefinesQUESTION-039 — What sets the order and the moment, in training time or in examples, at which a network acquires a structure, and is acquisition one kind of event or several?
Cited as “bears on” byTHEORY-481 — Under gradient flow averaged over tasks and data, a linear-attention layer trained on in-context regression learns as a two-layer linear network from an aligned start does: when tasks share the input covariance's eigenbasis and the weights start and stay aligned with it, then in the long-context limit each input mode is learned on its own logistic curve whose timescale falls with the square of its variance, so learning is staged by the input spectrum, and the layer converges to the estimator preconditioned by the inverse input covariance; at finite context length the modes decouple exactly only when inputs are isotropic and the modes start equal, and otherwise couple at order 1/N

Graph View

Why it is a question

THEORY-481’s derivation assumes the weights are aligned with the input covariance’s eigenbasis at initialisation and throughout (NOTE-658, the paper’s A.11.2), and its simulations start aligned. Whether training from a random start reaches that alignment, and so whether the staged learning holds in general, is not shown.

What would answer it

An analysis or simulation showing that from small random initial weights not aligned with the data’s eigenbasis, a linear-attention layer trained on in-context regression aligns with that basis before or while its modes are learned, so that the per-mode logistic curves and their ordering by input variance hold without the alignment being assumed; or the same staged learning measured in a trained layer whose tasks do not share the input covariance’s eigenbasis. Timing coincidences of spectral measures in non-linear transformers cannot settle it, and neither can more simulations from aligned starts.

Backlinks