nucleation

LIT-940: Optimal scaling laws in learning hierarchical multi-index models

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-805) in full; worth reading as an exact theory of when a feature becomes learnable. Take Gaussian inputs and a target Σ a_k g_k(⟨w_k, x⟩) with even links and a_k ~ k^(−γ). Direction k becomes weakly recoverable at n = Θ(a_k^(−2) d), so features emerge one by one as spikes leaving a random-matrix bulk. A target-agnostic spectral estimator attains the Bayes-optimal rates, Θ(α^(−1+1/(2γ))) and then Θ(m⋆/α). Read it knowing that the cascade is in the number of samples, not in training time. The link to gradient descent is a remark citing other work. The network theorem is for a spectral first layer plus a ridge readout on split data. Held as THEORY-283.
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-805 — Defilippis et al. — Optimal scaling laws in learning hierarchical multi-index models
Cited as “source” byTHEORY-283 — For a hierarchical multi-index target with isotropic Gaussian inputs and even, separable links, direction k becomes weakly recoverable by the Bayes-optimal estimator at a sample count proportional to its squared weight to the power minus one, so features emerge one at a time through sharp spectral transitions in the number of samples; a target-agnostic spectral estimator attains the thresholds and the optimal weighted error, and a two-layer network whose first layer is that estimator, with a ridge readout, attains them in excess risk

Graph View

Leonardo Defilippis, Florent Krzakala, Bruno Loureiro and Antoine Maillard (2026), arXiv:2602.05846.

Key takeaways

From the reading, NOTE-805.

  • Thresholds set by the target’s weights. With x ~ N(0, I/d), n = αd, and f⋆ = Σ_k a_k g_k(⟨w_k, x⟩), take orthogonal w_k, even links of generative exponent two, and a_k ~ k^(−γ). Then the Bayes estimator first correlates with w_k at α_k = Θ(a_k^(−2)) = Θ(k^(2γ)) for γ > 1/2 (Thm. 3.2).
  • Optimal rates.
    • For γ > 1/2, the weighted subspace error is Θ(α^(−1+1/(2γ))) while α ≪ m⋆^(2γ), and Θ(m⋆/α) after.
    • For γ < 1/2, it is Θ(1), then Θ(m⋆/α).
    • The error splits into the error on the k_α ≍ α^(1/(2γ)) directions already out, plus the weight of the rest.
  • A spectral method attains them. The top eigenvectors of Σ T(y_i)x_ix_iᵀ, for any bounded preprocessing T and cut at an eigengap, match both the rates and the thresholds (Thm. 3.6). Each spike correlates with one direction, because the even, separable links make E[zzᵀ − I | y] diagonal.
  • A network attains them in excess risk. A two-layer net whose first layer is that estimator times a random matrix, with a ridge readout fit on a second half of the data, reaches Θ(MMSE) + O(n^(−1/2)) for polynomial links (Thm. 3.8).
  • The cascade is in samples. No training trajectory is analysed. Early gradient descent on the first layer from a small sphere is a power iteration on that matrix only by Remark 3.5, which cites Bonnaire et al. and Zhang et al. (2025).

Standing in the record

Filed on 2026-10-12 from the owner’s papers-feed for QUESTION-039, and read on 2026-10-14 in NOTE-805. Its theorems are held as THEORY-283, a partial answer to QUESTION-039. In this model class the order of acquisition is the order of the target’s weights, and the moment is a sample count Θ(a_k^(−2) d). Each acquisition is one kind of event: a spike leaving a bulk.

  • Beside THEORY-229. The same spectral picture appears on another axis: training time on fixed data there, sample size here. The power-iteration remark is the only bridge.
  • Beside THEORY-204. The same signal-meets-noise form, n·s² ~ 1, appears there as an assumption for text.

Carries the anthology-candidate flag. The anthology was checked when it was filed, and it does not hold the work.

Backlinks