nucleation

QUESTION-217: When a learning system is said to go from coarse to fine, do the order in which its representations refine one another, the order of their relevance to a task and the order in which training acquires them admit one common chain, and under what conditions on the learning operator?

6 min read

Properties6
StatusOpen
Version2
Tagslearning-theory, representation-learning, information-theory
Linelearned-structure
Refined byQUESTION-220 — Given that information-bottleneck optima at two budgets need not be nested, what condition on P(X, Y), beyond Y a function of X or X and Y both binary, guarantees that the coarser optimum is a (possibly noisy) function of the finer, X → Z₂ → Z₁, and how much is lost by forcing nesting when it fails?
BearingTHEORY-759 — A six-layer, 256-unit ReLU network trained by full-batch Adam to regress a sum of sinusoids at 5 to 50 cycles on [0, 1] fits the components in order of frequency, and keeps that order when the amplitudes rise tenfold with frequency, so in this setting the order of acquisition is set by the learner and not by the strength of the target's components, CLAIM-394 — In a stationary linear Gaussian VAR(1) that is time-reversible (AΠ = ΠAᵀ, Π the stationary covariance), every slow-feature direction is a left eigenvector of A, so any span of slow features is an exactly dynamically independent coarse-graining; off reversibility a slow feature is in general neither independent nor free of memory, THEORY-775 — With a Gaussian random-walk prior and an instantaneous unit-variance Gaussian encoder, the KL half of the evidence lower bound is the unordered slowness loss plus a constant; with a linear Gaussian encoder and decoder on whitened input, the bound is maximized on the span of the slowest SFA directions whose lag-one autocorrelation exceeds one half, with the basis inside that span unidentified, and on unwhitened input the maximizer is neither SFA's subspace nor invariant to rescaling the input, THEORY-774 — In a one-dimensional lattice hopping model with 2N+1 free hopping probabilities, all particles starting at the origin and the density after t steps observed with fixed Gaussian error, the Fisher metric is exactly t² times the distribution of the difference of two independent (t−1)-step walks, evaluated at the lag between the two hops, so it does not depend on the drift; when the walk is aperiodic with nonzero variance, its k-th largest eigenvalue is of order t^(3/2−k) for k = 0, …, 2N, so successive eigenvalues separate by a factor of order t, THEORY-758 — For a self-adjoint diffusion operator T with spectrum in [0, 1], the diffusion-wavelet levels are spectral subspaces spanned by the eigenvectors with λ^s ≥ ε for an increasing sequence of powers s, so they form an exactly nested chain of T-invariant subspaces whose differences are bands of the spectrum, about one octave each, not spatial blocks; Coifman and Maggioni's compressed algorithm, which squares the operator on each orthonormal basis, thresholds T^{2^{j−1}} at level j and not T^{2^j−1} as the paper states; and since each level is T-invariant, each is an exactly closed linear coarse-graining whose localized basis is a choice the level does not fix, THEORY-746 — Over a fixed finite set of basis functions, slow feature analysis is a generalized eigenproblem: the outputs that minimize ⟨ẏ²⟩ under zero mean, unit variance and decorrelation are the bottom generalized eigenvectors of the derivative covariance against the covariance, in order of slowness, and a stack of such stages, each of whose function spaces contains its inputs, can only keep or lower each of its first m slowness values on the training signal, and may keep them, THEORY-745 — For the reversible walk p = k/d built from a symmetric non-negative kernel on a finite connected set, the t-step diffusion distance is exactly the Euclidean distance between the coordinates λ_l^t ψ_l, and keeping only the coordinates with |λ_l|^t > δ|λ₁|^t errs in D_t² by at most δ²|λ₁|^{2t}(1/π(x) + 1/π(y)), an absolute bound and not the relative accuracy δ that Coifman and Lafon state; the truncated maps are nested as t grows, so with δ fixed the steps t order a refinement chain by eigenvalue modulus, while untruncated they are all invertible images of one another

Graph View

Why it is a question

A2 of the exchange of 2026-10-19 separated three relations on a family of representations Z_ℓ:

  • refinement: Z_a is a function of Z_b (σ(Z_a) ⊆ σ(Z_b));
  • relevance: Z_b is at least as informative about a stated target;
  • acquisition: the distinctions in Z_a become recoverable no later in training than those in Z_b, with an acquisition time T_ℓ(δ) = inf{t : R(D_{ℓ,t}, Z_ℓ) ≤ δ} for a decoder D and a risk R.

“The information ordering and the acquisition ordering must be represented separately. The first is structural. The second is dynamical.” A coarse-to- fine schedule is the case where the three admit a common monotone chain. A2: “Crucially, the default should not be to assume these orderings agree.”

What the record holds

  • THEORY-186: in a deep linear network the order is by singular value, and “Fine distinctions come first whenever the data make them stronger”. So acquisition and refinement can be reversed. It is exact only for decoupled, balanced starts and white inputs.

  • THEORY-086: in the kernel regime the order is by NTK eigenvalue, “not by degree”.

  • THEORY-481: in linear attention the order is by input variance, with alignment to the eigenbasis assumed, and at finite context the modes couple at order 1/N.

  • THEORY-199, THEORY-195 and NOTE-687: orders in noise and in sample size that run opposite to THEORY-186’s, and “should not be cited as one account”.

  • Read on 2026-10-20:

    • THEORY-745 (diffusion maps, NOTE-985): the exactly commuting case. When the levels are the operator’s own spectral projectors, truncated diffusion maps form an exact refinement chain ordered by |λ_l|, and repeated application of the operator acquires them in the same order. The two orders agree only because they come from one eigenbasis. The order also depends on the normalization α, so it is not a property of the data alone.
    • THEORY-742 (Raju, Machta and Sethna, NOTE-972): along nested coarse-grainings the Fisher metric of the parameters can only fall, so refinement and relevance to the parameters share a chain. Under an exact RG step the singular part of the metric is kept. There is no learning operator, so it says nothing about acquisition.
    • CLAIM-387: among optimal compressions at different budgets the refinement order itself can fail to be a chain.

  • Third batch, read the same day:

    • THEORY-758 (diffusion wavelets, NOTE-984): the multilevel version of the commuting case. The levels are octave bands of an operator’s spectrum, nested exactly for any operator with spectrum in [0, 1]. Spectral decay buys compression, not nesting. The paper’s compressed algorithm thresholds T^(2^(j−1)), not the T^(2^j − 1) it states.
    • THEORY-746 (slow feature analysis, NOTE-971): a third commuting case. Slow features are the bottom of a generalized eigenproblem, nested by eigenvalue. A stack of SFA stages is a refinement chain by construction, and its slowness values cannot rise from stage to stage on the training signal. Slow features are coarse only in that sense: they include squares, products and location. The paper says nothing on predictive sufficiency or learning dynamics.
    • THEORY-759 (spectral bias, Proposed, NOTE-983): a finite ReLU network trained with Adam fits sinusoids in order of frequency, even when amplitude rises tenfold with frequency. So in that setting the learner, not the target’s strength, sets the acquisition order, against THEORY-186’s order by singular value. Fourier bands are not a refinement chain, so this order has no structural partner. THEORY-761 is the paper’s proved representational bound, which is about what a network can contain, not about training.
    • Open: where the amplitude ratio flips the order from by-frequency to by-strength. The record has no formula for the product of the learner’s spectrum and the data’s outside the kernel regime.

  • Machta, Chachra, Transtrum and Sethna (2013), THEORY-774 (Active): in their hopping model, observed at a later time t, the k-th Fisher eigenvalue scales as t^{3/2−k}, so the nested effective models (creation; then drift; then diffusion; …) are ordered by relevance. Since the metric is the Gauss–Newton Hessian of the least-squares cost, gradient flow near a fit acquires the same directions in the same order, in these coordinates only (natural-gradient flow would fit all at one rate). Another commuting case, as in THEORY-745. Its Ising half (THEORY-777, Proposed) orders relevance along decimation, with no learner.

  • Schüler and Wiskott (2025), THEORY-775 (Active): slow feature analysis as a variational objective with a random-walk prior. On whitened linear input its optimum is SFA’s span of directions with lag-one autocorrelation above ½, up to rotation, so the spans nest by slowness as more latents are allowed: a refinement order of spans, not of features. On raw input it is not SFA. CLAIM-394 (demonstrated): in a reversible linear Gaussian VAR(1) every slow direction is a left eigenvector of A, so slowness selects exactly closed coarse-grainings; off reversibility it does not. Neither speaks to the order of acquisition.

What the record lacks is the condition under which a spectral order is a refinement order. A2’s candidate (“Conjecture A”, in its words “a candidate research direction, not an established theorem”): let P_ℓ project onto the observables of level ℓ. If the learning operator A nearly commutes with every P_ℓ ([A, P_ℓ] ≈ 0) and its effective eigenvalues fall as finer distinctions are added, acquisition is nearly coarse to fine, with deviations set by the cross-level coupling and the spectral gaps. For a moving operator A_t it proposed ∫‖[A_t, P_ℓ]‖ dt ≪ 1 as the measure.

What would count as an answer

  • A theorem giving sufficient conditions, for a stated model class, under which the three relations share a chain; or
  • a small family of constructions in which each pair comes apart. One is already on file: THEORY-186’s fine-first case.

A2’s first project would test it: synthetic data with a known tree, with the spectral strength assigned to each level varied independently of the tree, trained in a deep linear network, a kernel-regime network and a small nonlinear one, reporting a phase diagram of coarse-first, fine-first, simultaneous and mixed acquisition. Readings registered with this question: spectral bias (LIT-1261), slow feature analysis (LIT-1249), diffusion maps (LIT-1246) and diffusion wavelets (LIT-1256), and the information geometry of coarse-graining (LIT-1254). Whether optimal representations nest at all is QUESTION-220.

Note of 2026-10-21: two order structures

A5 of the exchange of 2026-10-20 proposed “two different mathematical order structures”: what is identifiable grows with the access regime, while optimal representations need not nest. The second half is this question’s refinement order against its relevance order, with CLAIM-387 as the witness this question already cites. A5’s monotone half is elementary: inclusion of feasible sets gives a monotone optimal value, and refinement of the fibres gives monotone identifiability. For a fixed target, the monotone half is now filed as CLAIM-409. The proposal appears once.

Backlinks