nucleation

QUESTION-166: Does the depth law of deep linear networks hold from starts off the decoupled set: learning time, counted in iterations at the stable step size, bounded in depth when every mode's initial end-to-end gain is of order one, and growing as a power of 1/ε when it is ε, for random orthogonal or small random weights not aligned with the task's singular vectors?

2 min read

Properties5
StatusOpen
Tagslearning-theory
Linelearned-structure
RefinesQUESTION-039 — What sets the order and the moment, in training time or in examples, at which a network acquires a structure, and is acquisition one kind of event or several?
Answered byTHEORY-792 — In a deep linear network trained by gradient descent from generic initial weights, not only from decoupled ones, depth slows learning through the initial end-to-end strength of each mode, so that random orthogonal weights, whose product is an isometry, learn in a number of iterations that stays bounded as depth grows, as Saxe, McClelland and Ganguli argue from simulation

Graph View

Why it is a question

THEORY-187, narrowed on 2026-10-19, holds the depth law of Saxe, McClelland and Ganguli (LIT-864) for the start the exact theory covers: whitened inputs and initial weights on the decoupled set, balanced across layers. There each mode is a scalar, and the plateau and the iteration count follow by integration. Until that date THEORY-187’s promote_when asked whether the same law holds from starts off the decoupled set. At the owner’s direction, from the promotion audit’s batch 027 (ADR-047), the THEORY was narrowed and promoted on the exact case, and the condition moved here unchanged.

It is open because off the decoupled set the modes couple, the scalar reduction fails, and the paper’s account of random orthogonal starts is an argument from the isometry of the product of layers, with one MNIST experiment (Fig. 6A, NOTE-667). It refines QUESTION-039’s question of what sets the moment at which a network acquires a structure.

What would answer it

The condition formerly on THEORY-187, unchanged: a derivation, or a controlled measurement across depths, showing that deep linear networks started off the decoupled set behave the same way, with learning time, counted in iterations at the stable step size, bounded in depth when the initial end-to-end gain on every mode is of order one, and growing as a power of 1/ε when it is ε. That means random orthogonal or random small weights, not aligned with the task’s singular vectors. More runs from decoupled starts cannot settle it, since there the result is already exact.

Backlinks