Source
Fu, Chen, Jia and Sharan (2023), LIT-535 — NeurIPS 2024. Read as NOTE-278.
The practice
When two candidate algorithms are proposed for what a network computes internally, do not compare each one’s output to the network’s. Both candidates converge to the same answer on the task, so both will fit — and LIT-535 reports exactly that: gradient descent shows high output similarity with the trained model at later layers, and it is still the wrong description.
Two measurements do separate them.
A rate. If depth is iteration, then matching layer ℓ to the
best-fitting step count k(ℓ) of a candidate gives a relation whose shape
is the diagnosis. The candidate with the model’s own convergence rate gives a
straight line; one that is exponentially slower gives an exponential curve.
Here: roughly 3 Iterative Newton iterations per middle layer between layers 3
and 9, against an exponential correspondence with gradient descent, and single
layers advancing as far as hundreds of GD steps.
A regime where the rivals must differ. Pick a setting the candidates
respond to differently on theoretical grounds, and put the model in it. Here:
conditioning, because second-order methods depend on κ logarithmically and
first-order methods polynomially. At κ(Σ) = 100 with a per-sequence random
eigenbasis, the model still matched Newton at 21 iterations while gradient
descent needed about 2,000 steps — more than the twelve layers available, and
not rescuable by a preconditioner, because the eigenbasis is resampled.
Two independent separations pointing the same way is what “the model is doing
A and not B” needs. One fit is not.
Claim the result at the strength the method supports
LIT-535’s own discipline is part of the practice: BFGS shows the same linear trend as Iterative Newton, so the paper claims a second-order method rather than a named one. A rate identifies an algorithm’s class. Naming a specific member requires evidence the rate does not carry, and the record files the paper’s restraint as part of what it is recommending.
Why it is Proposed
One source and one problem class. Both measurements need a task whose optimal algorithm is known in closed form and whose candidate solvers have established rates — linear regression obliges, most things do not. Whether the method extends to a setting without an answer key is untested.
And it has only been run on purpose-trained models, which is the gap
SOTA-323 names. That is what promote_when asks for: the same
measurement on a model pretrained on a general objective. It is also the
experiment this record could not find anybody having done, on either side of
the dispute.
Relation to the other half
SOTA-323 says to check what family of models your evidence comes from. This says how to tell two mechanisms apart once you have the right models. They came out of different papers on the same dispute and neither subsumes the other: LIT-535 applies this method inside the purpose-trained setting SOTA-323 warns about, and reaches a sound conclusion about that setting.