Source
Li et al. (2017), LIT-014 — ARXIV-1712.09913.
What the source actually computes
Figure 7 of LIT-014 maps |λ_min/λ_max| — smallest over largest — at
each point of a filter-normalized loss surface. The quantity is interesting
because λ_min is negative: the ratio measures how much negative
curvature there is relative to positive, so it is a non-convexity measure,
not a conditioning one. Blue means near-convex; yellow means significant
negative curvature. For ResNet-56 the negative eigenvalues stay under 1% of
the positive curvatures across a large region — which is the quantitative
form of “skip connections keep the landscape nearly convex”
(THEORY-011).
The plot only means anything under filter normalization, because a network can be rescaled without changing its function and unnormalised curvature moves with the rescaling. That condition travels with this practice.
The condition number is a different diagnostic, and is not from here
This practice’s body used to describe the ratio the other way up — λ_max
over λ_min, the condition number of the local quadratic approximation — and
reason from it: a large ratio means a narrow valley, gradient descent
oscillates across the steep direction, the largest stable learning rate is set
by the sharpest direction. That is all true, and it connects to why adaptive
methods help (SOTA-001) and why the usable learning rate moves when
normalisation is added (SOTA-020).
It is also not what LIT-014 computes, and the two answer different
questions: the condition number asks how hard is this to optimise, while
|λ_min/λ_max| asks is this even locally convex. Stated the wrong way up
the practice reads as the first while citing the paper that did the second.
The conditioning material is kept because it is worth knowing and is signposted as unsourced, which is the honest treatment. If the record wants it as a recommendation it needs a paper that makes it.
The cost is why this is a diagnostic and not a monitor
The full Hessian is intractable at any interesting size. What is computed in practice is a few extreme eigenvalues by Lanczos or power iteration on Hessian-vector products. Each product costs several forward-backward passes, not one: LIT-746 measured it at 4.5–5.5 gradients of time and 3–4.5× a gradient’s memory in PyTorch, on ResNet-50 and a 124M-parameter nanoGPT, consistent with the 4–5× other PyTorch measurements report; JAX with forward mode and compilation is cited at 2–4×. So a useful estimate, tens of products, is on the order of a hundred extra passes, run periodically and not per step.
Hence “over training” in the title should be read as at intervals, and the practice is a study one runs deliberately rather than a signal on a dashboard.
That is the distinction SOTA-021 failed to make and was retired for; this one survives because the quantity is at least computable and the paper actually uses it.