anthology-of-the-sota

SOTA-021: Monitor loss landscape smoothness during training

1 min read

Properties4
StatusRejected — states no threshold and no response, and the smoothing it refers to is a mechanism from a paper the record had conflated with this one. Measuring landscape smoothness during training is not something the practice's source does or recommends; the finding it rests on is Santurkar et al.'s explanation of why batch normalization works, which is a fact about the technique rather than an instruction
Tagsmodel-stability
SourceLIT-015 — Understanding Batch Normalization
Introduced byLIT-015 — Understanding Batch Normalization

Graph View

Source

Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375.

Why this is retired rather than rewritten

Two problems, and either alone would be enough.

It is not actionable. “Monitor loss landscape smoothness during training” names no signal you could plot, no threshold, and no response — the same gap

that retired SOTA-073 and SOTA-074. Landscape smoothness is not a quantity training loops compute; the nearest practical proxy is the Hessian eigenvalue ratio in SOTA-011, which is expensive and periodic rather than monitored.

Its mechanism is another paper’s. Smoothing, predictive gradients and Lipschitzness are Santurkar, Tsipras, Ilyas and Madry — How Does Batch Normalization Help Optimization?, now filed in the record — and they were attributed to LIT-015 because that note carried the wrong author and takeaways. The finding is real and it is an explanation of why batch normalization works, not an instruction to anyone running a job.

The explanation survives where it belongs, in the literature note. What is retired is the attempt to turn it into a practice.

Backlinks