anthology-of-the-sota

SOTA-010: skip connections promote training stability by smoothing out the loss landscape

2 min read

Properties5
StatusSuperseded — by THEORY-011 — Skip connections make a deep network trainable by smoothing the loss surface, not by making it more expressive; Not a recommendation, and never was: it states what is true rather than what to do, which is a THEORY under ADR-031 and was one of the eight that decision named and declined to move. The claim is unchanged and is believed — it now lives at THEORY-011, with the figure's limits and the five practices it underwrites. `Superseded` rather than `Rejected` because `Rejected` on a practice means *do not do this*, and nobody should read this code as advice against skip connections. ADR-034 is the rule.
Version2
Tagstraining-optimization, model-stability
SourceLIT-014 — Visualizing the Loss Landscape of Neural Nets
Introduced byLIT-014 — Visualizing the Loss Landscape of Neural Nets

Graph View

Li et al. (2017), LIT-014 — ARXIV-1712.09913.

Moved

The claim is at THEORY-011. It is true, it is believed, and it is not a recommendation: it says why deep networks became trainable, which is what the THEORY scheme holds. ADR-031 listed this document as one of eight practices whose titles state a behaviour rather than an instruction and left them in place pending a decision about what a vacated code should say; ADR-034 is that decision and this is its first application.

The claim originates in LIT-014. Li et al. compared ResNet-56 with ResNet-56-noshort on CIFAR-scale data under filter-normalised loss-surface plots. As networks get deep, the landscape goes from nearly convex to chaotic, and that transition comes with worse generalisation and, eventually, a loss of trainability. Skip connections prevent it and “promote flat minimizers”, which the paper offers as why they are needed at extreme depth.

There is no practice under it. The instruction the claim would imply — use skip connections — is trunk, and DP-007 keeps the agreed trunk unfiled. That is why this is a move rather than a split.

Do not read Superseded as a judgement on skip connections. On a practice that status means a better recommendation exists; here it means the document is in the wrong scheme and the record says where the right one is.

Backlinks