Pascanu, Mikolov and Bengio (2012) — ARXIV-1211.5063
Key takeaways
- The vanishing and exploding gradient problems are examined from three angles at once — analytic, geometric, and dynamical-systems — and the geometric reading is the one that pays: an exploding gradient is what a cliff in the error surface looks like locally, so a step sized for the smooth region overshoots catastrophically at the wall.
- The remedy that follows is gradient norm clipping: when the gradient’s norm exceeds a threshold, rescale it to that threshold and keep its direction. It is a step-size correction, not a change to what is being optimized, which is why it costs nothing and does not bias the solution away from the cliff.
- A separate soft constraint is proposed for the vanishing direction; it is the clipping half that the field kept.
- Framed for recurrent networks, where depth-in-time makes the pathology acute, but the argument depends only on the error surface having cliffs — which is why the technique transferred to feedforward and transformer training unchanged.
Standing in the anthology
The source of SOTA-035. It arrived in the record under a fabricated citation and is filed properly here; the practice itself was never in doubt, since every large training run in the corpus clips. Its descendants are in the record too: SOTA-071 makes the threshold dynamic, and SOTA-131’s QK-Clip is the same move — rescale rather than let a quantity run away — applied to attention logits through the weights instead of to the gradient.