Bachlechner et al. (2020) — ARXIV-2003.04887
Key takeaways
- The whole intervention is one parameter per layer: multiply the residual branch by a scalar initialized to zero, so at initialization every block is exactly the identity and signal propagates through an arbitrarily deep stack without attenuation or amplification.
- That satisfies initial dynamical isometry — the condition Pennington et al. identified as what makes deep training work — and the paper’s claim is that this simplest possible change outperforms the more complex architectural and initialization schemes built to achieve the same thing.
- The gate is learned, so the network decides how much of each branch to admit as training proceeds. Zero initialization is a starting point, not a constraint.
- Depths reached: thousands of fully connected layers on CIFAR-10, and 120-layer Transformers trained without the warmup and normalization scaffolding that depth usually requires. At 12 layers it converges 56% faster on enwiki8.
Standing in the anthology
The source of SOTA-051 (initialize the final layer of a residual branch near zero) and SOTA-053 (gated architectures want their gate started closed). Both arrived under a fabricated citation; the advice is real and this is where it comes from.
It sits at one end of a line the record carries in full. ReZero starts the branch at zero and learns the scale; DeepNet (LIT-084) instead derives a fixed depth-dependent constant for it (SOTA-052); Learnable Multipliers (LIT-121) makes the scale of every matrix learned rather than set by the
LR/WD equilibrium (SOTA-122). The question they answer in common is who picks a layer’s scale — the initializer, the schedule, or the data.
Two controlled re-runs test the claim, and they agree once it is clear what each kept. Narang et al. (LIT-711) reimplemented ReZero in a 223M T5-style encoder-decoder with every other hyperparameter held fixed, and it lost to the pre-norm baseline on early pre-training loss: 2.262 with ReZero replacing LayerNorm and 2.223 with both, against 2.182. That is after it was moved from Adafactor to Adam because it trained so badly under the shared optimizer. Ndubuaku et al. (LIT-708) kept normalization and swapped a ReZero residual into a 20-layer attention-only stack in place of its sigmoid-gated one: +0.0032 nats, within a few noise floors, with scalar gates of either kind neutral from 8 to 48 layers. So at those scales the zero-initialized scalar does not replace normalization, and alongside it adds nothing measurable. Neither tested the 120-layer depths this paper reports.