LIT-639: ResiDual: Transformer with Dual Residual Connections

Xie, Zhou, Gu, He, Lin, Liu, Qin and colleagues (2023) — ARXIV-2304.14802

Key takeaways

• The collapse argument gets a rate, which is the reason to hold this. Theorem 3.3: assuming block outputs xᶠ_k ~ N(0, σ²I) independently, the change in the normalised hidden state between consecutive Pre-LN blocks is xˡⁿ_{k+1} − xˡⁿ_k ~ N(0, ω_k² I) with

  ω_k² = 2 / (√k (√(k−1) + √k))
  

  Corollary 3.4: E[|(xˡⁿ_{k+1} − xˡⁿ_k)ᵢ|] ~ O(1/√k). Corollary 3.5: adding a block to an N−1 block Pre-LN transformer changes the output by O(1/√N) per coordinate. “Adding [an] extra layer in the deep Pre-LN Transformer has little impact on the output.”

• The observation is not theirs and they say so. “The issue with the representation capability of Pre-LN was initially observed by Liu et al. 2020” — who showed the ratio √Var[xᶠ_k] / √Var[xᵃ_k + xᶠ_k] shrinks for higher blocks. This paper’s contribution is the convergence rate, via “a novel analysis approach that directly examines the distribution of hidden state changes.” The record holds neither Liu et al. nor this, and has been making the argument in prose.

• The two defects are symmetric and both are about where the normalisation sits. In Post-LN, xᶠ_k is normalised N−k times and so are its gradients, which is why gradient norm decays exponentially from deep layers to shallow ones — Xiong et al.’s result, LIT-114, restated here as the motivation for the other half. In Pre-LN, y = LN(Σₖ xᶠ_k), so each block output is normalised exactly once and nothing is blocked in either direction — and the price is that the sum stops moving.

• What Pre-LN’s no-warmup claim costs, measured. IWSLT BLEU:

  method	warm-up	E6D6	E12D12
  Post-LN	yes	35.37	fail
  Post-LN	no	fail	fail
  Pre-LN	yes	35.12	35.18
  Pre-LN	no	32.28	31.82
  ResiDual	yes	35.63	36.09
  ResiDual	no	35.76	35.57

  Pre-LN without warm-up loses 2.84 BLEU at E6D6 and 3.36 at E12D12. It trains, which is Xiong et al.’s claim, and it is meaningfully worse, which is not in Xiong et al.

• Their prose is looser than their own table. The text says “Pre-LN and our method can train effectively without it” — with a three-point drop in the row above. Anyone citing the sentence rather than the table carries Xiong et al.’s claim forward unqualified. The ResiDual rows are the ones that earn that phrasing: 35.76 without warm-up against 35.63 with it, at E6D6.

• Post-LN does not merely need warm-up, it stops working. E12D12 fails with warm-up. The depth at which the two architectures diverge is lower than the usual framing suggests.

• The method, for completeness. Two residual streams — one Pre-LN-like to avoid gradient vanishing, one Post-LN-like to avoid collapse — summed at the output. Filed here for the analysis rather than the architecture.

Standing in the anthology

Unit 4 of #326, reversed from #290’s D-variant declines. The paper is an architecture variant, so the decline was not a heading leak this time — it was the rule itself: the variant is the contribution and the analysis is what the record needed.

SOTA-032 recommends Pre-LN, is Active, and already carries the collapse argument in its own prose: “deep pre-norm models can see later blocks contributing proportionally less — the representation collapse argument.” With no citation. Nineteen files in this record discuss Pre-LN and ten discuss Post-LN; the cost of the recommended choice was stated everywhere and sourced nowhere.

Two things follow, both applied in this contribution. The argument gets a source and a rate (THEORY-096). And SOTA-032’s stated headline consequence — that Pre-LN makes warm-up removable — gets the measurement it did not have, which does not overturn it and does put a number on it.

extends: LIT-114 — it builds directly on Xiong et al.’s gradient result, restating it as the Post-LN half of a two-sided problem.

What it takes from LIT-114 is the placement argument and its Pre-LN corollary: Post-LN’s gradient problem comes from normalising between residual blocks, and moving the normalisation inside the block lets training proceed without warm-up. What it adds is the other side’s cost, derived as the O(1/√k) collapse rate, and a measurement of the no-warm-up claim LIT-114 made — Pre-LN without warm-up trains, at 32.28 against 35.12 BLEU at E6D6.