anthology-of-the-sota

LIT-141: Hyper-Connections

2 min read

Properties6
StatusSuperseded — by LIT-140 — mHC: Manifold-Constrained Hyper-Connections; mHC restores the identity mapping that free hyper-connection mixing gave up, and scales where this did not
Tagsmodel-architecture
Compared againstLIT-513 — mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
Corrected byLIT-140 — mHC: Manifold-Constrained Hyper-Connections
Cited as “introduced by” bySOTA-137 — Widen the residual stream into several streams with freely learned mixing (hyper-connections), SOTA-169 — Widen the residual stream into several streams and constrain the mixing between them
Cited as “source” bySOTA-137 — Widen the residual stream into several streams with freely learned mixing (hyper-connections), SOTA-169 — Widen the residual stream into several streams and constrain the mixing between them

Graph View

Zhu et al., ByteDance (2024; ICLR 2025) — ARXIV-2409.19606

Key takeaways

  • The residual connection’s variants trade one failure for another: too little skip and gradients vanish, too much and representations collapse into sameness across depth — the paper’s seesaw. Hyper-connections widen the residual into n parallel streams (n = 2 in most experiments) with learnable, possibly input-dependent, mixing between them.
  • The learned mixing lets the network adjust connection strength between features at different depths and effectively rearrange layers; streams exchange information laterally as well as vertically.
  • Pretraining experiments on dense and sparse (MoE) language models show consistent improvements over standard residual connections.

Standing in the anthology

Superseded by LIT-140. The successor’s premise is that the free mixing here discards the identity mapping a residual guarantees, which is what made HC unstable and hard to scale; constraining the mixing keeps the width and restores the property. Kept because what it fixed is the content: a reader who meets mHC needs to know what free mixing cost. The

practice drawn from this paper (SOTA-137) is retired the same way.

mHC-lite (LIT-513) ran this method as a baseline beside mHC and its own exact doubly-stochastic construction, on nanoGPT models up to 0.36B with four streams. Both constrained variants sit far below unconstrained hyper-connections in gradient norm, which is the instability measurement the successor’s diagnosis predicts. It is not a scale result.

Backlinks