Source
Zhu et al. (2024), LIT-141 — Hyper-Connections.
Replace the single residual stream with n parallel streams (n = 2 in most of the paper’s runs) and learn the mixing between them, so that the network can tune connection strength across depth and exchange information laterally between streams. The motivation is the seesaw between vanishing gradients and representation collapse that fixed residual variants cannot escape; the paper shows consistent pretraining gains on dense and MoE models.
The recommendation originates in LIT-141, which proposed hyper-connections as the escape from that seesaw: n parallel streams with learnable, possibly input-dependent mixing, so the network can adjust connection strength between depths and effectively rearrange layers, with consistent gains over standard residual connections in its dense and MoE pretraining runs.
This is the free-mixing branch of SOTA-169, the trunk that states what the whole line agrees on — widen the residual stream into several streams. The width is the part of this practice that survived; the free mixing is the part the trunk’s constraint was written to rule out.
Why this is superseded
The free mixing discards the identity mapping a residual guarantees, and the successor’s authors report that this is what made hyper-connections unstable and hard to scale. Constraining the mixing matrices (SOTA-136, LIT-140) keeps the width and the gains. Kept here so that the chain can be read from the residual forward; the claim above was right about the seesaw and wrong about what could be left free.