anthology-of-the-sota

LIT-140: mHC: Manifold-Constrained Hyper-Connections

2 min read

Properties8
StatusActive
Tagsmodel-architecture
Compared againstLIT-152 — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability, LIT-513 — mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
CorrectsLIT-141 — Hyper-Connections
Corrected byLIT-151 — oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions, LIT-181 — Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections
SupersedesLIT-141 — Hyper-Connections
Cited as “introduced by” bySOTA-136 — Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections)
Cited as “source” bySOTA-136 — Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections), SOTA-169 — Widen the residual stream into several streams and constrain the mixing between them

Graph View

Xie et al., DeepSeek-AI (2025; revised January 2026) — ARXIV-2512.24880

Key takeaways

  • Hyper-Connections (LIT-141) expand the residual stream to several parallel streams and let the network learn how they mix. The paper’s diagnosis is that the free mixing gives up the identity-mapping property a residual connection has by construction, and that this is what makes HC unstable and hard to scale.
  • The fix projects the residual-mixing matrix onto a manifold that restores identity mapping: the matrix that carries the streams forward is made doubly stochastic — non-negative, rows and columns summing to one — by Sinkhorn-Knopp iterations, which bounds its spectral norm, and the pre- and post-mixing maps are kept non-negative.
  • Infrastructure work keeps the wider stream affordable: kernel fusion, recomputation and communication overlap bring the cost of four streams down to a reported 6.7% additional training time at expansion rate n = 4, with the quality gains of HC retained. That figure is the system-level result at 27B; the scaling study that shows the advantage holding spans 3B, 9B and 27B, with a separate 3B run on 1T tokens for token scaling.

Standing in the anthology

The source of SOTA-136 and the successor of LIT-141, which it retires in this record: the reason to prefer it is the reason HC could not be scaled. DeepSeek-V4 (LIT-139) is the production instance, and the largest model in the record to replace the residual at all. LIT-152 is the one place this and the record’s other two replacements for the residual have been run in a single harness: they come out level, separated by inference cost rather than quality. The constraint this paper chose is separately argued against in LIT-151, which shows the doubly stochastic set bounds the mixing from above but not below.

mHC-lite (LIT-513) keeps this paper’s doubly-stochastic constraint and measures what the 20 Sinkhorn-Knopp iterations used here actually reach. The column sums of one mixing matrix can be off by up to 100%, and those of the layer-wise product by up to 220% in a 24-layer network. Building the matrix exactly, as a convex combination of permutation matrices, it reports matching or exceeding mHC with a smaller and steadier gradient norm. That comparison is on nanoGPT models up to 0.36B for about 1.3B tokens, one seed, with no stream statistic.

Backlinks