Guo et al., Hong Kong University of Science and Technology and Baidu (2026) — ARXIV-2609.02672
Key takeaways
- The diagnosis is a gap in mHC’s (LIT-140) argument rather than in its results. Restricting the mixing matrix to the doubly stochastic matrices caps the factor by which a layer can rescale the residual streams at one, which is what stops the compounding that destabilises plain hyper-connections — but nothing bounds that factor from below.
- The paper proves what the slack costs: inside the doubly stochastic set the mixing step can reduce the norm of the streams only by shrinking the differences between them, leaving their mean untouched. The reduction accumulates over layers, so the streams grow more alike and the diversity the extra width was bought for is spent with depth.
- oHC constrains the matrix to the rotation group SO(n) instead, which can neither amplify nor attenuate in any direction. At the n = 4 of current HC models the group has a closed form as a pair of unit quaternions: no added parameters, no iterative projection — a fixed pattern of signed additions — and construction faster than mHC’s Sinkhorn-Knopp.
- Four arms differing only in the mixing set — single-stream residual, HC, mHC, and iHC, which pins the matrix to the identity — put oHC ahead of all of them on bits-per-byte across the downstream suite.
Standing in the anthology
Filed for the argument, not yet for a practice. It is the first result in the record that reads against an entry the record already holds: the
mechanism it identifies is a property of exactly the constraint SOTA-136 recommends. It does not retire that practice. The evidence is one model size — a 3.9B-A0.4B latent-MoE on 73B tokens — from one group, and the paper says so itself, naming scale as the question it would investigate next; mHC meanwhile has a production deployment at 1.6T parameters (LIT-139) that oHC has nothing comparable to. What it changes is the
standing of SOTA-136’s reason: the doubly-stochastic constraint is now one known-imperfect answer among several rather than the settled one, and LIT-152 is a second group reaching a compatible conclusion by a different route.