Source
Xie et al. (2025), LIT-140 — mHC.
Keep the wider residual stream of hyper-connections (SOTA-137) but constrain how the streams mix: the matrix that carries them from one layer to the next is projected onto the doubly-stochastic manifold with Sinkhorn-Knopp iterations — entries non-negative, rows and columns summing to one — which bounds its spectral norm and restores the identity-mapping property that free mixing gave up. The pre- and post-mixing maps stay non-negative. The paper reports the quality gains of hyper-connections at 6.7% training overhead, and DeepSeek-V4 (LIT-139) ships it at 1.6T.
The constraint is LIT-140’s proposal, made on its diagnosis that free mixing is what made hyper-connections unstable and hard to scale. Its own numbers are 6.7% additional training time for four streams (n = 4) after kernel fusion, recomputation and communication overlap. That is one figure, stated for its in-house large-scale training, whose system-level results are on the 27B model; no overhead is reported for 3B or 9B. The advantage over the residual holds across the 3B, 9B and 27B scaling study and a separate 3B run on 1T tokens.
The V4 report was read in full for #18, because half this practice’s promotion condition asks for “a production report at depth whose streams stay distinct” and V4 is the production report at depth. It does not answer the objection. Its case for the constraint is numerical: projecting onto the doubly-stochastic manifold bounds the spectral norm at 1, so the residual transformation is non-expansive and both passes are more stable — which is the property LIT-151 grants and then argues cuts the other way, since a mapping that can only contract is a mapping that can only erode what distinguishes the streams. V4 measures no stream statistic, and the words homogenize, diversity and distinct do not appear in it. So the condition stands unmet by the one report that looked most likely to meet it — which is a thing to say out loud, because a reader who knows V4 ships mHC at 1.6T will otherwise assume the question was settled by shipping.
Construct the constraint, do not approximate it
Yang (2026), LIT-513 — read as NOTE-258 — shows the
Sinkhorn-Knopp projection does not arrive. Across SK inputs measured during
training, about 27.9% have relative range 1/ν ≥ 10¹³, where 20
iterations do not converge; a single residual matrix’s column sum can be off
by 100%, and the column sums of the layer-wise product ∏_l H^res_l by
220% in a 24-layer network.
The replacement follows from Birkhoff-von Neumann: every doubly stochastic matrix is a convex combination of permutation matrices, so parameterize the convex weights with a softmax and the matrix is doubly stochastic by construction. No iteration, no approximation gap, and no fused CUDA kernel — native matrix operations, at throughput matching a naive implementation.
So the instruction inside this branch is: if you take the constraint, build it rather than approximate it. What that does not settle is whether the constraint is the right one, which is the dispute below.
And it is the sharpest missed opportunity in that dispute. Under Sinkhorn-Knopp mHC the matrices are not actually doubly stochastic, so LIT-151’s objection has an escape hatch: whatever keeps the streams distinct might be surviving through the approximation gap. Exact construction closes it. That makes LIT-513 the cleanest available test of whether the doubly-stochastic set homogenizes the streams — the objection predicts it should homogenize more — and it reports no stream statistic at all. One histogram, on a model already trained, with code already public.
Why Proposed: the condition this was filed under has been met, and is not being applied. That is worth explaining rather than quietly leaving the status alone.
The condition was “promote on an independent result”. LIT-152 is one: a different laboratory ran mHC in its own harness and found it comparable to its own design. By the letter of the rule this should now be Active.
It is not, because two other independent results arrived at the same time and both attack the specific thing this practice recommends — the doubly-stochastic constraint. LIT-151 proves the Birkhoff polytope is bounded above but not below, so the mixing can only shrink what distinguishes the streams and they homogenize with depth. LIT-181 names three further defects: identity degeneration, an expressivity bottleneck from non-negativity forbidding subtractive interactions, and unstable Sinkhorn projection. Different arguments, opposite remedies, same verdict on the constraint.
So the honest position is that the practice’s claim has independent support and its mechanism has independent opposition, and a condition phrased as “an independent result” cannot tell those apart. The condition was the wrong condition.
What the five papers in this chain agree on is narrower and better attested than what this document currently says: widen the residual stream into several streams and constrain the mixing — with mHC’s doubly stochastic matrices, oHC’s rotations, sHC’s spectral sphere, or Qwen’s gate that drops the mixing matrix altogether all counting as instances. Restating the practice at that level is the right move and is deliberately not made here: it changes what the document claims rather than its status, and that is a curation decision rather than a backlog pass.
The alternative to promoting is restating the practice at the level its literature agrees on — widen the residual stream and constrain the mixing — which all four papers hold and none of them disputes.
Sequence and siblings
This is one branch of SOTA-169, the trunk practice of widening the residual stream and constraining the mixing: it keeps that practice’s width and names the doubly-stochastic manifold as the constraint. Everything disputed here is the choice of constraint, so the trunk stands whatever this branch’s standing becomes, and this branch cannot stand without it.
Residual connection → hyper-connections (SOTA-137, retired) → this. The sibling variation is Attention Residuals (SOTA-133, now Active): where mHC widens the stream and constrains a fixed mixing, AttnRes keeps one stream and lets each layer attend over its predecessors. Both are Proposed; nobody has compared them.