THEORY-176: A task-loss stitching penalty compares representations in one decision problem, through a fixed top network and a restricted map class, so it is not Blackwell's order: representations that are functions of each other can stitch differently, and a negative penalty shows the fitted map did not reproduce the layers it replaced

Source

• Bansal, Nakkiran & Barak (2021), LIT-850, §§2–3 (eq. 1, the definition of the penalty, and the invariance and asymmetry arguments) and §6 with Figs. 2C and 3B–C, as read in NOTE-653.
• Blackwell (1953), LIT-781, as read in NOTE-595 and stated in THEORY-156, for the order compared against.
• Lenc & Vedaldi (2014), LIT-363, as read in NOTE-309, for the identity stitch that fails between linearly equivalent networks.

What was actually shown

The paper’s definition. For a top network A, a layer ℓ, a candidate representation r and a family S of “simple” maps, L_ℓ(r; A) = inf over s ∈ S of L(A_{>ℓ} ∘ s ∘ r), and the stitching penalty is L_ℓ(r; A) − L(A) (eq. 1). L is the test error of A’s own task. In practice s is a 1×1 convolution between two BatchNorms, trained on the task’s training set. A negative penalty is read as r being better than A’s first ℓ layers, and §6 builds “more is better” on such readings: a bottom trained on 25K CIFAR-10 samples lowers a 10K network’s error by up to about 22 points (Fig. 2C).

The comparison. Treat a representation r as an experiment about the label Y: for each label y, the law of r(X) given Y = y. CIFAR-10 and ImageNet have finitely many labels, so THEORY-156 applies. There, r is at least as informative as r′ exactly when r attains, in every decision problem with bounded loss, every risk r′ attains, and exactly when r′ is a garbling of r. Four relations follow from the definitions. They are this record’s derivations, not the paper’s, and each is elementary.

1. One problem, not all. The penalty uses one loss. Blackwell’s order quantifies over every decision problem, and comparison by one problem is strictly weaker: two experiments can have the same, or ordered, Bayes risk for the task loss and be incomparable. So a negative penalty does not give r ⊒ A_{≤ℓ} in Blackwell’s order.
2. Restricted rules, so upper bounds. The rules the stitched network can use are A_{>ℓ} ∘ s with s ∈ S, not every measurable rule on r. Its loss bounds r’s best attainable risk for that problem from above, and L(A) does the same for A’s first ℓ layers. Comparing two upper bounds orders neither best risk. A positive penalty therefore does not show that r carries less task-relevant information than A’s layers; it may only be unusable by A’s head through S.
3. Not invariant under Blackwell equivalence. If r′ = g ∘ r with g invertible, each is a function of the other, and they are equivalent in Blackwell’s order. Their penalties need not agree when g is not in S. The paper says so and wants it (§3): because a 1×1 stitcher is weaker than general orthogonal maps, “model stitching can predict that shuffling pixels leads to a ‘worse’ representation”, although a pixel shuffle loses nothing. Lenc and Vedaldi’s identity stitch between networks that a learned linear map makes interchangeable gives more than 99% error (LIT-363, Table 4): with S = {identity}, linear equivalence is invisible.
4. Where the two orders meet. If some s ∈ S gave s ∘ r = A_{≤ℓ} on the input distribution, then given each label A_{≤ℓ}(X) is s applied to r(X), so A’s layers are a garbling of r and r dominates them in Blackwell’s order, and the stitched network is A, with penalty zero. That is Lenc and Vedaldi’s regression form of stitching (φ′ ≈ E φ), and it is the form whose success certifies a direction of Blackwell sufficiency, restricted to garblings in S. A negative penalty, the evidence for “more is better”, shows the opposite of reproduction: the stitched network beats A, so s ∘ r is not A_{≤ℓ}. What the fitted map found is an input on which A’s head does better than on its own layers.

What could have come out otherwise. Relations 1, 2 and 4 are consequences of definitions and cannot fail except by an error in a step. Relation 3 is a possibility the definitions allow, and the paper argues that it occurs. The note of 2026-10-19 below shows it with a constructed example; it has not been measured in trained networks. The pixel-shuffle experiment the paper describes, and does not run, is the test QUESTION-163 asks for.

What this does not say

• Not that stitching is uninformative. A low penalty between two networks shows that, for that task, one network’s head can use the other’s layers through a cheap map. That is a real and useful fact about interchangeability, and the paper’s comparisons with CKA rest on it.
• Not that “more is better” is false. A 25K-sample representation that lowers a 10K network’s error is better for that task through that head. The claim here is about what kind of ordering that is, not whether it holds.
• Not that a stitching order could never recover Blackwell’s. Stitching over every decision problem, with an unrestricted decoder, would compare best risks problem by problem, which is Blackwell’s order (THEORY-156). The paper’s protocol fixes the problem and restricts the decoder; what happens between the two is not addressed here.
• Not about information measures in general. Kernel-based measures such as CKA, which THEORY-008 relates to linear readout, are a different comparison again; and invariance under the architecture’s symmetry group (THEORY-112) is a smaller class than Blackwell equivalence.
• Not a measured result. The paper reports no experiment that separates the two orders. The relations are derivations, and the measurement relation 3 would need is QUESTION-163.

Note of 2026-10-19: an example for relation 3

Carried out for the promotion audit’s batch 027. Let the label Y ∈ {0, 1} be uniform, and the input X have one channel and two positions, with X₀ = 2Y − 1 and X₁ a fair ±1 coin independent of Y. Let A’s first ℓ layers be the identity, so r = A_{≤ℓ}(X) = X, and let its head read position 0, A_{>ℓ}(z) = 1[z₀ > 0]. A has error 0. Let r′ swap the two positions. Then r and r′ are invertible functions of each other, so each is a garbling of the other and they are equivalent in Blackwell’s order.

The paper’s stitcher, a 1×1 convolution between BatchNorms, is here the same affine map z_p ↦ αz_p + β at every position p. Through it the head sees αX₁ + β, which is independent of Y, so every stitched network has error 1/2. The penalty is 0 for r and 1/2 for r′. The two are Blackwell-equivalent and stitch differently, so the penalty order is not Blackwell’s order. Whether the difference is large in trained networks is QUESTION-163.