Source
Geiger, Wu, Potts, Icard and Goodman (2024), LIT-1551, read in full in NOTE-1192: §4.4 and Figure 3.
What was actually shown
Randomly initialized networks of the equality architecture, frozen at chance accuracy (50%), with four 4-dimensional input vectors (16 input dimensions) and hidden sizes 16, 64, 256, 1028 and 4096, intervention subspaces of half the layer (k = 8 to 2048). DAS trained a rotation for the two-equality causal model at layer 1. IIA: 0.50, 0.50, 0.51, 0.55, 0.64. The authors: “in small networks, there is no ability to increase interchange intervention accuracy. However, as we increase the size of the hidden representation to be orders of magnitude larger than the input dimension of 16, the interchange intervention accuracy increases.”
What could have come out otherwise. IIA could have stayed at chance at every width, which would have shown that the search adds nothing a network does not have.
What this does not say
- Not that trained-network results are artefacts. The trained network reached 1.00 at width 16, where the random control is at 0.50.
- Not how the effect scales in trained wide models such as language models, whose layers are far wider than this input; one task, and the paper does not say how many seeds or which data the figure scores.
- Not a mechanism. Why width helps (more random features for a rotation to combine) is the authors’ hypothesis, not tested.
Connections
- THEORY-968. MIB’s best causal-variable method is DAS, trained against the model it is scored on; this is the only measured bound in the record on how much such training can manufacture.
- LIT-968 (NOTE-782). Makelov et al.’s illusion is a mechanism by which a learned subspace patch can change behaviour without the variable being there; this control shows the size of the effect grows with width.
- THEORY-802. An abstraction that passes only because the search is expressive is one more way an interchange test underdetermines what the network computes.