nucleation

LIT-1551: Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1192) in full, from arXiv v4, the CLeaR 2024 version; worth reading for distributed interchange interventions and their training objective, for the exact example in which an algorithm fails in the neuron basis and holds after a rotation, and for the finding that BERT's entailment subspace decomposes into the two words' identities while an equality network's relation does not (THEORY-983). Read it knowing that IIA is reported on the rotation's training data, that Table 2's brute-force row is a copy of Table 1's, that the learned subspaces are fixed only up to rotations within them, and that its random-network control reaches 0.64 at width 4,096 (THEORY-1002).
Version2
Tagscausality, representation-learning, anthology-candidate
Cited as “paper” byNOTE-1192 — Geiger, Wu, Potts, Icard and Goodman — Distributed alignment search (DAS)
Cited as “source” byTHEORY-1002 — On frozen randomly initialized feed-forward networks at chance on hierarchical equality, with 16 input dimensions, distributed alignment search for the two-equality causal model reached interchange intervention accuracy of 0.50 to 0.51 at hidden sizes up to 256 but 0.55 at 1,028 and 0.64 at 4,096, so in a hidden layer much wider than the input a learned rotation can produce above-chance interchange accuracy for an algorithm the network does not implement, THEORY-983 — In Geiger et al.'s three-layer feed-forward network trained on hierarchical equality and in BERT fine-tuned on monotonicity inference, interchange interventions on subspaces found by a learned rotation aligned the hypothesised two-variable causal models with interchange intervention accuracy of 1.00 on the rotation's training data, where the axis-aligned alignment closest to the learned rotation reached 0.73 and 0.51; in BERT, the learned lexical-entailment subspace was in turn aligned, at 0.97 to 0.98, with a model holding only the identities of the two words

Graph View

Geiger, A., Wu, Z., Potts, C., Icard, T. & Goodman, N. D. (2024). Finding alignments between interpretable causal variables and distributed neural representations. Proceedings of the Third Conference on Causal Learning and Reasoning, PMLR 236. arXiv:2303.02536 (5 March 2023).

Read in full on 2026-10-26 (NOTE-1192). Two accounts were drawn from it: THEORY-983 and THEORY-1002.

Why this is here

The owner asked on 2026-10-26 how the record stood on neural circuit theory; the answer found it held almost none of the core circuits literature, and the owner said “go ahead and register and read those”. DAS is the method at the centre of the record’s question on alignments: QUESTION-234 and THEORY-802 ask whether interchange interventions fix the direction that mediates a variable, and the record held that debate only through Makelov et al. (LIT-968), Wu et al.’s reply (LIT-961) and MIB (LIT-1550), all of which test or use DAS without the record holding DAS itself. It also brings causal abstraction (LIT-1310) to trained networks, and its own example shows a verdict that changes with the basis.

Backlinks