Why it is a question
The reply’s construction runs over checkpoints θ_0 … θ_S of a small transformer trained on two or three synthetic task families. It has three layers.
- Features. A binary allocation B_{ck,s} of components c (head outputs, MLP channels) to latent features k at checkpoint s, with r_{c,s}(x) ≈ Σ_k B_{ck,s} a_{k,s}(x) v_{ck,s} and an Indian buffet prior on B: “several components can participate in the same feature, and one component can participate in several features”. This is CASE-165’s model with components for employees. “This factorization is an approximation, not an exact decomposition of transformer computation.”
- Development. A temporal prior p(B_s | B_{s−1}, θ_s, 𝒟_s) allowing features to be born, persist, redistribute and disappear.
- Regimes. Above the features, candidate regimes of participating features, their routing, and the contexts in which they are consequential: “an inferred structure, not a primitive in the transformer or the initial probabilistic model”. That is CLAIM-454 in a new setting (its sixth appearance).
Three analyses are to be compared: feature-only, circuit-only, and coupled feature–circuit dynamics, against “standard sparse dictionaries or crosscoders, direct circuit localization, and simpler checkpoint-level state-space models”. The reply: “The dynamic IBP must earn its complexity.”
What the record already holds
Most of the reply’s apparatus is on file, and it cited four codes (CLAIM-367, THEORY-169, CLAIM-329, THEORY-253).
- Who governs training. “The transformer itself is not necessarily governing this development” is CLAIM-367: individuated for training, the unit is at least the network–optimizer pair, and its six relations map onto the reply’s governance table (routing, gradient allocation, optimizer, curriculum).
- Inference is not training. “That does not mean the network’s persistent parameters change during an ordinary forward pass” is THEORY-169’s reading of in-context conditioning. The reply’s three timescales (training, layer, token) are CLAIM-118’s sources of change.
- The update rule and its interventions. (θ_{s+1}, m_{s+1}) = 𝒰(θ_s, m_s, ℬ_s), and “clone the full training state, modify a particular training exposure or update pathway”, are CASE-113’s update and crossed interventions. What the reply adds is a third arm, training exposure (task mix and curriculum), beside CASE-113’s optimizer and learned state.
- The optimizer arm. THEORY-253: in the Geometry model, gradient descent learns independent skills together and sign descent in sequence. That is a simulation in a solvable model. That Adam-like optimizers cause sequential onset in trained networks is the Proposed THEORY-783, and its test is the open QUESTION-193. The reply cites THEORY-253 for transformers without that scope.
- Task interaction. The reply’s g_kᵀg_j “under ordinary Euclidean gradient descent” is CLAIM-366’s object: its sign is not invariant under reparameterization, and a negative product does not exclude common descent. The reply’s two caveats, Euclidean geometry and Adam, are one fact (CLAIM-366’s note of 2026-10-26).
- Continuity. The reply’s “representational” and “computational” continuity are TERM-071’s state and dynamical senses. It cited CLAIM-329 for “distinct forms of organizational persistence”, which is TERM-071’s job. CLAIM-329 is the thesis that persistence through turnover is by causal transmission, not resemblance, and the reply’s §6 asks its question without it: “Can we distinguish a continuing computation from a newly learned replacement with similar output behavior?” (CLAIM-329’s note of 2026-10-26).
- Influence on inference against influence on learning. “A component might have a large effect on current predictions but receive little additional training signal” is QUESTION-134’s split of influence, competence and access, and CASE-117’s coupled and decoupled training exposure (QUESTION-134’s note of 2026-10-26). The rigidity and recovery material is CLAIM-286, and the dormant-alternative caveat is CLAIM-287’s reinstatable capacity.
- Acquisition. “Does a circuit emerge by acquiring new features, reorganizing existing features, or both?” is QUESTION-039’s second half, whether acquisition is one kind of event or several. THEORY-039, THEORY-481 and THEORY-229 answer parts of it, and so now does THEORY-973: in small attention-only transformers, in-context learning arrives in the window where induction heads form, and removing the need for two heads to compose moves it.
What is new is the coupling: an organizational reading of circuits over a developing feature allocation, with a stated way to lose.
The works the reply drew on, read
The six works T24 cites by paraphrase were read on 2026-10-26. They supply the comparators and show what is still missing.
- A circuit-only baseline. The induction-heads paper’s per-snapshot head scores and ablation attributions (NOTE-1173), and the refined local learning coefficient of each head over training, whose curves diverge and cluster by the heads’ functional types (THEORY-975).
- A feature-only baseline. Cross-checkpoint crosscoders, which score each feature’s causal use on a task at every checkpoint (NOTE-1186, THEORY-1004).
- Reorganization, observed. In the refined-LLC study’s model the previous-token heads, which precede the induction heads, start as n-gram heads, and n-grams pass from one head to another (THEORY-967). That is the question’s “reorganizing existing features”, and a feature moving between carriers, seen at the level of components in two checkpoints of one run.
- An exact version of the three layers. In Musat, Pimentel, Zucchet and Hofmann’s attention-only transformers on block-list inductive tasks, the data’s symmetries fix a subspace of weights that gradient descent never leaves, spanned by elementary token and position actions (THEORY-981). The components’ features are given by symmetry rather than inferred under a buffet prior; circuits are sets of shared coordinates, so their overlap is exact; and development is the reduced dynamics on those coordinates, in which circuits that solve overlapping data interact although their coordinates are orthogonal (THEORY-1009), and initialization picks among competing circuits (THEORY-1001). Inside its idealization, both conditions below are met: the state is the coordinates themselves, and the commutant and the paper’s Assumption 5 remove the basis and split symmetries.
- Localization scored on its own hypothesis. In the interpretability benchmark T24 cites, sparse-autoencoder features never beat the model’s own neurons at localizing causal variables, and the best method, distributed alignment search, is trained against the causal model it is scored on (THEORY-968).
- Relations against components. The fine-tuning study’s finding, that circuits keep their nodes and change their edges (THEORY-997), is what a fixed edge budget alone produces in the record’s null simulation, so it needs a null at equal edge budget before it bears on the coupling.
None of the six fits anything to early checkpoints and predicts later ones. The prospective arm is still undone in the literature the record holds.
The circuits literature, read
The core circuits works, registered and read on 2026-10-26 at the owner’s request, supply the question’s units, instruments for its circuit-only arm, and cautions for both arms. None of them studies training.
- Where the units come from. Zoom In (LIT-1543) names features and circuits as the units of trained networks, and universality as the hypothesis that they recur across models, with evidence it calls “anecdotal” (THEORY-1013, THEORY-986). The question couples exactly these two units.
- Circuit-only measurements, computable at every checkpoint.
- The framework’s composition scores and eigenvalue statistics (LIT-1534, NOTE-1189) are read from weights alone, and its path expansion is exact (THEORY-988). Their fragility is on record too: the authors published a correction to their own composition diagrams (a cyclic permutation inside a Frobenius norm).
- ACDC (LIT-1547) runs on the same two-layer attention-only induction setting and finds its heads.
- AtP and AtP* (LIT-1531) make node-level patching cheap enough to run per checkpoint, and Pythia releases its checkpoints (THEORY-996).
- Causal scrubbing’s recovered loss (LIT-1549) is a candidate “how much does the circuit explain” score per checkpoint.
- Transcoder or cross-layer-transcoder weights (LIT-1554, LIT-1552) describe circuits through MLP layers, but neither paper matches features across checkpoints, and the cost is large (about 210 H100-hours for 2M features on Gemma 2 2B, NOTE-1185).
- Cautions for the circuit arm.
- Knockouts reveal heads that ordinary operation hides: in IOI, backup name movers recover most of the effect when the name movers are ablated (THEORY-980). A head “born” at a checkpoint may have been there as a backup.
- The metric, the ablation type and the visiting order must be held fixed across checkpoints: a greedy circuit is one of several sufficient subgraphs, and which one is returned depends on the order (THEORY-966, THEORY-970). Attribution misses concentrate in queries and keys behind a saturated softmax (THEORY-996).
- Attribution graphs freeze attention patterns, so they cannot see induction and cannot serve as the circuit-only arm in the induction setting (NOTE-1185).
- Cautions for the feature arm.
- Sparse-autoencoder features are a resolution set by dictionary width (THEORY-1016), and which concepts get a feature depends on their frequency relative to the dictionary’s size (THEORY-992). An apparent split across checkpoints may be the dictionary’s.
- Features can be spread across layers (Templeton et al.’s cross-layer superposition, NOTE-1178), which confounds tracking within one layer.
- Circuits over features are faithful only with the dictionaries’ error terms kept (THEORY-1007), so a circuit arm built on features must carry the remainder.
- A concrete developmental target. Sparse feature circuits find subject–verb agreement across a relative clause carried by two pathways that converge without fully intersecting (THEORY-1014). When the clause-boundary pathway forms relative to the direct one is a question the coupled description could be asked to predict.
What would answer it
A prospective test, as the reply puts it: “fit the model using early training checkpoints, then test its predictions about later functional changes, including changes under held-out training interventions”. The interventions are CASE-113’s arms plus training exposure, and architecture is a fourth with a known effect on timing: letting a single layer attend by the previous token moves the induction transition (THEORY-973). The reply’s induction-circuit setting is a natural first case: which partial capacities exist before the circuit is effective, and whether it forms by acquiring features or by reorganizing them. The answer is yes if the coupled description out-predicts the feature-only and circuit-only ones at matched complexity (CLAIM-427’s third grade), and no if it does not. In the induction setting the baseline to beat is the reduced dynamics on the symmetric subspace’s coordinates (THEORY-981), which is much stronger than a checkpoint-level state-space model. The design should avoid the induction-heads paper’s confounds: one run per model, one data order for every model, and a transition inside learning-rate warm-up (NOTE-1173).
Two conditions come from checks made when this was filed (CASE-165):
- The temporal prior must not condition on θ_s, from which B_s is read. Otherwise it predicts nothing, and the prospective test is empty. The dependent IBP in its collapsed form is a candidate prior over B_s that does not (THEORY-1006), though it still treats components as exchangeable.
- Births and splits must be told apart from the model’s free symmetries. Rescaling, splitting a feature into copies and a zero contribution vector all leave the reconstruction unchanged.
What would not answer it
- “Merely recovering visually plausible circuits from already-trained models” (the reply).
- A tie with a state-space model built on the same kernels, which is an identity. The lab’s comparator pilot (dmarx/PerspectiveLab@dce302b) already showed this for its own models (CLAIM-427’s note of 2026-10-26).
- Feature births or splits that a change of basis or the buffet’s split symmetry produces (CLAIM-263’s quotient, THEORY-018).
- Continuity of a feature judged by resemblance alone (CLAIM-329, TERM-071). A crosscoder’s “maintained” feature is one index of a jointly trained dictionary, and an n-gram’s migration between heads is the same tokens affected at two checkpoints; neither tests inheritance (CLAIM-329’s note of 2026-10-26).
- A localization scored only on the hypothesis it was trained against; held-out interventions are needed (THEORY-968, CLAIM-278).
- Success defined in the framework’s own terms (CLAIM-278).
Not in scope
Instructions for training transformers, which are the anthology’s (ADR-048). Whether a circuit regime is a perspective in any sense beyond TERM-072’s organizational one: the reply itself says “satisfying this organizational definition would say nothing by itself about phenomenal experience or autonomous agency”.