Source
Lad et al. (2024), LIT-417 — ARXIV-2406.19384.
What the account says
Four stages, hypothesized across five model families from the localized-sensitivity result in THEORY-020:
- Detokenization — local context is integrated to lift raw token embeddings into higher-level representations.
- Feature engineering — task- and entity-specific features are iteratively refined.
- Prediction ensembling — hidden states are aggregated into plausible next-token predictions.
- Residual sharpening — irrelevant features are suppressed to finalize the output distribution.
These are LIT-417’s stages, proposed as its own interpretation of the layer-deletion and swap result, with only the last backed by a dedicated experiment of its own.
The framework is offered as a resolution of a standing tension the paper names in its introduction: iterative inference, where every layer nudges the same distribution, against the circuit hypothesis, where components have delineated specialized roles. Stages are the shape that would let both be partly right — specialization between stages, iteration within them.
What is actually measured, and what is inferred
Stage 4 carries an experiment, and it is the strongest part. Duplicating blocks of layers in the latter half of a model consistently lowers output entropy against baseline, and repeats at 80-90% depth improve benchmark performance. That is a prediction with a direction that could have come out the other way, and it did not. It is supported by a measured rise in suppression neurons near the end of the network, outstripping prediction neurons in the final few layers.
Stage 1 rests on a structural argument plus other people’s results. The first layer is the only one mapping from the embedding basis into the residual stream and is a function of the current token alone — so ablating it makes the rest of the network blind to local context. What happens next, the concatenation of nearby tokens belonging to the same word or entity, is imported from prior work rather than established here.
Stages 2 and 3 are read off depth-dependent curves. They are consistent with the measurements and they are not independently tested.
So the four stages are not four findings. They are one measurement, one experiment, one structural argument and two interpolations, and the status here reflects that rather than the framework’s plausibility.
What this does not say
The authors hedge it themselves, in the title and in the limitations.
Stage boundaries are approximate; multiple stages may co-occur within a single
layer; the framework reflects aggregate trends while individual tokens may
follow distinct paths; model-specific differences are not isolated. A
framework whose boundaries are approximate and whose stages overlap is
difficult to falsify, which is why promote_when above asks for a boundary
predicted before it is tested.
It does not itself establish that detokenization is a stage rather than an artifact of where the tokenizer split things. The strong version — that a network spends early depth reassembling units its vocabulary broke — is not what this paper showed; it imports the claim. The direct evidence is now filed separately as THEORY-021, on the two papers this one cites for it, and it is considerably stronger than what is here. Stage 1 is the one stage of the four that does not rest on this paper’s own measurements.
It does not connect to the signal-side claim, and neither does the stronger account. THEORY-022 says the lexical unit in English is delimited by construction rather than frequency; THEORY-021 says models rebuild units their tokenizer split. Neither was measured against the other, and the experiment that would connect them has not been run.
And it is not a claim about where to cut a model. No practice follows — see the same section in THEORY-020.