Source
- Nanda, Chan, Lieberum, Smith & Steinhardt (2023), LIT-345, §§4–5 and Apps B–D, as read in NOTE-283.
- Engels et al. (2024), LIT-322, §5 and App. K, as read in NOTE-273.
- Liu, Kitouni, Nolte, Michaud, Tegmark & Williams (2022), LIT-858, Fig. 1, Fig. 2d and App. H, as read in NOTE-662.
- Karkada, Korchinski, Nava, Wyart & Bahri (2026), LIT-860, §§3–4 and Fig. 4, as read in NOTE-664.
- Power et al. (2022), LIT-341, §3.1 and §3.4, as read in NOTE-287.
- The machinery: the convolution theorem (Kondor & Trivedi, LIT-305, Prop. 2) and characters of finite abelian groups (O’Donnell, LIT-346, §8.5).
What was actually shown
The circuit. The network is a one-layer ReLU transformer: P = 113, 30% of pairs, full-batch AdamW with λ = 1. Its embedding is sparse in the Fourier basis, with key frequencies k ∈ {14, 35, 41, 42, 52}. Attention and the MLP form cos/sin(w_k(a+b)), with FVE of 93–98%. W_L is about rank 10: one cos and sin direction per key frequency. The logits are fitted by Σ_k α_k cos(w_k(a+b−c)). Ablating the other logit Fourier components improves the loss by 70%, and projecting onto the null space of W_L destroys it. Using several frequencies makes the sum interfere constructively only at a + b ≡ c (App. B). Other seeds use three or four different key frequencies. Every generalising weight-decay model in Table 5 uses a variant of the circuit (NOTE-283).
The representation-theoretic reading (NOTE-283; not the paper’s words). For each k, the functions cos(w_k x) and sin(w_k x) span a real two-dimensional irreducible of ℤ/113, the conjugate character pair χ_k, χ̄_k. The angle-addition step computes χ_k(a)χ_k(b) = χ_k(a+b). The readout is Re Σ_k α_k χ_k(a+b) χ̄_k(c). This is the convolution theorem for δ_a ∗ δ_b, evaluated at a few frequencies only. The near-equal cos and sin coefficients in Table 1 are what a rotation pair requires. The paired singular values of W_L are plotted as norms but not reported (NOTE-283).
The same object elsewhere. Engels et al.’s weekday feature (cos 2πα/7, sin 2πα/7) is the frequency-1 real irreducible of ℤ/7. Their regression on Llama months adds frequency-2 and sign characters of ℤ/12 (NOTE-273). Patching the circle causally changes the answer in Mistral 7B and Llama 3 8B. Power et al. saw only a qualitative t-SNE “number line” in the output layer (NOTE-287). The circles are not planar, though. Modell et al. (LIT-935, Fig. 2, read in NOTE-802) show the same Mistral 7B representations weaving through a third principal component. That is consistent with the extra frequency-2 and sign characters on Llama months, and not with a single real two-dimensional irreducible for each feature.
The earliest sighting. Liu et al. (LIT-858, v1 May 2022) saw the circle first: in the top two principal components of a transformer’s embeddings for modular addition (Fig. 1), and in a toy whose embeddings are summed and decoded (Fig. 2d). They give no Fourier analysis and no frequencies, so this is the circle without its characters. In the toy, the operation, a sum, is built into the architecture, which is another case of the group coming from the task (NOTE-662).
Where the group sits in language data. Karkada et al. (LIT-860) give the data-side mechanism for word embeddings. When co-occurrence among words on a latent cycle, such as the months, depends only on their separation, the co-occurrence matrix is circulant, and a spectral embedding places the words on its Fourier modes (THEORY-183). The ℤ/12 is then in the corpus statistics, and Engels et al.’s frequency-2 month component is the next harmonic. Their embeddings are computed by diagonalising the matrix, not trained, and they do not test Llama or Mistral, so this explains where the circle could come from in a language model without showing that it does (NOTE-664).
What this does not say
- That networks find the group. Nanda et al. chose the DFT because they knew the group. Engels et al. took ℤ/7 and ℤ/12 from the task (NOTE-283, NOTE-273). A spectral test could see these irreducibles only as pairs, and could not tell their frequencies apart (NOTE-307).
- That generalisation requires this circuit. The dropout models grok but are not Fourier-sparse (NOTE-283). The founding grokking run used no weight decay (NOTE-287).
- Anything about non-abelian tasks. Whether S₅ generalisers use higher-dimensional irreducibles is untested (NOTE-283). Liu et al.’s S₃ experiment (LIT-858, App. H) does not test it: its embeddings are learnable 3 × 3 matrices multiplied by hand, so a matrix representation is imposed, not found (NOTE-662).
- That the group always comes from the task. Bird (LIT-976, read in NOTE-759; THEORY-288) finds small autoencoders whose latents are arranged by B_n or S_n, groups imposed by the elementwise activation and absent from the task, which vanish when the activation is made rotation-equivariant. That is discrete axis alignment, not characters, so it is compatible with this account; but “not the network” holds for these cases, not in general.
- That language models do modular arithmetic this way. Both models get trivial accuracy on plain modular prompts. The downstream algorithm is only suggested (NOTE-273).
Connections
THEORY-019, on symmetry and degeneracy, covers why a degeneracy test would see these pairs but not their frequencies. THEORY-017: the Fourier basis is supplied, and only the isotypic planes are fixed by the group.
Note of 2026-10-27: addition in a language model is not all Fourier
Ameisen et al.’s Circuit Tracing (LIT-1552, NOTE-1185), read on 2026-10-26 at the owner’s request, traces two-digit addition in Claude 3.5 Haiku. Its features include ones-digit lookup tables (_6 + _9 → sum = _5), low- and moderate-precision magnitude ranges and “starts with 51” patterns, some of which “aren’t describable in Fourier terms”, beside mod-10 and mod-100 features that are. This account concerns a network trained on modular addition and the circles found in language models; the two are compatible, and the language model’s addition is not only the character products found here. The same article proves why some combining step is forced: a unit whose pre-activation is additively separable in the two operands cannot detect a sum that has two decompositions (THEORY-971). The grokking network’s products of characters are one such step. The account is unchanged.