Contribution
It takes one neuron family of one trained vision model and argues, with several kinds of evidence that could have disagreed, that it is what it looks like: a family of curve detectors. The article is explicit that it is meant to settle a dispute, whether “neural network representations [are] composed of meaningful features”, and it separates what it claims into four parts (causality, generality, purity, family), each with the experiment it thinks best supports it. It is the thread’s first article devoted to a single family, and the vision case the transformer circuits work later points back to.
Key insight
“What exactly are we claiming when we say these neurons detect curves?” That a unit fires on curves when it fires strongly is easy to show. The contentious claims are that curves cause the firing, that it generalizes across curve stimuli, that the unit does nothing else, and that the family covers the space. Each needs its own test, and the strongest single tests are the cheap ones artificial networks allow: optimize an input, render a synthetic stimulus, rotate a real image.
Assumptions
- Model. InceptionV1 trained on ImageNet. The curve family studied is the ten units of mixed3b (“3b”), which the article calls the third of four contiguous layers with curve units (conv2d2, 11 units; 3a, 11; 3b, 10; 4a, 12, by the opening figure’s labels).
- The unit studied in depth. The dataset analysis, the attribution quilts and the hand-labelling are of 3b:379 only, “because some experiments required non-trivial manual labor”; the authors say the core ideas “apply to all curve detectors in 3b”.
- Attribution is the linear one: 3b’s pre-activation is a linear function of 3a, so a 3a unit’s contribution is its activation times its weight, and the article renders FeatureVisualization(abs(W ⊙ h_prev)).
- Activation statistics are of pre-ReLU values over receptive-field sized crops of ImageNet images.
- Human labels are one author’s (Nick Cammarata), made on pixels alone with a four-class rubric: curve, imperfect curve, unrelated, opposing curve (orientation more than 45° from the unit’s).
Key results
- Feature visualization. Every optimization for maximal firing produces a curve, “even when we explicitly incentivize the creation of different kinds of images using a diversity term”, which the authors take as “strong evidence that curve detectors aren’t polysemantic in the sense we usually use it”. The visualizations also show opposite hues across the boundary and small perpendicular lines (“combing”).
- Dataset statistics of 3b:379. Pre-activation mean “about -200”; it fires on “just 11% of cases across the dataset”, and both tails of the pre-activation distribution are exponential. Randomly sampled images by activation level: strongest positive activations are curves like the visualization, weak positive ones imperfect curves, near-zero ones lines or curves about 45° off, and the most negative ones curves more than 45° away.
- Attribution of activating images back through 3a shows the curves that caused the firing, which rules out, for instance, a clock detector (many top images are clocks).
- Blind hand-labelling. Over 800 images, sampled at random in activation bins of 100 (“We did not cherry-pick from these bins”), labelled without activations. The labels “separate cleanly into different activations”. Many images labelled unrelated contain subtle curves that attribution reveals, and the authors call the unit “a superhuman curve detector”.
- The weak tail. By probability density the strongly firing cases are invisible, and “the majority of stimuli classified as curves also fall in these cases” of weak or no firing. By contribution to expected value, x·p(x), “curves and imperfect curves form 55%”. No precision–recall curve is given, because recall “would be dominated by potential labeling error”.
- Joint tuning. Rotating each unit’s top dataset examples in 1° steps gives each unit a “gaussian-like bump” around its preferred orientation, and the ten together “jointly span all orientations”.
- Synthetic curves. Rendered curves varied in orientation and curvature drive 3b:379 “up to 24 standard deviations above the average dataset activation”; the response region is a triangle, wider in orientation at higher curvature, with “wisps” where tiny changes swing activations several standard deviations, which the authors call fragile and non-robust and see as early as conv2d1’s Gabor family. Responses are invariant to fill and colour. Synthetic angles drive the unit where one arm is tangent to its preferred curve, most where both are; rounder stimuli drive it more.
- Neighbouring families. conv2d2 has concentric-curve and combed-edge detectors, with an “echo” 180° away; 3a curve units also have echoes; 3b’s do not, and gaps in 3b’s coverage are partly filled by double-curve detectors; 4a’s curve units “are better thought of as corresponding to specific worldly objects”, for example 4a:406, a tilted 3D circle (tops of cups and pans). The authors call 4a the point where “interpretability can be subjective”.
- Uses. Attribution to curve units as curve extraction on an angiogram, compared qualitatively with Canny edge detection; spline tracing by backpropagating into a differentiable spline; butterfly tracing by composing with NMF factors (“We traced 23 butterfly images and chose our 15 favorites”). None is benchmarked, and the authors say so.
- Combing. Excitation by short perpendicular lines is seen in curve units of AlexNet, InceptionV1 and InceptionV3, in a Places365 model, and in one V4 neuron’s optimized stimulus in a macaque. Four explanations are offered, none preferred; the last is that it is a shortcut in how inhibition is implemented.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | When 3b:379 fires strongly, the stimulus is reliably a curve of its preferred orientation | strong for this unit | random activation-binned samples, blind labels, attribution |
| C2 | Curves cause the firing, not a correlate (causality) | moderate | feature visualization and attribution; an optimized input shows what can drive the unit, not what does on data |
| C3 | The units respond to a wide range of curves, invariant to fill, colour and much of radius (generality) | strong for the stimuli tested | synthetic curves and angles, rotated dataset examples |
| C4 | The units have no meaningful secondary function (purity) | weak to moderate here | feature visualization with diversity, activation-binned labels; most weak firings are unlabelled, and the mechanistic evidence is deferred |
| C5 | The ten 3b units jointly span all orientations (family) | strong | joint tuning curves, radial tuning curve; gaps acknowledged |
| C6 | Curve families exist in four contiguous layers, becoming object-specific by 4a | moderate | radial tuning and dataset examples, described qualitatively |
| C7 | Combing is common to curve detectors across vision models and perhaps biological ones | weak | feature visualizations of three models and one macaque neuron |
Concepts
- Neuron family: a set of units implementing variants of one feature, here curves at different orientations.
- Joint tuning curve: every family member’s response to one unit’s top dataset examples rotated through 360°, aligned and averaged.
- Radial tuning curve: a polar plot of each family member’s response to synthetic curves by orientation.
- Contribution to expected value: x·p(x) over the activation distribution, the article’s measure of how much each activation level matters to the unit’s output.
- Echo: a weaker response 180° from a unit’s preferred orientation.
- Combing: excitation by small lines perpendicular to the curve.
Connections
It is the third article of the Circuits thread (DOI suffix .003), after Zoom In (LIT-1543) and An Overview of Early Vision in InceptionV1, both by the same team, and the first of a planned three on curves: behaviour, then “how they’re built from earlier neurons”, then “their prevalence across models”. It cites the neuroscience of curvature-tuned V4 neurons (Pasupathy and Connor 1999; Jiang, Li and Tang 2019) as the model for its synthetic-stimulus experiments, and its own Feature Visualization and Building Blocks articles for the methods. Neel Nanda’s attribution-patching post (LIT-1581) names “the curve circuits work” as the circuits work that “comes closest to being general”, and the hand-coded curve neurons as the stronger kind of mechanistic evidence.
Bearing on the record
- THEORY-1013. Zoom In’s seven arguments are feature visualization, dataset examples, synthetic curves, joint tuning under rotation, the implementing circuit, the downstream use and a hand-written reimplementation. This article supplies the first four (with attribution and blind labelling added) for one family. The other three are deferred again: “We’ll see in the next article that the algorithm used is straightforward and understandable, and we’ve since reimplemented it by hand.” It gives no count over a layer. The family is ten of mixed3b’s 480 units, about 2%. THEORY-1013’s promote_when asks for “the fraction of units that meet” a stated criterion, and this article does not supply it. The thread’s overview of early vision, which sorts the early layers’ units into families, is the place that count would come from (proposed for registration). So the reading confirms the example and leaves the account Proposed for the reason it gives: chosen examples, no denominator.
- THEORY-986 (universality). Combing in three architectures and a Places365 model is the only cross-model evidence here, and the article defers “prevalence across models” to its third part.
- THEORY-802 and QUESTION-274. The units are taken in the neuron basis throughout, and no other basis is tried. The authors’ footnote on polysemanticity concedes that “not every neuron is meaningful”. The case is a basis-relative one, a neuron family that happens to be clean.
- CLAIM-263. Combing across models is the kind of match CLAIM-263’s test presupposes can fail. Here it is observed, not tested.
- Accounts filed. THEORY-1028 (Active): the measured behaviour of the 3b family. THEORY-1022 (Proposed): the authors’ full claim that these units detect curves causally, generally and purely.
No instruction for machine-learning practice; the anthology may want the
work as a reading (anthology-candidate).
Limitations
- One unit for the dataset analysis, and one labeller, whose labelling the authors describe as hard (“many images seemed to fall within the boundaries of the rubric”) and affected by afterimages.
- The weak tail is not explained. Most firings are weak, and most images labelled curves fall where the unit fires weakly or not at all. The authors attribute this “at least partly” to labelling error and the rarity of curves, and give no precision–recall analysis.
- Purity is not shown here. The article’s own words are that any secondary function “would need to be rare”, and the mechanistic evidence is deferred.
- Feature visualization’s causal reading is overstated as Zoom In’s was (NOTE-1182). An optimized input shows that curves can drive the unit, not that curves are what drive it on natural images. The attribution and labelling arguments carry that.
- The figures were not rendered in this reading, so no number was taken from a chart.
- Internal inconsistencies.
- The opening section says curve detectors fire on “only 10% of spatial positions”; Dataset Analysis and its caption say 11%.
- Curve detectors are “only a couple convolutional layers deep”, and in the same section “implemented using five convolutional layers”. mixed3b is the fifth convolutional layer (conv2d0, conv2d1, conv2d2, 3a, 3b), so the second is right.
- Typos with no effect on content: “syntheti stimuli”, “our cure neuron”, “Perpindicular”.
- The applications are prototypes with no quantitative comparison, as the authors say.
Open questions
- What fraction of mixed3b’s 480 units, or of its activation variance, is accounted for by families as well understood as this one?
- What do the weak firings encode? Are they curve evidence the labeller missed, or a secondary function too diffuse to see in a quilt?
- Does the curve family survive a change of basis, for example a sparse dictionary over mixed3b, or does it split or merge?
- Is combing a feature of the data or a shortcut of the circuit? The authors’ own suggestion is to test it in adversarially robust models.
Note of 2026-10-27: Early Vision and Curve Circuits, now held
The bearing on THEORY-1013 says the thread’s overview of early vision is “proposed for registration”. It was registered and read on 2026-10-27, at the owner’s request (“file the next readings”), as LIT-1624 (NOTE-1206). Its census of conv2d0 to mixed3b (THEORY-1042) confirms the counts used here: the curve family is 10 of mixed3b’s 480 units, and mixed3a’s curves group holds 11. conv2d2’s “tiny curves” group holds 12, one of them a circle, against the 11 curve units the opening figure labels in that layer. The census is the denominator this note found missing, though not the criterion (the groups are sorted by eye).
The second of the planned curve articles, on “how they’re built from earlier neurons”, is Curve Circuits, registered and read the same day as LIT-1629 (NOTE-1207). It supplies the circuit, a downstream use and the hand-written reimplementation this article defers, with one weight edit tested (THEORY-1038). The third, on “prevalence across models”, is not registered. The reading is unchanged.