capability-thresholds.
9 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-052 v2 | Progress measures for grokking via mechanistic interpretability | Reverse-engineers a one-layer transformer trained on modular addition and finds it computes a Fourier multiplication algorithm — embed inputs as rotations, combine with trigonometric identities. Uses that to define continuous progress measures showing grokking is three phases, and finds grokking vanishes entirely above about 60% data. | Read |
| NOTE-219 | Emergent Abilities | The paper that named the phenomenon, read for what it actually argues rather than what it is cited for. It raises the metric explanation itself, in §5.1, and declines it on two grounds — one of which was later answered and one of which still stands. | Read |
| NOTE-220 | Emergence as a Mirage | Reading it: the strongest evidence is not the rebuttal of GPT-3’s arithmetic curve but the vision experiments, where emergence is manufactured in models that had never shown it. The weakest part is what “mirage” gets read as — the paper explicitly declines the claim that emergence is impossible. | Read |
| NOTE-281 | The founding paper already measured the regime dependence | Read because the record staked a claim on grokking being a data-starved-regime phenomenon while holding only the paper that put a number on it. The founding paper measured the dependence too, and reported it as a headline: converged accuracy is flat across a range of training-set sizes while time-to-generalize explodes as the set shrinks. It also names weight decay as the strongest intervention it found, which is the thread every later mechanism pulls on. | Read |
| NOTE-282 | Induced on images, text and molecules, and eliminated on demand | Read to settle whether grokking is specific to algorithmic data. It is not — but producing it on MNIST, IMDb and QM9 takes two deliberate departures from standard practice: a much smaller training set and an inflated initialization scale. At standard initialization there is no grokking on any of them. The control runs both ways: constraining the weight norm nearly eliminates grokking on algorithmic data. | Read |
| NOTE-283 | Two novel phenomena predicted from the theory and then observed | Read as the mechanistic account with the best evidential shape in this cluster. From “weight decay prefers the circuit with more logit per unit norm, and memorisation gets less efficient as the dataset grows” it derives a critical dataset size, and from that derives ungrokking — a grokked network losing its generalization when trained on less data — and semi-grokking. Both were then observed. The mechanism needs weight decay, which is where the counterexample lands. | Read |
| NOTE-284 | A counterexample with no weight decay and a rising weight norm | Read because it is the reason this cluster cannot be filed as a single account. A two-layer MLP groks on modular arithmetic with no weight decay and a parameter norm that goes up through the transition — which neither the LU mechanism nor circuit efficiency can produce, since both turn on the norm coming down. The replacement offered is a transition out of lazy dynamics, with two measurable knobs; what it does not address is ungrokking. | Read |
| NOTE-287 | Stage boundaries found without knowing in advance what to look for | Read as the application that makes the LLC matter to this record. Critical points of the LLC curve divide two transformers into five stages each, and the boundaries land on bigrams, n-grams, previous-token heads and the induction circuit — found without a mechanistic hypothesis in hand, which is what LIT-085’s progress measures require. Its in-context regression model acquires in-context learning and then loses it. | Read |
| NOTE-288 | The transition the posterior makes and the transition SGD makes are not the same transition | Read for the distinction in its title, which the record needed and did not have. Bayesian phase transitions are changes in where the posterior concentrates as sample size grows, and are well defined; dynamical transitions are changes in the SGD trajectory, and are not. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600 and observed at 600–700 — has no dynamical counterpart at all. | Read |