NOTE-233: TinyStories

Contribution

Before this, “small models cannot produce coherent text” was a fact with two available explanations and no way to choose between them: natural language is intrinsically too hard below some scale, or web-scale corpora ask a small model to learn English and an encyclopaedia simultaneously. This paper builds a corpus that removes the encyclopaedia and keeps the English, and the small models start speaking. That converts a scale claim into a data claim.

It also introduced, as instrumentation, grading generated text with a large model on separate axes — what the field later called model-as-a-judge — on the argument that the benchmarks available in 2023 could not score free generation at all.

Key insight

Coherence was never the expensive part; breadth was. A model with an embedding dimension of 256 has enough capacity for grammar, discourse and simple reasoning, and does not have enough for the factual mass of the Pile. Train it on a corpus whose vocabulary and world are a three-year-old’s and what was missing turns out not to have been linguistic competence.

The corollary is the one that made this a research instrument rather than a curiosity: you can now have a model organism — something small enough to interpret and real enough to behave like the thing you care about.

Assumptions

• The dataset is generated by GPT-3.5 and GPT-4, so the target distribution is a commercial model’s idea of a children’s story. Everything downstream inherits whatever that is.
• Diversity has to be engineered in. Prompting for stories at high temperature yields a repetitive corpus; the mechanism here is three words (a noun, a verb, an adjective) drawn at random from a ~1,500-word vocabulary and required to appear, plus a random subset of features.
• GPT-Neo architecture, window size 256, context 512, the GPT-Neo tokenizer truncated to its top 10K tokens.
• Evaluation is a language model’s judgement, validated against the authors’ reading rather than against human raters at scale.

Key results

• Models below 10M parameters (embedding dimension 256) produce fluent, consistent, multi-paragraph stories with near-perfect grammar.
• A one-transformer-block model at 21M produces syntactically correct English but loses the plot — the one-layer case is where the separation between grammar and coherence is starkest.
• Reasoning of the disjunction-elimination kind appears at 2.5M.
• Factual knowledge tracks the embedding dimension; context-tracking and instruction-following track the number of layers. Grammar is what a shallow model gets right first.
• Training runs in under a day on a single GPU, which is what made the hyperparameter sweeps in the paper possible at all.
• One- and two-layer models show attention heads separating cleanly into local and semantic, and MLP neurons firing on tokens with a shared role.

Claims

Well supported: that corpus breadth rather than linguistic complexity is what small models were failing on. The intervention is on the data, the architecture is held, and the effect is large enough not to need statistics.

Well supported and often overstated in citation: the depth/width separation. It is a real sweep across 1–8 layers and several embedding dimensions, and it is a sweep over one architecture family on one corpus with GPT-Eval as the instrument.

Anticipated and tested, not merely asserted: that the models are not memorizing templates. Word and n-gram overlap against the training set plus out-of-distribution prompts; the Figure 1 prompt was verified to have no 6-gram overlap.

Assumed rather than shown: that the generation procedure produced a diverse corpus. The paper argues the mechanism should work and does not measure the result. LIT-485 measured it, and 59.38% of the stories begin “once upon a time”.

A claim about the models, not the data: that small models are more interpretable. Demonstrated by inspection of one- and two-layer models rather than by a metric.

Method

Generate a synthetic corpus of short stories under a vocabulary constraint, train GPT-Neo models from ~1M to ~35M parameters and 1 to 8 layers on it, and grade completions with GPT-4 on separate axes. Sweep depth against embedding dimension, head count against width, and compute against model size.

Connections

• LIT-485 is the descendant that measures the diversity this one engineered, finds it insufficient in a specific way, and replaces the mechanism.
• SOTA-123 — pretrain tiny specialized models from scratch on the target data rather than pretrain-then-finetune — is the same intuition applied to a different corpus: at this size, what you train on is the specification.
• SOTA-127 — filter chain-of-thought traces out of a tiny model’s training data — reads as the same principle again, and this is the paper that says why removing material can help rather than hurt.
• SOTA-124 is about repeating high-quality data within a memorization window; this is the adjacent move of shrinking what has to be learned.

Bearing on the record

It is the trunk of the tiny-models line, which the record was eight practices into without holding.

It does not satisfy SOTA-125’s promotion condition, and the reason is worth stating rather than leaving to inference. That practice trades depth against MLP width in a hybrid at 90M and asks for a hybrid from another group or another 90M-class ablation. This sweeps depth against embedding dimension in pure-attention GPT-Neo at 1–35M and has no SSM state dimension to trade against at all. Two findings that share the words “depth” and “width” and are not the same measurement.

Limitations

One corpus, and a strange one. Everything is about a distribution designed to be simple. Whether the depth/width separation or the emergence scales transfer to a corpus with real factual mass is exactly what the paper removed in order to get its result.

The judge is a language model, on a task where the generating and grading distributions are both GPT-family.

“Below 10M parameters” excludes embeddings, which is not stated in the headline and is a live source of confusion — LIT-485 puts TinyStories-33M at 68M all-in and recommends the convention be changed.

Interpretability is argued by example. Clear attention patterns in a one-layer model are shown, not quantified against a baseline.

Open questions

• Does the corpus-breadth explanation have a quantitative form? The paper establishes the direction and puts no number on how much breadth costs how much capacity.
• What is the smallest model that speaks, counted honestly? The paper’s own answer excludes embeddings, and LIT-485 reopens the question under a stricter count.