Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Tiny models — claims that hold at the small end and not in general — sub-billion-parameter training, where the usual scaling advice inverts.
12 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-121 v3 | Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Stable at nearly the same optimal LR as AdamW, better evaluations; used for every Falcon-H1-Tiny model. | Active |
| SOTA-122 v2 | Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Up to 20% relative gain on MMLU, BBH and GSM8K over a Muon baseline at 200 GT. | Proposed |
| SOTA-123 | For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 90M, 25% SFT data in the pretraining mix beat a separate SFT stage by 10 IFEval points and yields one checkpoint that is both base and instruct. | Active |
| SOTA-124 v5 | Repeat high-quality data freely when its epoch size exceeds the model’s memorization window | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Roughly 100–500 GT for a 7B model, scaling linearly; the authors call the understanding early. | Proposed |
| SOTA-125 v3 | At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. 24–27 layers × 512 hidden and a large SSM state dimension won every ablation at 90M; 50 layers gained MMLU but halved throughput. | Proposed |
| SOTA-126 | Run DPO on tiny models for one epoch only | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. One epoch at LR 1e-6 to 3e-6 took a 90M model’s IFEval from about 50 to over 65; a second epoch degraded it while the reward kept rising. | Active |
| SOTA-127 v2 | Filter chain-of-thought traces out of the training data of tiny specialized models | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Reasoning traces interleaved with tool-calling data sent a 90M model into repetition loops; removing them fixed it at once. | Active |
| SOTA-128 | Train fill-in-the-middle data without masking the loss on non-FIM tokens | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 80 GT and 90M, computing the loss on every token beat masking the prefix and suffix; matches what the reference FIM recipes appear to do. | Active |
| SOTA-293 | Count embedding parameters when you report a tiny model’s size | Finke et al. (2025), LIT-485 — at these sizes the embedding table is not a rounding error, it is most of the model: TinyStories-33M is 68M parameters all-in, and an 8M model in the same paper is “more than half embeddings”. Two papers can each claim the smaller model and both be telling the truth. Print the all-in figure. | Proposed |
| SOTA-294 | Fit the tokenizer to the corpus when the corpus is deliberately narrow | Finke et al. (2025), LIT-485 — holding the dataset and the architecture fixed and changing only the tokenizer, a 4,096-token WordPiece vocabulary seeded with English affixes beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality. It is the one fully controlled comparison in a paper that is mostly about something else, and the effect is larger than the architecture change measured beside it. | Proposed |
| SOTA-295 | Parameterize the generating prompt above the word list, and constrain how each sample opens | Finke et al. (2025), LIT-485 — requiring a story to contain three words drawn from a vocabulary constrains the lexicon and leaves the frame alone, and the frame is where a generator repeats itself: 59.38% of TinyStories begins “once upon a time”. Name a theme, a topic, a style and a narrative feature per sample, and require the sample to open with a given part of speech and letter. Five diversity metrics move; measured simplicity does not. | Proposed |
| SOTA-419 | Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count | Geiping and Goldstein (2023), LIT-681. Under a 24-hour, one-GPU budget, every reshape tried (depth 4 to 24, deep-narrow, wide, funnel, FFN-every-k, shared layers) finishes within about 0.1 of the same MLM loss. A smaller model’s extra throughput cancels its slower per-token learning. So the architecture changes that pay are the ones that make each step cheaper without changing how much the model learns per token: dropping biases, predicting only masked tokens, and pre-norm for the learning rate it permits. | Proposed |