Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Tiny models — claims that hold at the small end and not in general — sub-billion-parameter training, where the usual scaling advice inverts.
8 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-119 | Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale | Falcon-LLM Team (2026), technical blogpost. Holds the parameter budget at 90M and pulls the other three levers — data strategy, architecture, optimizer — one at a time: pretraining from scratch on the target SFT or reasoning mix beats pretrain-then-finetune at this scale, Muon with learnable multipliers beats the AdamW recipe, and depth and SSM width beat MLP width. | Active |
| LIT-123 | Wait, Wait, Wait… Why Do Reasoning Models Loop? | Pipis et al. (2025), ARXIV-2512.12895. Reasoning models loop — repeat the same text — at low temperature; larger models loop less and distilled students loop far more than their teachers, which the paper traces to errors in learning and, on a synthetic task, to risk aversion: when the progress-making action is hard to learn and a cyclic one is easy, probability shifts to the cycle. | Active |
| LIT-129 | Can Tiny Language Models Reason? | Jakimovski (2025), a write-up with code, no paper. SmolLM2-135M-Instruct taken through SFT on plain dialogue, SFT on reasoning traces in think tags, and one epoch of DPO on 50k preference pairs, on a single GPU; the DPO stage converges and the final model gains 3 to 9 points over its base on ARC-C, BBH, MMLU and IFEval. | Active |
| LIT-427 | SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model | Ben Allal et al. (2025), ARXIV-2502.02737. The model report behind SmolLM2-135M-Instruct, which is the base LIT-129’s whole experiment runs on — and the source of the overtraining ratio that makes a 135M model worth fine-tuning at all. | Active |
| LIT-484 v2 | TinyStories: How Small Can Language Models Be and Still Speak Coherent English? | Eldan and Li (2023), ARXIV-2305.07759. Restrict the training corpus to what a three-year-old’s vocabulary can say and a model below 10M parameters — or one with a single transformer block — writes fluent, grammatical, multi-paragraph stories. The finding is about the corpus, not the architecture: what stops small models speaking is the breadth of what they are asked to learn, not the complexity of language. | Active |
| LIT-485 | Parameterized Synthetic Text Generation with SimpleStories | Finke et al. (2025), ARXIV-2504.09184. Sampling three words from a child’s vocabulary was not enough: 59.38% of TinyStories begins “once upon a time”. Parameterize the generating prompt by theme, topic, style, narrative feature, grammar feature and author persona, and constrain the opening part of speech and letter, and every diversity metric moves. The one clean ablation in the paper is about something else — a 4,096-token domain tokenizer beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality at fixed data and architecture. | Active |
| LIT-681 | Cramming: Training a Language Model on a Single GPU in One Day | Geiping and Goldstein (ICML 2023), ARXIV-2212.14034. A BERT-style MLM trained from scratch for 24 hours on one consumer GPU, which reaches 79.2–80.4 GLUE against BERT-base’s 80.9. The transferable finding is negative: model shape does not beat the scaling law at this budget. Every reshape it tried lands near the same final loss, because a smaller model’s extra throughput cancels its slower learning. So the only architectural lever is to cut step time at constant parameter count. Its data result is the cheapest filter in the record: drop text the tokenizer compresses badly. | Active |
| LIT-719 | Discovering cognitive strategies with tiny recurrent neural networks | Ji-An et al. (2023), DOI-10.1038/s41586-025-09142-4. GRUs with only 1–4 units predict individual animals’ and humans’ choices in six reward-learning tasks better than 30+ classical cognitive models of the same dimensionality. Because they are so small, they can be read as dynamical systems, exposing strategies (state-dependent learning rates, perseveration, reward-induced indifference) that no classical model contains. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |