anthology-of-the-sota
Search
Search
Dark mode
Light mode
Reader mode
Explorer
tiny-models
Graph View
20 items with this tag.
Sep 25, 2026
LIT-681: Cramming: Training a Language Model on a Single GPU in One Day
training-optimization
tiny-models
model-architecture
data-pipeline
model-stability
analysis-and-evaluation
Sep 25, 2026
LIT-719: Discovering cognitive strategies with tiny recurrent neural networks
analysis-and-evaluation
tiny-models
Sep 25, 2026
SOTA-419: Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count
model-architecture
training-optimization
tiny-models
Sep 21, 2026
LIT-484: TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
tiny-models
data-pipeline
analysis-and-evaluation
signal-structure
Sep 21, 2026
LIT-485: Parameterized Synthetic Text Generation with SimpleStories
data-pipeline
tiny-models
representation-and-encoding
analysis-and-evaluation
Sep 21, 2026
SOTA-293: Count embedding parameters when you report a tiny model's size
tiny-models
analysis-and-evaluation
Sep 21, 2026
SOTA-294: Fit the tokenizer to the corpus when the corpus is deliberately narrow
representation-and-encoding
tiny-models
Sep 21, 2026
SOTA-295: Parameterize the generating prompt above the word list, and constrain how each sample opens
data-pipeline
tiny-models
Sep 18, 2026
LIT-427: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
tiny-models
data-pipeline
analysis-and-evaluation
Sep 05, 2026
LIT-119: Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale
training-optimization
tiny-models
adaptation-and-tuning
model-stability
Sep 05, 2026
LIT-123: Wait, Wait, Wait... Why Do Reasoning Models Loop?
analysis-and-evaluation
tiny-models
Sep 05, 2026
LIT-129: Can Tiny Language Models Reason?
adaptation-and-tuning
tiny-models
in-context-learning
Sep 05, 2026
SOTA-121: Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW
training-optimization
tiny-models
Sep 05, 2026
SOTA-122: Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD
training-optimization
tiny-models
Sep 05, 2026
SOTA-123: For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune
adaptation-and-tuning
tiny-models
Sep 05, 2026
SOTA-124: Repeat high-quality data freely when its epoch size exceeds the model's memorization window
data-pipeline
tiny-models
model-architecture
Sep 05, 2026
SOTA-125: At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width
model-architecture
tiny-models
Sep 05, 2026
SOTA-126: Run DPO on tiny models for one epoch only
adaptation-and-tuning
tiny-models
Sep 05, 2026
SOTA-127: Filter chain-of-thought traces out of the training data of tiny specialized models
data-pipeline
tiny-models
in-context-learning
Sep 05, 2026
SOTA-128: Train fill-in-the-middle data without masking the loss on non-FIM tokens
data-pipeline
tiny-models