Frankle and Carbin (2018) — ARXIV-1803.03635
Key takeaways
- The hypothesis: a randomly-initialized dense network contains a subnetwork which, trained in isolation from the same initialization, matches the full network’s accuracy in at most the same number of iterations.
- The reset is the whole claim. Prune after training, rewind the survivors to their original values, retrain: that works. Reinitialize the same sparse structure randomly instead: it does not. So the finding is about the pairing of a structure with an initialization, not about the structure being a better architecture.
- Found by iterative magnitude pruning, and the tickets are small — 10–20% of the original parameters on the paper’s vision benchmarks, sometimes learning faster and generalizing better.
- The procedure needs the dense training run first, which is what stops it being a way to train cheaply. It is an existence result about what a dense run produces.
Key takeaways — what this does not say
- It does not say you can find the ticket without training the dense network.
- It does not report at language-model scale; the results are small vision networks, and later work at scale needed rewinding to an early-training point rather than to initialization.
Standing in the anthology
Carries no practice, deliberately. The record holds no sparsity or pruning recommendation, and this paper is not the one that would create one: the procedure it describes costs a full dense training run before it produces anything, so as an instruction it reads “train the model, then train a smaller one” — which is a finding about neural networks rather than advice about how to train one.
That the record holds no pruning practice at all is a real gap rather than a
scope decision — sparsity is expressible in model-architecture and there is
live work on it. This note is simply not the source that would close it.
LIT-039 is the companion analysis, asking why the tickets win.