Evci et al. (2020) — ARXIV-2010.03533
Key takeaways
- The question is why sparse networks trained from a random initialization do worse than the same structure obtained by pruning a dense one. The answer offered is gradient flow: sparse networks at initialization have poor flow, and that is what training from scratch has to overcome.
- The claim about lottery tickets is deflationary and is the reason to read this alongside LIT-019. A rewound ticket re-learns the same solution the dense pruning run had already found — it lands in the same basin — rather than discovering that the sparse structure was independently a good architecture.
- That reframes the hypothesis from “sparse architectures are findable” to “the initialization carries information about the solution the dense run reached”, which is a much weaker and much more specific statement.
Standing in the anthology
Carries no practice, deliberately. It is analysis of a phenomenon the record holds no recommendation about, and its own conclusion is a reason not to expect a recipe from that phenomenon.
Filed as a real paper rather than retired because it is what stops LIT-019 being read as more than it is. A corpus containing the lottery-ticket hypothesis and not its deflation would be carrying a claim the field has since qualified, with nothing beside it saying so.
The note also carried the wrong first author for a year — attributed to Frankle, presumably because it is a lottery-ticket paper and Frankle wrote the lottery-ticket paper. That is the failure mode of describing a work from its subject: the description is plausible in every respect except being true.