Optimization (2) — optimizers, schedules, batch size: 001 · 002
Stability (0) — normalization, initialization, gradient handling.
Systems (0) — hardware, kernels, memory.
By status: Current practice (1) · Not yet judged (1) · Replaced (0)
What the status column means in this scheme — the words are luria’s, the meanings are this project’s.
| Status | Means | |
|---|---|---|
Active | Current practice | What you should do today, and the reason is one click away |
Deferred | Not yet judged | Probably right, not yet worth asserting |
Superseded | Replaced | Something newer does this better, and the record says what |
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-001 | Use a learning rate between 1e-4 and 1e-3 for most tasks | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-002 | Prefer decoupled weight decay to L2 regularization added to the loss | Deferred rather than Active: not a doubt about the result, a doubt about the practice. | Deferred — the finding is settled; how widely it is adopted is not |