anthology-of-the-sota

LIT-011: Mixed Precision Training

1 min read

Properties4
StatusActive
Tagsnumerics-and-precision, training-optimization
Cited as “introduced by” bySOTA-013 — Use dynamic loss scaling that doubles every 2000 successful steps, SOTA-014 — Maintain master weights in FP32, SOTA-015 — Store optimizer states in FP32, SOTA-016 — Perform forward/backward passes in FP16, SOTA-072 — Implement early warning system for NaNs
Cited as “source” bySOTA-013 — Use dynamic loss scaling that doubles every 2000 successful steps, SOTA-014 — Maintain master weights in FP32, SOTA-015 — Store optimizer states in FP32, SOTA-016 — Perform forward/backward passes in FP16

Graph View

Micikevicius et al. (2017) — ARXIV-1710.03740

Key takeaways

  • Enables FP16 training with FP32 master weights
  • Loss scaling for numerical stability
  • Memory and compute benefits
  • No accuracy degradation

Backlinks