anthology-of-the-sota

LIT-027: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

1 min read

Properties5
StatusActive
Tagsdistributed-optimization
Cited as “introduced by” bySOTA-028 — Stage optimizer states across data parallel ranks (ZeRO-1), SOTA-029 — Partition gradients and optimizer states (ZeRO-2) for larger models, SOTA-030 — Use ZeRO-3 only when other strategies insufficient, SOTA-031 — Keep micro-batch size per GPU as large as memory allows
Cited as “paper” byNOTE-103 — ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Cited as “source” bySOTA-028 — Stage optimizer states across data parallel ranks (ZeRO-1), SOTA-029 — Partition gradients and optimizer states (ZeRO-2) for larger models, SOTA-030 — Use ZeRO-3 only when other strategies insufficient, SOTA-031 — Keep micro-batch size per GPU as large as memory allows

Graph View

Rajbhandari et al. (2020) — ARXIV-1910.02054

Key takeaways

  • Micro-batch size determined by GPU memory
  • Gradient accumulation steps determined by target global batch
  • Introduced DeepSpeed ZeRo optimizations
  • Large batch size training w/o model parallelism

Backlinks