anthology-of-the-sota

LIT-069: PaLM: Scaling Language Modeling with Pathways

1 min read

Properties4
StatusActive
Tagsmodel-architecture, training-optimization
Cited as “introduced by” bySOTA-095 — consider rewinding to earlier checkpoint and skipping a few batches to mitigate unusual loss spikes, SOTA-098 — Monitor the training loss for unexpected spikes, for the whole run
Cited as “source” bySOTA-092 — smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training, SOTA-093 — larger batch sizes are beneficial later in training due to better gradient estimates, SOTA-094 — count accelerator efficiency alongside sample efficiency when choosing a batch size: larger batches mean larger matrix multiplications, SOTA-095 — consider rewinding to earlier checkpoint and skipping a few batches to mitigate unusual loss spikes, SOTA-098 — Monitor the training loss for unexpected spikes, for the whole run

Graph View

Chowdhery et al. (2022) — ARXIV-2204.02311

Key takeaways

  • Pathways system architecture
  • Scaling to 540B parameters
  • Novel model parallelism techniques
  • Emergent capabilities analysis
  • Efficient scaling to 540B parameters
  • Emerging capabilities analysis
  • Novel training optimizations

Backlinks