anthology-of-the-sota

LIT-054: GLM-130B: An Open Bilingual Pre-trained Model

2 min read

Properties5
StatusActive
Version2
Tagsmodel-stability
Cited as “introduced by” bySOTA-069 — Monitor exp(loss) for stability, SOTA-070 — Track gradient norm ratios between layers, SOTA-071 — Use gradient clipping with dynamic threshold, SOTA-099 — Track gradient norm statistics to detect training instabilities
Cited as “source” bySOTA-069 — Monitor exp(loss) for stability, SOTA-070 — Track gradient norm ratios between layers, SOTA-071 — Use gradient clipping with dynamic threshold, SOTA-072 — Implement early warning system for NaNs, SOTA-099 — Track gradient norm statistics to detect training instabilities

Graph View

Zeng et al., Tsinghua and Zhipu.AI (2022) — ARXIV-2210.02414

Key takeaways

  • A 130B-parameter bilingual model, open-sourced together with its training logs and its “lessons learned” — which is the reason it is in the record. The paper describes itself as unveiling how a model of this scale can be successfully pre-trained, and names loss spikes and divergence as the numerous unexpected challenges it faced.
  • Stability is treated as a design problem with training strategies attached, not as bad luck: the choices that make the run survive are reported next to the choices that make it efficient.
  • On benchmarks it outperforms GPT-3 175B (davinci) across a range of English tasks — an advantage the authors note is not observed for OPT-175B or BLOOM-176B, which is a claim about the training recipe rather than the scale.
  • A scaling property of the model permits INT4 quantization with no post-training and almost no loss, making it the first 100B-scale model runnable on 4×RTX 3090 or 8×RTX 2080 Ti.

Standing in the anthology

The source of the record’s instability practices: SOTA-069 (monitor the loss on a scale where a spike is visible early), SOTA-070 (track gradient norm ratios between layers), SOTA-071 (clip with a threshold that adapts, extending LIT-037) and SOTA-072 (catch NaNs before they propagate). It also carries SOTA-099, gradient-norm statistics as an instability detector.

These practices reached the record with a fabricated citation, which is a particular shame for this subject: instability handling is folklore precisely because most runs do not publish their failures. GLM-130B and PaLM (LIT-069) are in the record because they did.

Backlinks