Jaderberg et al. (2017) — ARXIV-1711.09846
Key takeaways
- A population of models trains in parallel and asynchronously; each member periodically exploits (copies a better member’s weights and hyperparameters) then explores (perturbs them), continuing from the copied weights rather than restarting
- The output is a schedule, not a setting — because hyperparameters change mid-run, PBT expresses time-varying values no fixed-hyperparameter search can. On WMT En-De it rediscovers a small initial learning rate, a jump of ~3 orders of magnitude, then exponential decay
- Beats an already highly tuned Transformer baseline: validation BLEU 23.71 → 24.23, test 22.30 → 22.65, and trains faster
- Population of 20–40 is enough for consistent improvement; ≤10 shows high variance and local optima, since the algorithm is greedy
- The GAN experiment deliberately selects on CIFAR Inception score and reports ImageNet Inception score, to avoid selecting on the test metric
Standing in the anthology
Read — NOTE-041.
Nothing is sourced to this paper and the reading files no practice. PBT’s
economics do not survive the move to frontier scale — a population of 20–40
concurrent runs is the reason — and the record’s answer to the same problem is
µP (SOTA-144, SOTA-159): transfer the hyperparameters analytically from one
small sweep instead of running many large ones.
Kept for the question PBT asks that µP does not: should this hyperparameter be a schedule at all? The record has practices about what to set several hyperparameters to and few about which of them should be moving.