Qiu et al. (2025) — ARXIV-2509.24372
Key takeaways
It overturns an assumption rather than beating a number. The belief was that searching a billion-dimensional parameter space directly is hopeless, so prior ES work on LLMs reduced the dimension — last layer only, or a low-rank subspace. This searches the full parameter space and reports the first successful application at that scale.
What it takes from OpenAI ES, and what it changes. The algorithm follows the design of Salimans et al. (LIT-242): fixed-covariance Gaussian perturbations, and perturbations reconstructed from shared random seeds so that only seeds and scalar rewards cross between processes. What it adds is scale, applying that estimator to every parameter of a billion-parameter model, with layer-by-layer in-place perturbation so the population fits in GPU memory. What it removes is LIT-242’s set of enhancements: no rank transformation of rewards, no mirrored sampling, no weight decay, no virtual batch normalization, no Adam. And the population is 30 rather than the 10,000 or more the earlier work used on networks with millions of parameters.
The comparison is deliberately tilted against itself, and that is the result worth carrying. ES ran with one fixed hyperparameter set across every experiment. RL got a per-experiment grid over the KL penalty β and the learning rate α, because the authors found RL “did not make much progress if they were not set precisely” and chose the best configuration each time. Averaged across models, ES improves over the base by 36.4%, PPO by 17.9%, GRPO by 21.3% at group size 8 and 21.4% at group size 30 — a conservative comparison that ES wins anyway. GRPO here is DeepSeekMath’s critic-free PPO variant (LIT-127), run on Countdown with its KL penalty and learning rate grid-searched per experiment.
Where the advantage is claimed to come from. Tolerance to long-horizon and delayed rewards; robustness across different base models, which is the failure the paper’s own citation of Gandhi et al. establishes for RL on this task; reduced susceptibility to reward hacking; and steadier training. Being backpropagation-free is the structural difference underneath all four.
What the evidence does not cover
Two tasks — Countdown, a reasoning puzzle, and a conciseness objective — at 8B and below, with no frontier deployment. That is a long way from the post-training this record describes, which runs multi-stage on models an order of magnitude larger. The claim to have scaled ES is well supported; the claim that it should replace RL in a production recipe is not made here and should not be read in.
What later comparisons found
Three later papers ran this method against GRPO, and each changes what its comparison means.
- Gan and Isola (LIT-233) used ES as a baseline for RandOpt, at matched training FLOPs on seven tasks across Qwen2.5, OLMo3 and Llama-3.1. ES ran a single configuration while PPO and GRPO were grid-searched, and ES came out ahead in most cells. On Countdown with OLMo3-7B-Instruct it scored 71.0% against GRPO’s 68.5%. RandOpt’s 85.0% beat both, but only with 50-way voting. Yulu Gan is this paper’s second author, so it is a further result from an overlapping group, not an outside replication.
- Hoy et al. (LIT-235) followed this paper’s ES setup on Qwen3-4B-Instruct across four tasks, with both arms swept. ES at 300 iterations gave the best peak accuracy on all four, for example Chemistry 76.5% against GRPO’s 74.9%. They also checked this paper’s low-KL claim. It held for single-task Countdown, 0.21 nats against 0.38, but not in sequential training: there ES shifted other tasks far more, with Math training moving Chemistry by 0.62 nats against GRPO’s 0.01.
- Ba et al. (LIT-230) used this paper’s reward standardization and asked what each method buys rather than which wins. GRPO had the larger average pass@1 gain. ES raised pass@1, pass@16 and pass@32 together, while GRPO fell below its base model on both pass@16 and pass@32 in 15 of 18 comparisons. So the accuracy win reported here is one operating point. On average pass@1 alone, GRPO came out ahead in that study.
Standing in the anthology
A rival to the record’s whole post-training spine, and the first one.
SOTA-129 makes reinforcement learning with verifiable rewards the third stage
of the reasoning recipe; SOTA-145 recommends the group baseline inside it;
SOTA-146 corrects the objective. All three assume the paradigm. This is a
different paradigm reaching the same goal, filed as SOTA-154 and
Proposed, and recorded as compared_against: SOTA-145 — the relation for a
rival somebody measured rather than one asserted.
It does not disturb SOTA-145’s argument, which is worth being precise about. That practice observes that every work which runs or reworks GRPO keeps the group baseline. ES does not run GRPO, so it is not a counterexample — it is evidence about whether to be in that family at all.
One detail bears on SOTA-145 directly and mildly: raising GRPO’s group size from 8 to 30 moved the average by 0.1 points here. That is one task, but it is a data point on what the group baseline’s width buys.