anthology-of-the-sota

SOTA-145: Estimate the RL baseline from a group of samples for the same prompt instead of training a critic

5 min read

Properties8
StatusActive
Version4
Consensusconverged
Tagsadaptation-and-tuning
SourceLIT-127 — DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, LIT-119 — Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale, LIT-167 — Understanding R1-Zero-Like Training: A Critical Perspective, LIT-168 — DAPO: An Open-Source LLM Reinforcement Learning System at Scale, LIT-180 — Group Sequence Policy Optimization, LIT-779 — Flow-GRPO: Training Flow Matching Models via Online RL
Introduced byLIT-127 — DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Compared againstSOTA-154 — Fine-tune with evolution strategies instead of policy-gradient reinforcement learning, SOTA-212 — Post-train by scoring many random weight perturbations in one parallel pass and majority-voting the best of them
Corrected bySOTA-146 — Correct the GRPO objective before running it — the published form has three independently identified defects

Graph View

PPO needs a value model to tell it whether an outcome was better than expected. At LLM scale that critic is a second network of comparable size, trained alongside the policy, and it is the largest avoidable cost in the RL stage.

GRPO (LIT-127) removes it. Sample a group of outputs for the same prompt, score them, and use the group’s own statistics as the baseline: an output is good relative to its siblings rather than relative to a learned prediction. The critic disappears and the advantage estimate becomes a within-group comparison.

Why this is the settled part

The record holds four papers that run or rework GRPO, and all four keep the group baseline:

  • LIT-119 runs it on a 0.6B reasoning model and reports it sensitive above all to the learning rate — a tuning finding, not an objection to the method.
  • LIT-167 identifies a length bias in the objective and publishes Dr. GRPO to remove it. The group baseline stays.
  • LIT-168 decouples the clipping range and adds dynamic sampling. The group baseline stays.
  • LIT-180 moves the importance ratio from token to sequence level. The group baseline stays.

Three independent groups examined this objective closely enough to publish a correction to it, and none of them proposed bringing the critic back. That is what converged is recording here: not that nobody has looked, but that people looked hard and changed something else.

Outside language, and how big the group has to be

Flow-GRPO (LIT-779) runs the group baseline on an image generator. GRPO’s group-normalized advantage, clipped ratio and KL penalty are used unchanged on SD3.5-M with LoRA, after the flow model’s sampler is turned into a same-marginal SDE so that each step is a Gaussian policy. There is still no critic. GenEval rises from 0.63 to 0.95 and OCR accuracy from 0.59 to 0.92, on rewards it is also trained on.

It adds a condition the language papers did not state: the group has to be large enough for the baseline to hold. At 24 samples per prompt training was stable; at 12 and at 6 it collapsed on PickScore (its Fig. 5). One run per setting, on one task.

DDPO (LIT-776, App. D.3) is the earlier case, and a weaker one: it also drops the critic, but its baseline is a running mean and standard deviation of each prompt’s rewards across iterations, not a group sampled together, and it does not compare against a critic. Flow-GRPO’s per-prompt group is the within-step version of the same idea.

DPOK (LIT-778, App. A.5) went the other way and trained a critic. A learned V(x_t, z) raised ImageReward on one prompt from 0.86 to 1.51 against no baseline at all, and v3 uses it for multi-prompt training. That is a critic against nothing, not against a group or per-prompt baseline, so it does not bear on the choice this practice makes. Its v1 had reported the critic destabilizing multi-prompt training, and v3 dropped that without explanation.

What this does not say

It does not say run GRPO as published. That is SOTA-146, and it is Proposed for a reason: the same three papers that kept the group baseline each found a different defect in the rest of the objective, and no two of them fixed the same one.

It also does not say when to run RL. SOTA-129 is the

three-stage recipe and SOTA-130 the variation that skips the SFT stage; both name RLVR as a stage and neither names an algorithm. This is the algorithm, and it is orthogonal to that argument — the group baseline is what you run either way.

And it does not say policy-gradient RL is the right family. Evolution strategies (SOTA-154, Proposed) reach the same goal without backpropagation and report a larger average improvement than either PPO or GRPO on the one task they share — on a single fixed hyperparameter set, against RL tuned per experiment. That is not a counterexample to the argument above, which is about what works that run or rework GRPO keep; it is evidence about whether to be in that family at all, at 8B and below and on two tasks. The practice stands, and the assumption underneath it is now a named rival with a promotion condition rather than an assumption.

The second rival ran its comparison against this practice directly. RandOpt (SOTA-212) scores thousands of random weight perturbations in one parallel pass and majority-votes the best 50, and its paper matched it on training FLOPs against GRPO at 200 iterations across seven tasks at 0.5B–8B: it won most cells, 85.0% against GRPO’s 68.5% on Countdown with OLMo3-7B-Instruct and 87.1% against 83.2% on GSM8K with Qwen2.5-3B-Instruct. But GRPO answered with one sample against a 50-way ensemble, and the paper says so; the comparison is about whether to leave the family, not about whether to keep the group baseline inside it.

Backlinks