anthology-of-the-sota

SOTA-146: Correct the GRPO objective before running it — the published form has three independently identified defects

2 min read

Properties6
StatusProposed
Consensusemerging
Tagsadaptation-and-tuning
SourceLIT-167 — Understanding R1-Zero-Like Training: A Critical Perspective, LIT-168 — DAPO: An Open-Source LLM Reinforcement Learning System at Scale, LIT-180 — Group Sequence Policy Optimization
Introduced byLIT-167 — Understanding R1-Zero-Like Training: A Critical Perspective
CorrectsSOTA-145 — Estimate the RL baseline from a group of samples for the same prompt instead of training a critic

Graph View

SOTA-145 is what everyone kept. This is what everyone changed, and they did not change the same thing.

paperthe defectthe fix
LIT-167an optimization bias that inflates response length, especially on incorrect outputsDr. GRPO removes the bias; token efficiency improves at held accuracy
LIT-168clipping range and sampling behaviour at scaledecoupled clip and dynamic sampling; 50 points on AIME 2024 from a Qwen2.5-32B base
LIT-180the importance ratio defined per tokendefine it on sequence likelihood; clip, reward and optimise at sequence level

Why this is Proposed and not Active

Because the record cannot tell you which correction to run, and a practice that says “correct it somehow” is not yet advice.

Each of the three is well-evidenced on its own terms and none of them is tested against the others. They are not even obviously alternatives: LIT-167’s length bias and LIT-180’s token-level ratio are different parts of the objective, and a recipe could plausibly want both. Nobody has published that recipe.

The LIT-167 finding is the one worth reading twice. “The model learned to think longer” is the usual reading of a length increase under RL; part of that increase is the objective rewarding verbosity on answers that were wrong.

Why emerging rather than contested

The three groups are not arguing with each other. Each found a real defect, published a real fix, and left the others alone — which is a field converging on “this objective needs work” while nothing yet converges on which work.

contested would be the wrong word and the record now enforces the difference: ADR-016 requires a contested practice to name what contests it, and there is nothing honest to put there. Three sources agreeing that a thing is broken are not evidence against the practice that says so.

What would change this

The promote_when: above asks for a comparison rather than a fourth paper. LIT-180 is the one most likely to force the issue: it argues the standard objective is unstable specifically on mixture-of-experts models, and every frontier model this record holds is one.

Backlinks