anthology-of-the-sota

LIT-167: Understanding R1-Zero-Like Training: A Critical Perspective

2 min read

Properties5
StatusActive
Tagsanalysis-and-evaluation, adaptation-and-tuning
Cited as “contested by” bySOTA-130 — Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model
Cited as “introduced by” bySOTA-146 — Correct the GRPO objective before running it — the published form has three independently identified defects
Cited as “source” bySOTA-145 — Estimate the RL baseline from a group of samples for the same prompt instead of training a critic, SOTA-146 — Correct the GRPO objective before running it — the published form has three independently identified defects

Graph View

Liu et al. (2025) — ARXIV-2503.20783

Key takeaways

  • Splits R1-Zero-style training into its two components and examines each. On the base model side: DeepSeek-V3-Base already exhibits the “Aha moment” before any RL, and Qwen2.5 bases show strong reasoning even without a prompt template — which the authors read as pretraining bias. Some of what looked emergent under RL was already there.
  • On the RL side: an optimization bias in GRPO that artificially inflates response length, and does so especially for incorrect outputs. That is a sharp finding, because “the model learned to think longer” is the usual reading of a length increase during RL, and here part of it is the objective rewarding verbosity on failures.
  • Dr. GRPO removes the bias, improving token efficiency while holding reasoning performance — a cheaper model at the same accuracy rather than a better one.
  • A minimalist R1-Zero recipe built on these findings reaches 43.3% on AIME 2024 with a 7B base.

Standing in the anthology

The corrective to LIT-164, and the reason SOTA-130 should stay Proposed: if part of the “emergence” is a property of the base model’s pretraining, then “run RLVR directly on the base” is a claim whose result depends on which base, and the record has one data point per base.

It also belongs to a sequence on LIT-127, which introduced GRPO. The variations the record now holds each name a defect and fix it: Dr. GRPO the length bias here, LIT-168 the clipping and sampling behaviour, and LIT-180 the token-level importance ratio. GRPO is the practice’s source (LIT-119 runs it at 0.6B); this is the literature saying it is not the last word.

Backlinks