anthology-of-the-sota

LIT-127: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

3 min read

Properties8
StatusActive
Version2
Tagsadaptation-and-tuning, training-optimization, data-pipeline
ExtendsLIT-241 — Proximal Policy Optimization Algorithms
Extended byLIT-779 — Flow-GRPO: Training Flow Matching Models via Online RL
Compared againstLIT-211 — Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning, LIT-233 — Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights, LIT-230 — Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO, LIT-235 — Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training, LIT-237 — Evolutionary Strategies lead to Catastrophic Forgetting in LLMs, LIT-239 — ESSA: Evolutionary Strategies for Scalable Alignment, LIT-240 — ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
Cited as “introduced by” bySOTA-145 — Estimate the RL baseline from a group of samples for the same prompt instead of training a critic
Cited as “source” bySOTA-145 — Estimate the RL baseline from a group of samples for the same prompt instead of training a critic

Graph View

Shao et al. (2024) — ARXIV-2402.03300

Key takeaways

  • DeepSeekMath 7B continues pretraining DeepSeek-Coder-Base-v1.5 7B on 120B math-related tokens selected from Common Crawl by an engineered pipeline, mixed with natural language and code, and reaches 51.7% on competition-level MATH without external tools or voting.
  • Group Relative Policy Optimization: a PPO variant that drops the value model and estimates the baseline from the scores of a group of outputs sampled for the same prompt, which cuts the memory footprint of RL and improves mathematical reasoning.
  • A unified view of the RL-style methods — rejection-sampling fine-tuning, DPO, PPO, GRPO — as variants of one paradigm, used to reason about what each is actually optimising.

Standing in the anthology

Filed for GRPO, which LIT-119 runs on its 0.6B reasoning model and finds sensitive above all to the learning rate. The data-selection half of the

paper is the same idea SOTA-124 generalises: mine and repeat the high-quality slice.

GRPO is PPO (LIT-241) with one term taken out. It keeps the clipped probability-ratio surrogate and drops the learned value function, taking the baseline from the scores of a group of outputs sampled for the same prompt. Dropping the critic is what cuts RL’s memory footprint.

It has become the baseline the evolution-strategies line measures itself against:

  • LIT-211 ran full-parameter ES against PPO and GRPO with per-experiment RL tuning. On Countdown, averaged across models, ES improved over the base by 36.4% against 21.3% for GRPO at group size 8.
  • LIT-237 tried to reproduce that ordering on GSM8K, MATH and OlympiadBench at 1–1.5B, and found GRPO still ahead in three of four settings. Its GRPO updates were about 95% sparse, against dense ES updates with roughly 1000× the drift.
  • LIT-230 found that GRPO wins average pass@1 but falls below its own base model on both pass@16 and pass@32 in 15 of 18 comparisons, while ES raises all three.
  • LIT-235 ran both on Qwen3-4B. ES at 300 iterations had the higher peak accuracy on four tasks (Chemistry 76.5% against GRPO’s 74.9%), and the two solutions are linearly connected despite near-orthogonal update directions.
  • LIT-233’s one-step random-perturbation ensemble matched or beat GRPO at equal training FLOPs: 87.1% against 83.2% on GSM8K with Qwen2.5-3B-Instruct, though its answers are majority votes over fifty models.
  • LIT-239 (ESSA) passed 0.80 on GSM8K with Qwen2.5-Math-7B in under 25 minutes against GRPO’s 50 or more, from 100 examples against GRPO’s 5,978. The two arms also ran on different runtimes.
  • LIT-240 (ESSAM) tied GRPO on GSM8K across four Qwen2.5 sizes (78.27% against 78.34%) at inference-level memory, reported as a 10–18× reduction.

Backlinks