Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Reinforcement — PPO · GRPO · RLVR

Signal = reward / verification. Fully on-policy, online, uses the whole reward landscape (push winners up and losers down). Most powerful, most expensive/unstable. This is the family driving every 2025–2026 reasoning model. Reinforcement is one of three fixed paradigm presets on the canonical on/off-policy axis (the other two are imitation and preference) — see Foundations for the axis itself and Contested edges §6 for why “three independent knobs you toggle / a combinatorial grid” is a retired teaching scaffold: the axes aren’t independent, so what differs across PPO/GRPO/GSPO/DAPO/RLVR is how the advantage is computed and where the reward comes from, not a free combination. In the staged recipe this is the last stage — the recipe is a sequence places RLVR/GRPO after cold-start SFT, rejection-sampling, and preference-opt; proven-first ranking has the T1 pick inside this family (RLVR-via-GRPO); is the recipe a loop? covers revisiting this stage across rounds.

PPO

  • What: clipped-surrogate policy gradient with a value/critic network + KL-to-reference (arXiv:1707.06347).
  • Eats: prompts + a reward (learned RM or verifier).
  • 2026 status: still used for classic preference-RL at the proprietary labs, but declining share for reasoning-RL (the critic is expensive; GRPO/GSPO replaced it there).

GRPO (Group Relative Policy Optimization)

  • What: drop the critic. Sample a group of N completions per prompt; the group mean reward is the baseline; advantage Aᵢ = rᵢ − mean(r) (optionally std-normalized). Introduced in DeepSeekMath (arXiv:2402.03300).
  • Eats: prompts + a reward fn; no fixed target dataset.
  • 2026 status: the reasoning-RL default — DeepSeek-R1’s core algorithm (arXiv:2501.12948), still the base of V3.2’s mixed RL (arXiv:2512.02556).
  • Requirement (project rule): baseline solve rate must sit in 30–60% per prompt-group — all-pass or all-fail groups give zero advantage → zero gradient (llmresearch-handbook.md rule 7; mechanics in handbook.md §10, shared memory).

GRPO successors

  • GSPO (Qwen) [R][L] — the problem with vanilla GRPO/PPO at scale: the token-level importance ratio r_t = π_θ(a_t|s_t)/π_old(a_t|s_t) compounds multiplicatively over a long response, and noisy per-token drift is specifically what destabilizes MoE RL (expert routing shifts mid-rollout, under-policy). GSPO clips at the sequence level instead:

    # GRPO/PPO: one ratio PER TOKEN, clipped per token — variance compounds over length L
r_t = pi_theta(a_t|s_t) / pi_old(a_t|s_t)

# GSPO: one ratio for the WHOLE sequence (length-normalized geometric mean)
r_seq = (pi_theta(y|x) / pi_old(y|x)) ** (1 / len(y))
loss  = -mean(min(r_seq * A, clip(r_seq, 1-eps, 1+eps) * A))

    This is Qwen3’s actual stated production RL algorithm (arXiv:2507.18071) — the first GRPO-successor with a flagship behind it, and it gets more relevant, not less, as episodes lengthen: a 100-turn agentic trajectory with tool calls interleaved is exactly the long-sequence regime where token-level ratios drift furthest from 1 by the last token. If a future GRPO/RLVR run on the CTF agent shows training instability, sequence-level clipping is the first thing to try — not more KL-coefficient tuning.

  • DAPO (ByteDance Seed) [E][R][L] — four concrete engineering fixes, not one new algorithm, each independently adoptable as a verl loss-mode flag (arXiv:2503.14476):

    1. Clip-Higher — decouple the PPO clip bounds (eps_low ≠ eps_high, e.g. 0.20 / 0.28 vs. the symmetric PPO-default 0.20/0.20) so a rare-but-good token can gain probability faster than a bad one loses it. Symmetric clipping caps how fast a rare-correct action can ever be reinforced — a direct driver of entropy collapse (below).
    2. Dynamic Sampling — resample any prompt whose whole group of G rollouts is all-correct or all-incorrect (std(group_rewards) == 0 → zero advantage → zero gradient in GRPO) instead of paying for a wasted rollout batch.
    3. Token-level loss — average the policy-gradient loss over every token in the batch, not per-sample-then-averaged, so long correct/incorrect responses aren’t down-weighted relative to short ones. [L] — a credit-assignment fix that matters more the longer responses get.
    4. Overlong reward shaping — a soft length penalty instead of a hard truncation penalty, so a response cut off by the context window isn’t punished as if it were simply wrong.
    eps_low, eps_high = 0.20, 0.28
ratio   = exp(logp_new - logp_old)
clipped = clip(ratio, 1 - eps_low, 1 + eps_high)
loss_pg = -min(ratio * adv, clipped * adv)          # per-token, mean over ALL tokens in the batch

while std(group_rewards) == 0:                       # dynamic sampling
    prompt = resample_prompt()
    group_rewards = rollout_and_score(prompt, n=G)

    Mainstream in OSS RL tooling (verl and open GRPO reproductions default to these four fixes) and independently reproduced as a 50-point AIME 2024 result beating R1-Zero-Qwen-32B with half the training steps — one of the few fully open (algorithm + infra + data) large-scale reasoning-RL reproductions. Project rule 7 (GRPO baseline must hit 30–60%) is DAPO’s dynamic-sampling problem, stated as a portfolio-composition constraint instead of a training-loop fallback — keeping the baseline in-band is how you avoid feeding all-pass/all-fail groups into the update in the first place; DAPO’s dynamic sampling is the fallback for whatever still lands there. At ~100 turns per rollout, resampling a whole-group-zero-reward challenge is expensive — prefer upstream curriculum/difficulty filtering (drop challenges outside the 30–60% band) over paying for resamples on genuinely-unsolved-yet challenges.

    Designed to fix a common failure: no-methodology pivoting after one failed attempt, and committing to a single ungrounded guess → Clip-Higher. Agentic policies commonly abandon a promising line of attack after a single setback and default to whichever guess is left standing when no path clearly dominates. Symmetric clipping caps how much probability mass a rare, correct enumeration branch (or a well-grounded, as opposed to lucky, guess) can ever accumulate — which is precisely what narrows a policy onto one brittle script. See Exploration and entropy below for the mechanism this is patching.

  • Dr. GRPO [R][E] — a smaller, easy-to-miss companion fix: vanilla GRPO’s per-sample length- and std-normalization secretly rewards longer wrong answers and shorter right ones (an optimization artifact, not a real preference). Fix is a two-line change — drop the 1/|response| length term and the group-std division, keep only A = r − mean(r) (arXiv:2503.20783). Matches or beats vanilla GRPO’s accuracy at the same compute while removing the length-inflation drift. For a 100-turn agent this bug has a much bigger attack surface than a single-turn math answer: a policy trained on the unfixed objective can learn to “look busy” (extra tool calls, redundant enumeration) after a wrong guess without the enumeration being useful — nearly indistinguishable from legitimate PTES-style enumeration unless you’re specifically checking for it.

RLVR (RL with Verifiable Rewards)

  • What: GRPO/PPO where the reward is a deterministic verifier (unit tests, math checker, flag check) rather than a neural RM. No parameters to game.
  • Eats: prompts + a verify(state) → {0,1} function. This is your setup — the CTF flag verifier is a textbook verifiable reward.
  • 2026 status: arguably the defining technique of the era. Every reasoning model (o1/o3, R1, Gemini-thinking, Qwen3, Kimi) scales RL against verifiable/rule-based rewards as the capability driver; Gemini 2.5 explicitly allocates increased RL compute to “verifiable rewards” (arXiv:2507.06261; OpenAI “Learning to reason with LLMs”; R1, arXiv:2501.12948).

Exploration and entropy: the GRPO graduation trigger

Cybersecurity is exploration — every technique in the PPO/GRPO/GSPO/DAPO family above is [E]-tagged, and entropy collapse is what turns “graduate SFT → GRPO/RLVR” from a vague heuristic into a measurable trigger. The graduation trigger, precisely stated: don’t wait for the reward curve to plateau — watch mean(entropy); once it’s tracking toward the flat part of the fitted collapse curve, more rejection-sampling-SFT epochs on the same policy distribution won’t move the needle (you’re re-sampling an already-narrowing distribution) — that’s the signal to graduate to GRPO/RLVR. Full mechanism (the fitted law, the covariance driver, Clip-Cov/KL-Cov, the clip-asymmetry result, high-entropy minority tokens, ProRL’s boundary-expansion evidence, and the amplify-vs-elicit contested-edges reconciliation): Long-horizon & exploration RL §2.1–2.7.

Everything past this point — pass@k as a training signal, diversity/curiosity/count-based intrinsic rewards, parameter-space noise for temporally-coherent exploration, tool-call-sequence diversity as the project’s own novel opportunity, and the full turn-level/step-level credit-assignment literature for the ~100-turn setting — is covered in depth in the dedicated long-horizon and exploration-sweep chapters, and in Agentic & multi-turn RL for the multi-turn training-loop shape itself. Read this section for the graduation trigger; read those for the harder credit-assignment and boundary-expansion questions once you’re past the initial DAPO-recipe GRPO baseline.

The reward-model question (PRM vs outcome/rubric)

  • PRM (process reward, dense step-level) — score each reasoning step (Lightman et al., arXiv:2305.20050). Niche / avoided in production: DeepSeek explicitly rejected PRM for R1 due to step-level reward hacking (arXiv:2501.12948).
  • Outcome verifier + rubric/critic grading — the real 2026 answer to “what replaced PRM”: not dense step rewards, but LLM-judge/rubric-based outcome grading. Gemini’s “Critic” (prompted rubric grader, arXiv:2507.06261) and OpenAI’s RFT “model grader” are both this in production. Your deterministic flag verifier is the ungameable end of this spectrum — keep it there (gameability ladder in Contested edges).

When to reach for RL

An execution gap where rejection-sampling FT has plateaued (entropy collapsed), and you want the negative-sample gradient + online updates. Cost: online rollout infra, reward plumbing, KL control, instability, and the train↔inference precision mismatch (its own rabbit hole — see the shared memory note research/fp8-quantization-mechanics-training-serving.md).

This is not a one-shot decision. Is the recipe a loop? covers revisiting GRPO/RLVR across rounds (continue-from-checkpoint vs. restart-from-base, loop-exit criteria); ordering rules covers where a DPO/preference stage sits relative to this one and whether SFT applied afterward erodes the RL gains; data mixing & forgetting covers what happens to reasoning traces when RL-stage output gets mixed back into later SFT rounds; and proven-first ranking is the concrete tier table for picking an optimizer inside this family (GRPO/RLVR T1, PPO/DAPO T2 fallbacks, GiGPO/GTPO T3/T4 watch-list).