Start here: a proven-first ranking of the methods
Learnings gave you the general theory — every method, unattached to any project. Understanding turns that theory toward your bottleneck, and this is where it starts: not another explainer, but the ranked, proven-first answer to “what do I run first.”
Every other chapter in this book explains a method on its merits. This one ranks them, on purpose,
because a time-constrained team with a north star of a fine-tuned open-weight dense model
(Sequence B — frontier-recipe-is-a-sequence.md §2) is not running a research
program. A time-constrained team will not validate unproven/novel methods before starting — it wants the PROVEN,
widely-adopted, high-impact default for each decision, with novel methods explicitly deferred to later.
If you only read one chapter to decide what to run first, read this one; every other chapter is the
“why,” this is the “what, right now.”
1. The metric, and why it’s the right one for a time-constrained team
Proven-ness = ADOPTION BREADTH × FLAGSHIP USAGE × MEASURED IMPACT, with NOVELTY PENALIZED.
- Adoption breadth — how many frontier/flagship models, open recipes, and papers actually use the method (not just cite it as related work). Citation count is a proxy, not the answer — a method can be well-cited and still never chosen in production (see SimPO below: real citations, explicitly bake-off’d and passed-on by Tülu 3).
- Flagship usage — is the method named in a flagship tech report as the production choice? This is the strongest single signal in the table below, stronger than raw citation count, because it means a lab with the resources to run a bake-off ran one and picked this.
- Measured impact — are gains actually disclosed (a number, an ablation), not just claimed?
- Novelty penalized — a promising 2026 preprint with one paper’s worth of evidence ranks below a battle-tested method with multiple independent flagship confirmations, even if the preprint’s numbers look better on paper. This is deliberate risk management, not conservatism for its own sake: a time-constrained team betting its one shot on an unreplicated result is the failure mode this metric exists to prevent.
Why this is the right metric here, not just a generically reasonable one: the typical diagnosis in this
situation is an
execution gap (decision.md, diagnosis/framework.md) —
the model already has the capability, it just doesn’t fire reliably — and the reward is a deterministic
ground-truth flag verifier, i.e. a textbook verifiable reward. That combination has a well-known answer
in the literature (rejection-sampling SFT → RLVR-via-GRPO); there is no research question left to answer
about whether this shape of pipeline works, only about executing it well. Spending scarce time
validating a T3/T4 method when a T1 default already fits the reward shape is the exact overthinking this
ranking is built to head off.
Stance honored throughout: no academic cybersecurity-LLM project (CTF-Dojo, Cyber-Zero, Pentest-R1,
HackSynth, AutoPenBench, DRLRM-PT, Cybench, NYU CTF Bench, EnIGMA, InterCode-CTF) is load-bearing evidence
anywhere below — same rule as this book’s proven-by-usage dataset registry
(post-training-dataset-registry.md), applied to methods instead of
data. Grounding is frontier-lab technical reports, frontier open post-training recipes (Tülu 3,
DeepSeek-R1, Qwen3, Llama 3/4, Llama-Nemotron), and citation/adoption counts verified live via Exa on
2026-07-02.
2. RL-algorithm tier table — START HERE: RLVR-via-GRPO
This is the layer that consumes the flag verifier’s reward directly. It fits this setup exactly: a binary
pass/fail signal needs no learned value function, which is precisely GRPO’s design point relative to PPO.
Full detail and sourcing: methods/reinforcement.md.
| Method | Tier | Adoption evidence (live-verified) | Impact | Why this tier |
|---|---|---|---|---|
| GRPO arXiv:2402.03300 | T1 — PROVEN DEFAULT | 6,720–7,285 citations (two live pulls, same day) — highest of any RL optimizer here; trains DeepSeek-V3/R1/V3.2; default GRPOTrainer across HF TRL’s entire supported-model list (Llama, Qwen, Gemma, GLM-MoE, GPT-OSS); base algorithm inside verl. A 2026 practitioner survey: “become the standard RL algorithm for LLM posttraining.” | Proved reasoning-RL at frontier scale in DeepSeek-R1 (Nature-published). | Highest citation count by a wide margin, proven at flagship scale, and literally the default trainer name in the field’s most-used open library. Removes the critic PPO needs — matches a 0/1 flag-verifier reward exactly. |
| RLVR (paradigm, run via GRPO) arXiv:2501.12948 | T1 — PROVEN DEFAULT (paired w/ GRPO) | Nature-published (peer-reviewed, not just preprint); every 2025–2026 reasoning model (o1/o3, R1, Gemini-thinking, Qwen3, Kimi) scales RL against verifiable rewards as the capability driver. | R1 vs. R1-Zero ablation is the disclosed, controlled comparison for cold-start-before-RL. | Not a separate optimizer — the reward-design paradigm (rule-based/verifiable reward, no learned RM) that GRPO runs under. A deterministic flag verifier is a textbook verifiable reward; this is not an analogy for your case, it’s the literal mechanism. |
| PPO arXiv:1707.06347 | T2 — proven fallback | 28,823 citations — highest raw count in the whole comparison; InstructGPT/ChatGPT’s original RLHF, Llama-2-Chat. | Longest track record of any method in this book. | Proven at the largest historical scale, but needs a learned critic — extra memory/instability, and redundant complexity for a single scalar 0/1 reward. Keep as the escalation path if GRPO’s group-relative baseline degenerates (near-0%/near-100% pass-rate collapse). |
| DAPO arXiv:2503.14476 | T2 — proven conditional fix | 1,679–2,113 citations in <18 months; canonical recipe inside verl; MiniMax-M1’s own tech report benchmarks against it directly. | Reproduced 44→50 AIME24 pts. | Its four fixes (clip-higher, dynamic sampling, token-level loss, overlong filtering) exist specifically for GRPO’s zero-gradient collapse on all-correct/all-incorrect groups — exactly the failure shape a binary pass/fail flag verifier risks. Adopt on top of GRPO if that’s observed, don’t start here. |
| GSPO arXiv:2507.18071 | T2 — flagship-confirmed, narrower fit | Qwen Team’s own paper: “these merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models” — direct flagship usage. | Stabilizes MoE RL training. | Real flagship confirmation, but its headline fix (sequence-level clipping for MoE instability) doesn’t target Sequence B’s dense architecture. Keep as a stability lever, not the starting algorithm. |
| GiGPO arXiv:2505.10978 | T3 — promising, watch | NeurIPS 2025 accepted; 0 external citations at check time; real infra adoption (verl-agent, Alibaba ROLL) but small-model (Qwen2.5-1.5B/3B/7B), non-cybersec validation only. | Best conceptual fit for turn-level credit assignment in a multi-turn CTF agent. | Right shape of idea for your multi-turn tool-use agent, but zero independent citations and no flagship adoption. Episode-level GRPO (whole CTF run = one group member) is the proven multi-turn pattern until this clears the bar. |
| GTPO (ACL 2026) | T4 — skip for v1 | Single-paper evidence, no infra adoption, no external citations possible yet (too new). | +3.0–3.9% over GRPO in its own paper only. | The purest novelty-penalty case in this table — one paper’s own numbers, nothing external. Revisit in 6–12 months. |
| PRM (process reward model) arXiv:2305.20050 | T4 — skip (flagship rejected) | DeepSeek explicitly rejected PRM for R1 due to step-level reward hacking (2501.12948). | — | Not merely unproven — a named flagship anti-pattern. |
| Learned/neural reward model (replacing the deterministic verifier) | T4 — skip (architectural anti-pattern) | N/A when a deterministic verifier already exists. | — | Introduces a gameable component where a perfect deterministic verifier already exists — the entire point of RLVR is to avoid this. |
START HERE: RLVR-via-GRPO. It is simultaneously the most-cited method in this table, the literal
mechanism DeepSeek-R1 used to prove RLVR at frontier scale, the default trainer in the field’s most-used
open post-training library, and an architectural match for a binary verifiable reward. Keep DAPO’s fixes in
your back pocket as the T2 escalation path for exactly the group-collapse failure mode a hard pass/fail
verifier is prone to. See methods/reinforcement.md for the mechanics.
3. Preference-method tier table — START HERE: DPO (KTO if unpaired)
This layer sits between the SFT stages and GRPO/RLVR — it’s for signals that are pairwise/binary but not
independently verifiable (report quality, tool-use elegance), not a substitute for the verifier-driven RL
stage. Full detail: methods/preference.md.
| Method | Tier | Adoption evidence (live-verified) | Impact | Why this tier |
|---|---|---|---|---|
| DPO arXiv:2305.18290 | T1 — PROVEN DEFAULT | 9,399 citations, 2,011 influential — second-highest cited method in this whole comparison after PPO/SFT/LoRA. Named in Llama 3’s Herd of Models report (“each round of post-training involves SFT followed by DPO”), Zephyr-7B’s reference recipe, Tülu 2/3’s final pick after an explicit bake-off vs. PPO/SimPO, Qwen-Chat model cards. Default trainer in TRL, Axolotl, LLaMA-Factory, Unsloth. | Llama 3 runs it iteratively across ~6 rounds at 405B scale. | Largest adoption breadth of any preference method by far; no reward model, no RL loop — the simplest infra match for a time-constrained team, and directly compatible with building (accept, reject) pairs from the flag verifier’s own outcomes. |
| KTO arXiv:2402.01306 | T2 — proven, strong conditional fit here | 1,081 citations, 189 influential; official KTOTrainer in TRL; ablated (not chosen primary) in Tülu 3. | “Matches or exceeds DPO… despite only learning from a binary desirable/undesirable signal.” | Not flagship-primary at DPO’s scale, but its unpaired binary-label requirement is an unusually close match to the log shape you’ll typically have: a pile of verified flag-captures and a pile of failed runs, with no natural same-prompt pairing. Use when constructing DPO pairs is the harder engineering lift. |
| RLHF-PPO (reward model + PPO) — InstructGPT arXiv:2203.02155 | T2 — proven, wrong tool here | The original recipe; foundation of ChatGPT/Claude/Gemini lineage; Llama-2-Chat = SFT + rejection-sampling + PPO. | Proven at the largest historical scale of any method in this book. | Needs a trained reward model + full RL rollout infra — exactly the complexity DPO removes, and you’re already building GRPO/RLVR infra downstream. A second heavier RL loop for the preference stage is redundant. Tülu 3’s own bake-off found DPO-variants beat PPO on this specific stage. |
| ORPO arXiv:2403.07691 · SimPO arXiv:2405.14734 · IPO arXiv:2310.12036 | T3 — promising, watch | ORPO 619 cites (639 derivative HF models); SimPO 1,055 cites, NeurIPS 2024, shipped in Ai2’s open-instruct; IPO 1,034 cites, DeepMind, AISTATS 2024. Real OSS traction on all three. | SimPO beat DPO on AlpacaEval2/Arena-Hard in community Llama-3-8B checkpoints. | Real, non-novel, but none of Llama/Qwen/DeepSeek/GPT/Claude/Gemini names any of the three as its primary production choice — and Tülu 3 explicitly bake-off’d SimPO against DPO-norm and picked DPO-norm (“Length-normalized DPO achieved better performance … including PPO, DPO, and SimPO”). A concrete flagship-adjacent rejection, not absence of evidence. Revisit only if DPO shows a specific failure mode these target. |
| RLAIF / Constitutional AI arXiv:2212.08073 | T2 — proven, out of current scope | 3,245 citations; core method at Anthropic; loosely echoed in Gemini 2.5 safety work. | Anthropic’s production alignment method. | Proven and mainstream at the labs that use it, but targets harmlessness/persona alignment, not execution reliability — not applicable to a current execution-gap loop. Relevant only if/when harmlessness-shaping enters scope. |
START HERE: DPO on (accept, reject) pairs built from the flag verifier’s own outcomes. Switch to
KTO specifically when the mined agent logs don’t naturally pair (pass/fail without a matched same-prompt
counterpart). See methods/preference.md for the loss and the DPO/KTO/ORPO
family map.
4. SFT / distillation tier table — START HERE: off-policy SFT + rejection-sampling SFT
Full detail: methods/imitation.md.
| Method | Tier | Adoption evidence (live-verified) | Impact | Why this tier |
|---|---|---|---|---|
| Off-policy SFT (curated / teacher / synthetic instruction data) | T1 — PROVEN DEFAULT | Universal stage-0 of every disclosed recipe checked: InstructGPT arXiv:2203.02155, Llama 2 arXiv:2307.09288 (27,540 curated examples), Tülu 3 arXiv:2411.15124 (939,344-prompt mix), Qwen3 arXiv:2505.09388 (“cold-start” SFT), DeepSeek-R1 arXiv:2501.12948, Llama-Nemotron arXiv:2505.00949. Synthetic variant WizardLM/Evol-Instruct arXiv:2304.12244: 1,700 citations. | Zero disclosed frontier recipes skip this stage. | The oldest, most battle-tested move in post-training. Establishes format/tool-syntax stability before anything else runs. Zero novelty risk. |
| Rejection-sampling SFT (STaR/RAFT/ReST family, “RL without RL”) | T1 — PROVEN DEFAULT, execution-gap bridge | Meta’s Llama-2-Chat runs this as its named primary RLHF-V1–V3 alignment step, before PPO is even introduced (2307.09288: “we used only Rejection Sampling fine-tuning, and after that, we combined … PPO on top”); smaller 7B/13B/34B Llama-2-Chat models are fine-tuned entirely on rejection-sampled data distilled from the 70B. DeepSeek-R1 stage 3/4 = rejection sampling on the RL checkpoint → the 800K-sample SFT set for the second training pass (2501.12948). Academic formalizations: STaR arXiv:2203.14465 (NeurIPS 2022), RAFT arXiv:2304.06767, ReST arXiv:2308.08998 (DeepMind). | Two independent frontier labs (Meta, DeepSeek) run it as production infrastructure. | This is the execution-gap bridge — it plugs directly into the deterministic flag verifier with zero adaptation: sample K, keep the flag-verified successes, fine-tune. Cheapest on-policy move available; reuses your existing SFT pipeline. |
| Off-policy (teacher) distillation | T2 — proven, conditional here | Origin: Hinton et al. arXiv:1503.02531. Flagship production: Gemma 2 arXiv:2408.00118 (cross-references Gemini 1.5’s use), DeepSeek-R1-Distill (6 dense checkpoints, most-downloaded reasoning models on HF, 32B beats o1-mini on several benchmarks). | Large, disclosed impact across 3 independent flagship efforts. | Proven and high-adoption as a method, but conditional here: it needs a stronger available teacher already competent at your specific offensive-security tool-use domain, which doesn’t reliably exist off-the-shelf. Supporting role (cold-start data seeding), not the core loop. |
| On-policy distillation (GKD-style) | T3 — promising, watch | GKD arXiv:2306.13649 (DeepMind, 2023), GKDTrainer shipped in TRL. Thinking Machines Lab blog (2025-10-27, ~76.7% AIME’24, rank-128 LoRA) — a research-lab writeup, not a flagship tech report. Qwen3 blends it in as a secondary technique alongside off-policy KD. | +70–111% relative gains reported at T5-scale in GKD’s own paper. | Real trajectory toward T1/T2 — Qwen3’s partial adoption is new positive signal — but no flagship names it a standalone production stage yet. Re-check in 3–6 months. |
| Academic-cybersec-teacher distillation | T4 — skip | Excluded by stance. | — | No flagship precedent, unverified teacher quality — the stance rule’s direct application. |
START HERE: off-policy SFT (cold-start) → rejection-sampling SFT (on-policy). Both clear T1
independently; the second is the specific mechanism that turns the flag verifier into training data at
essentially zero extra engineering cost. See methods/imitation.md.
5. The one proven end-to-end starting sequence
SFT (off-policy cold-start) → rejection-sampling SFT (on-policy) → DPO (preference) → GRPO/RLVR (reinforcement) → iterate.
This is not a novel combination — it is the modal recipe across every disclosed open frontier pipeline
checked (Tülu 3 arXiv:2411.15124, DeepSeek-R1
arXiv:2501.12948, Qwen3
arXiv:2505.09388, Llama-Nemotron
arXiv:2505.00949), and independently reconfirmed by a fresh 2026
practitioner survey as “the dominant posttraining recipe” a year later — see
frontier-recipe-is-a-sequence.md §2 (Sequence B) for the full
stage-by-stage derivation.
Why this is the lowest-risk, highest-coverage starting point for your case specifically:
- Every stage independently clears T1 in the tables above — zero stages require betting on an unproven method.
- It matches the diagnosis. An execution gap is exactly what rejection-sampling SFT is for, and if you already have the free ingredient it needs — a real deterministic flag verifier generating on-policy correct/incorrect labels — this stage is essentially free.
- It matches the reward shape exactly. RLVR via GRPO is the literal mechanism for a deterministic verifier, not an analogy.
- Cold-start-before-RL is a disclosed stability requirement, not a stylistic choice — DeepSeek-R1 vs. R1-Zero (same base, same GRPO, only SFT differs) shows pure-RL-from-scratch gets real gains but “poor readability, language mixing”; skipping straight to GRPO is the riskier path, not the leaner one.
- DPO-before-RLVR is the safer default ordering here because the two signals are genuinely orthogonal
in this setup — flag-capture (verifiable) vs. report quality/tool-use elegance (not verifiable) —
which is precisely the condition
frontier-recipe-is-a-sequence.md§7 flags as the case for sequencing rather than folding into one RL stage (DeepSeek’s own pattern).
What this sequence deliberately is not: a from-scratch pretraining run (Sequence A, irrelevant if you’re starting from an existing open-weight dense checkpoint); a jump straight to GRPO/RLVR with no SFT cold-start (higher instability risk, R1-Zero’s own documented failure mode); a fold-everything-into-one-RL design (the wrong default here because the two signals are orthogonal, not the same behavior).
This sequence is Rung-1-scoped — it resolves execute reliably on the portfolio you have. It is a
prerequisite for, not a substitute for, the frontier-capability question addressed in
roadmap-inputs.md and frontier-cyber-model-path.md.
6. One-glance consolidated tier list
| # | Method | Family | Tier | Where it lives |
|---|---|---|---|---|
| 1 | Off-policy SFT | Imitation | T1 | methods/imitation.md |
| 2 | Rejection-sampling SFT (STaR/RAFT/ReST) | Imitation | T1 | methods/imitation.md |
| 3 | DPO | Preference | T1 | methods/preference.md |
| 4 | GRPO | Reinforcement | T1 | methods/reinforcement.md |
| 5 | RLVR (paradigm, via GRPO) | Reinforcement | T1 | methods/reinforcement.md |
| 6 | KTO | Preference | T2 (strong conditional fit) | methods/preference.md |
| 7 | Off-policy (teacher) distillation | Imitation | T2 (conditional on teacher availability) | methods/imitation.md |
| 8 | RLHF-PPO (reward model + PPO) | Preference / Reinforcement | T2 (proven, redundant infra here) | methods/preference.md, methods/reinforcement.md |
| 9 | PPO (as RL optimizer) | Reinforcement | T2 (fallback if GRPO baseline degenerates) | methods/reinforcement.md |
| 10 | DAPO | Reinforcement | T2 (fix for GRPO group-collapse) | methods/reinforcement.md |
| 11 | GSPO | Reinforcement | T2 (flagship-confirmed, MoE-specific) | methods/reinforcement.md |
| 12 | RLAIF / Constitutional AI | Preference | T2 (proven, out of current scope) | methods/preference.md |
| 13 | On-policy distillation (GKD-style) | Imitation | T3 — watch | methods/imitation.md |
| 14 | ORPO / SimPO / IPO | Preference | T3 — watch | methods/preference.md |
| 15 | GiGPO | Reinforcement | T3 — watch | methods/reinforcement.md, methods/rl-long-horizon-exploration.md |
| 16 | GTPO | Reinforcement | T4 — skip for v1 | methods/rl-long-horizon-exploration.md |
| 17 | PRM (process reward model) | Reinforcement | T4 — skip (flagship rejected) | methods/reinforcement.md |
| 18 | Learned reward model (replacing verifier) | Reinforcement | T4 — skip (architectural anti-pattern) | methods/reinforcement.md |
| 19 | Academic-cybersec-teacher distillation | Imitation | T4 — skip (stance-excluded) | methods/imitation.md |
The starting sequence, once more: SFT → rejection-sampling SFT → DPO → GRPO/RLVR → iterate. Read the
diagnosis first (decision.md), the sequencing evidence next
(frontier-recipe-is-a-sequence.md), and check
contested.md before treating any T2 escalation as settled — the DPO-vs-fold-into-RLVR
ordering and the RL-boundary-expansion question are both flagged there as genuinely unsettled, not smoothed
over. roadmap-inputs.md is where this ranking meets your actual
challenge set and its segmentation gates.
Confidence
High on the T1 picks in all three families (off-policy SFT, rejection-sampling SFT, DPO, GRPO/RLVR) —
each is corroborated by 2+ independent flagship technical-report disclosures verified live, plus a fresh
2026 practitioner survey independently confirming the same stage-skeleton is still the mainstream picture,
not stale training-data knowledge. Moderate-high on the T2 conditional calls (KTO’s project-fit read is
this book’s own synthesis, not a claim any external source makes about your case specifically —
flagged as such). Explicitly contested, not settled: DPO-vs-fold-into-RLVR staging order, and whether
RL expands or merely elicits the reasoning boundary — see contested.md; neither affects
the T1 picks above, only how individual stages get tuned once the sequence is running. Lower confidence
on the exact production RL/preference algorithm inside fully closed flagships (OpenAI, Anthropic, Gemini) —
those tech reports withhold the relevant specifics; this ranking cites what’s disclosed and flags what
isn’t, per the same practice used throughout this book.
Sources
All arXiv ids above were crawl-verified live (Exa) on 2026-07-02, cross-checked against Semantic
Scholar citation counts pulled the same session. Full per-family source lists, exact citation-count
snapshots, and the underlying research threads: artifacts/overnight-ranking/research/rl-algo-ranking.md,
artifacts/overnight-ranking/research/preference-method-ranking.md,
artifacts/overnight-ranking/research/sft-distillation-ranking.md, and
artifacts/overnight-ranking/research/proven-default-recipe-and-tiers.md (this chapter’s direct synthesis
source). See also this book’s own references.md for the canonical arxiv-id registry.