The family map
You now have the axis (on-policy vs off-policy, from the previous chapter) and the shape of the data object (a trajectory, not an answer). This chapter zooms out one level and puts every named method in this book on those two coordinates — the map every family chapter that follows hangs off of.
Two canonical axes, not a grid you build methods from: what signal you learn from (imitation / preference / reinforcement — three families) and on/off-policy (Foundations — whose rollouts the training data comes from), plus one orthogonal axis, PEFT, that is a delivery mechanism, not a learning signal. Every named method below is a fixed preset on those two axes, not a free combination of them — signal + policy-source largely determine what a method does, so “pick a family, then pick a policy-source, then invent the method” produces non-methods, not real ones. This page used to teach the two axes as “three independent knobs you toggle,” a scaffold that over-reached; see Contested edges §6 for why that framing was retired. Learn the family, then the preset within it — that’s the whole map.
graph TD ROOT["Post-training<br/>push mass toward good behavior"] ROOT --> IM["IMITATION<br/>signal = demonstrations"] ROOT --> PR["PREFERENCE<br/>signal = comparisons A≻B"] ROOT --> RL["REINFORCEMENT<br/>signal = reward / verifier"] IM --> SFT["SFT"] IM --> DIST["Distillation<br/>off-policy / on-policy"] IM --> RS["Rejection-sampling FT<br/>= 'RL without RL'"] PR --> RLHF["RLHF (RM + PPO)"] PR --> DPO["DPO · KTO · IPO · ORPO · SimPO"] RL --> PPO["PPO"] RL --> GRPO["GRPO → GSPO / DAPO"] RL --> RLVR["RLVR (verifiable reward)"] RL --> AG["Agentic / multi-turn RL"] PEFT["PEFT: LoRA · QLoRA · DoRA<br/>a HOW, applied to any of the above"] ROOT -.delivery.-> PEFT
Picking a family here only answers which signal. Four questions this map deliberately leaves open, each with its own chapter, once you know which preset you’re reaching for: does the recipe run once or loop back over earlier stages? In what order can stage-types safely interleave, and how do you batch N fixes into one round? How do you mix data so a later stage doesn’t erase what an earlier one taught? And — for this project specifically — which preset should you actually start with, ranked by proven adoption rather than novelty? The status table below is what’s real in 2026; those four chapters are sequence, iteration, mixing, and rank.
What’s actually load-bearing in 2026 (verified)
The status column below is from a fresh Exa pass over model tech reports + lab blogs (2025–2026), not recalled — sources cited per row throughout the method chapters.
| Method | 2026 status | Anchor |
|---|---|---|
| SFT | Mainstream, universal — stage 0 of every recipe | — |
| Off-policy distillation | Mainstream — DeepSeek distills R1 → V3/V3.2 as a named stage | arXiv:2412.19437, arXiv:2512.02556 |
| On-policy distillation | Niche / promising — NOT yet confirmed in any frontier lab’s production recipe | GKD arXiv:2306.13649; Thinking Machines blog 2025-10-27 |
| Rejection-sampling FT | Mainstream — named stage in Llama 3 & DeepSeek-R1 | arXiv:2407.21783, arXiv:2501.12948 |
| RLHF (RM+PPO) | Mainstream at proprietary labs (Gemini 2.5, GPT-5) | arXiv:2507.06261 |
| DPO | Mainstream — Llama 3’s offline preference stage, Tülu 3’s post-bake-off pick | arXiv:2407.21783 |
| KTO | Niche overall, but a genuine T2 conditional fit for unpaired pass/fail logs — official TRL trainer, ablated (not chosen primary) in Tülu 3 | arXiv:2402.01306 |
| IPO/ORPO/SimPO | Niche — real OSS traction, but Tülu 3 explicitly bake-off’d SimPO against DPO-norm and kept DPO-norm; no flagship names any of the three as primary | arXiv:2503.11701 |
| GRPO | Mainstream — the reasoning-RL default | arXiv:2402.03300 |
| RLVR | Mainstream — arguably the defining 2025-26 technique | arXiv:2501.12948, arXiv:2507.06261 |
| GSPO (Qwen3) | Mainstream — first GRPO-successor with a flagship behind it | arXiv:2507.18071 |
| DAPO | OSS-tooling mainstream; ByteDance-origin, not confirmed elsewhere | arXiv:2503.14476 |
| PRM (process reward) | Niche — explicitly rejected for R1 (step-level reward hacking) | arXiv:2501.12948 |
| Rubric/critic outcome reward | Mainstream & growing — the real replacement for PRM | arXiv:2507.06261 |
| Agentic / multi-turn RL | Mainstream & the frontier edge — see its own chapter | Deep Research; Kimi K2/K2.5 |
| LoRA/QLoRA/DoRA | Mainstream in the applied layer; labs post-train flagships full-parameter | arXiv:2106.09685 |
| Self-play | Experimental — no confirmed frontier-lab production use as of this pass | — |
Read the family chapters for the mechanism + “what data it eats” + when to reach for each.
Cross-links: this page answers which family; The recipe is a sequence, not a pick answers in what order the families compose into an actual pipeline; Is the recipe a loop? and Ordering rules: interleaving stages & batching cover whether and how you revisit a family once you’ve left it; Data mixing, ratios & not forgetting how to think covers what happens to an earlier stage’s behavior when a later one trains on foreign data; The one axis that predicts everything is the on/off-policy half of this page’s two-axis claim, spelled out in full; and Start here: a proven-first ranking turns this family-level status table into one ranked starting sequence for this project specifically.