Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Preference — RLHF · DPO · KTO

Signal = comparisons (A ≻ B). This family exists for the case where you cannot write verify() — “helpful / harmless / on-brand” has no programmatic checker, but a human (or an AI judge) can rank two outputs. Preference is one of three fixed paradigm presets on the canonical on/off-policy axis (the other two are imitation and reinforcement) — see Foundations for the axis itself and Contested edges §6 for why “three independent knobs you toggle / a combinatorial grid” is a retired teaching scaffold: the axes aren’t independent, so what differs across RLHF/DPO/KTO/IPO/ORPO/SimPO is where the pairs come from and how the loss is shaped, not a free combination. Load-bearing property: preference methods reshape ranking over behaviors π_θ can already produce — they inject no new capability (lessons/post-training/dpo-kto-for-agent-tool-selection.md, shared memory). This is grounded in the elicit-not-expand genealogy (RLVR/RL reweights mass already in the base distribution rather than adding new ones — Yue et al., arXiv:2504.13837) and the stronger SFT-memorizes/RL-generalizes anchor (Chu, Zhai et al., arXiv:2501.17161, ICML 2025, 694 citations) — not on the single 0-citation “Scalpel vs. Hammer” preprint (arXiv:2507.10616), whose own authors call their result a “preliminary indication” and which is contested by pushback re-reading the SFT→RL relationship as OOD-forgetting restoration rather than replacement (arXiv:2509.12235); see Contested edges §1.

This chapter is also stage 4/5 of the ordered skeleton (preference opt, after cold-start SFT and rejection-sampling, before/around RLVR) in the recipe is a sequence, one leg of the iterated tail in is the recipe a loop (DPO run N times across rounds, not once), and the T1/T2/T3 tiering in proven-first ranking §3 — read those for where in the pipeline and why start here; this chapter is the what/mechanism.

RLHF (reward model + PPO)

  • What: train a reward model on preference pairs (Bradley-Terry), then optimize π_θ against it with PPO + KL-to-reference (PPO mechanics: Reinforcement). The canonical pipeline is InstructGPT (arXiv:2203.02155).
  • Eats: preference pairs → a learned scalar reward.
  • Still alive in 2026, not dead: Gemini 2.5 runs an explicit Reward-Model + Critic + RL loop (“RLF”, arXiv:2507.06261 §2.4); GPT-5’s sycophancy fix scores conversations and uses that as a training reward (OpenAI GPT-5 system card / model-training page).
  • Gotcha: a learned RM has parameters to exploit → reward hacking. Deterministic verifiers (RLVR) avoid this; see the gameability ladder in Contested edges.

DPO and the direct-preference family

  • What: skip the RM + RL loop — a closed-form loss directly raises logπ_θ(chosen) − logπ_θ(rejected) against a frozen reference, provably equivalent to the RLHF objective under Bradley-Terry (DPO, arXiv:2305.18290). Key hyperparameter: β (KL strength).
  • Eats: (prompt, chosen, rejected) triples.
  • Policy: off-policy by default (pairs usually from another model / earlier checkpoint) — its weakness; iterative/online DPO resamples from current π_θ each round to make it on-policy. Off-policy preference optimization “often suffers from a distributional gap between the policy and the preference data” (arXiv:2406.11827); regenerate the preference pairs from the current checkpoint before each new round rather than reusing stale ones — see is the recipe a loop for the full mechanics and failure mode.
  • Production proof: Llama 3 chose DPO over PPO for its offline preference stage for stability/scalability at their scale, and runs it iteratively (their “iTeC” = rejection-sampling + SFT/DPO/IPO + online RL, several rounds) (arXiv:2407.21783); Llama 2 runs ~6 rounds of SFT+DPO. Qwen2.5 stages an explicit SFT → offline DPO → online GRPO sequence (arXiv:2412.15115).
  • Ordering caveat (contested, not settled): DPO before or after RLVR depends on whether the preference signal and the verifiable-reward signal target the same behavior (fold into one RL stage, à la DeepSeek) or orthogonal behaviors — report quality/style vs. flag-captured — (sequence them, harder-to-specify objective last, à la Tülu 3/Nemotron). Full table: the recipe is a sequence §5. Also: heavy pre-RL DPO is erosive to downstream RL exploration (“SFT and DPO can over-constrain the model, restricting exploration during the online RL stage” — Llama 4 blog); a light, on-policy-anchored DPO pass after RL (Llama 4’s own move) is conditional/safe. See stage ordering & batching and data mixing & forgetting for the dose-dependence.

Variants and their niche

  • KTO (arXiv:2402.01306): learns from unpaired good/bad labels (Kahneman-Tversky value model) — no matched pairs needed. This fits mined agent logs exactly (a pile of failed runs + a pile of clean solves).
  • IPO (arXiv:2310.12036) stabilizes DPO’s tendency to collapse both logprobs at high β; ORPO (arXiv:2403.07691) folds preference into SFT with no reference model; SimPO (arXiv:2405.14734) drops the reference via length-normalized reward.
  • Honest status: DPO is the T1 proven-first pick here (9,399 citations, named in Llama 3’s Herd-of-Models report, Tülu 2/3’s post-bake-off choice, default trainer in TRL/Axolotl/LLaMA-Factory/Unsloth); KTO is T2 — strong conditional fit for unpaired pass/fail logs specifically. IPO/ORPO/SimPO are otherwise niche/T3 — real, used in fine-tuning shops and ablated in Tülu 3 (which explicitly bake-off’d SimPO against DPO-norm and kept DPO-norm), but no Llama/Qwen/DeepSeek/GPT/Claude/Gemini tech report names them as the production choice (survey: arXiv:2503.11701). Plain DPO + iterative DPO are the mainstream ones. Full tier table with citation counts and the “why start with DPO here”: proven-first ranking §3.

RLAIF / Constitutional AI

  • What: replace human preference labels with AI feedback against a written constitution (Constitutional AI, arXiv:2212.08073).
  • Status: mainstream at Anthropic (it is the core method) and partially adopted at Google (Gemini 2.5 safety is “loosely inspired by Constitutional AI”, arXiv:2507.06261). 2026 refinement: Anthropic now teaches the constitution via synthetic document fine-tuning (SDF) → SFT → RL, because “demonstrating desired behavior is insufficient — the model must learn why” (alignment.anthropic.com, “teaching Claude why”, 2026).