Imitation — SFT · distillation · rejection sampling

Signal = demonstrations. Objective = cross-entropy on target tokens. Imitation is one of three fixed paradigm presets on the canonical on/off-policy axis (the other two are preference and reinforcement) — see Foundations for the axis itself and Contested edges §6 for why “three independent knobs you toggle” is a retired teaching scaffold: the axes aren’t independent, so what differs across SFT / distillation / rejection-sampling is whose trajectories you imitate, not a free combination. This chapter is also Sequence B’s stage 0 (cold-start) and stage 3 (execution-gap bridge) in the recipe is a sequence, and the two T1 proven-first picks in proven-first ranking — read those for where in the pipeline and why start here, this chapter is the what/mechanism.

Per-method template: what · data it eats · on/off-policy · when · gotcha · cite.

SFT (Supervised Fine-Tuning)

• What: MLE on (prompt → target); for agents, target = a full trajectory (What “data” means). The original instruction-tuning result is InstructGPT (arXiv:2203.02155).
• Eats: curated/human/teacher demonstrations.
• Policy: off-policy (targets aren’t π_θ’s samples).
• When: inject a capability or format the model lacks; establish a cold-start before RL. Stage 0 of the ordered skeleton — never the whole recipe (the recipe is a sequence §3, stage ordering).
• Gotcha: off-policy ⇒ blind to execution gaps (the εT² compounding, Foundations) and it tends to memorize rather than generalize — the load-bearing evidence is Chu, Zhai et al., “SFT Memorizes, RL Generalizes,” arXiv:2501.17161 (ICML 2025, 694 citations): outcome-reward RL transfers to unseen rule/visual variants where SFT overfits the training distribution, plus the elicit-not-expand genealogy in Contested edges §1. A smaller, 0-citation preprint (“Scalpel vs. Hammer,” arXiv:2507.10616) frames the same split at the weight level as “GRPO amplifies, SFT replaces” — its own authors call this only a “preliminary indication” and their follow-up ablation “inconclusive,” so treat it as a rhyming, supporting citation, not the basis for the claim (see Contested edges §6 and the fuller writeup in RL long-horizon/exploration §4). Also worse on small models (less capacity to absorb without forgetting) — see data mixing & forgetting for the LoRA-format-collapse case study. And don’t over-invest here before RL: Llama 4’s recipe deliberately keeps SFT lightweight because heavy SFT/DPO restricts downstream RL exploration (Contested edges §4).

Distillation (a kind of SFT — the teacher supplies the demonstrations)

Knowledge distillation originates with Hinton et al., arXiv:1503.02531. Two variants, and the split is the on/off-policy axis:

• Off-policy distillation = SFT on the teacher’s completions. Mainstream: DeepSeek transfers R1’s reasoning into V3/V3.2 as a named post-training stage (DeepSeek-V3, arXiv:2412.19437; V3.2, arXiv:2512.02556); “distilled from GPT-4/R1” datasets are how most small OSS models get capability.
• On-policy distillation = student samples its own rollouts, teacher grades them densely (reverse-KL per token). Fixes the fixed-dataset train/inference mismatch (GKD, arXiv:2306.13649: +90% relative on GSM8K vs supervised-KD). Thinking Machines’ 2025 write-up reports Qwen3-8B ← Qwen3-32B reaching ~70% AIME’24 in 150–200 steps at ~9–30× less compute than RL-from-scratch (thinkingmachines.ai, 2025-10-27).
  • Honest status: niche / promising, not lab-confirmed. As of a 2026 pass, no frontier lab (OpenAI/Anthropic/Google/DeepSeek/Qwen) has stated on-policy distillation as its production recipe — evidence is GKD + one lab blog. Treat the efficiency numbers as directional, not settled. (This corrects an earlier over-strong “sleeper” framing.)
• Eats: teacher completions (off) / teacher-graded student rollouts (on). Requires a teacher genuinely better at your task.

Rejection-sampling FT (“RL without RL”)

Terminology landmine: “RFT” is overloaded. This chapter’s rejection-sampling FT (STaR/RAFT/ReST family, cheap, positives-only SFT) is a different thing from Reinforcement Fine-Tuning (the OpenAI/Fireworks product term for actual online RL/GRPO against a grader, expensive). Say “rejection-sampling SFT” to avoid accidentally speccing a GRPO run — see Contested edges §2.

• What: sample N completions from π_θ, keep verifier-accepted winners, SFT on them; iterate. The lineage: STaR (arXiv:2203.14465), ReST (arXiv:2308.08998), RAFT (arXiv:2304.06767), RFT (arXiv:2308.01825).
• Eats: your own verifier-passed trajectories — which for a CTF harness with a flag check you already generate. Reward must be ground-truth-verified against real tool/server output, never format-matched, or SFT on the resulting set trains confabulated flags (Contested edges §5).
• Policy: on-policy data, SFT update. It’s the first-order special case of policy gradient (reward∈{0,1}, upweight winners).
• When: an execution gap, and you have a verifier but no stronger teacher. Cheapest on-policy move; reuses your SFT pipeline. “Iterate” here is the micro version of the pipeline-level loop question — see is the recipe a loop? for whether you restart from base or continue from checkpoint each round.
• Gotcha (measurable graduation trigger): positives-only ⇒ policy-entropy collapse — fast early gains then plateau. GRPO’s real edge over it is not group-normalization (ablated → negligible) but discarding all-same-reward groups (implicit filtering). See “A Minimalist Approach to LLM Reasoning: From Rejection Sampling to Reinforce” (arXiv:2504.11343). Watch entropy; when it collapses, graduate to GRPO/RLVR.
• Production proof: explicit named stage in Llama 3 (arXiv:2407.21783) and DeepSeek-R1 (~800K rejection-sampled examples between its two RL stages, arXiv:2501.12948). Both are also the T1 proven-first picks ranked in proven-first ranking §4.