The one axis that predicts everything

You’re leaving the Problem section behind — the diagnostic chapters told you the agent solves a minority of your challenge set and left the “why.” Learnings starts the general theory from scratch, unattached to your specific case, beginning with the one axis every method in this book is built on.

If you internalize one thing, make it this: on-policy vs off-policy. It predicts which methods can fix which failures, and it’s the reason a colleague’s SFT run can move the eval by two points and stall.

Definitions (precise)

• Off-policy: training targets are sampled from a distribution other than the model’s current policy π_θ — a human, a teacher model, a frozen dataset. The model raises the likelihood of sequences it did not generate.
• On-policy: the training data is sampled from π_θ itself (the current weights), then scored/labeled. The model learns from its own rollouts.

This is standard RL vocabulary, not a framing I invented — see any policy-gradient treatment; the LLM-specific consequences are laid out formally in the imitation-learning reduction below.

Why off-policy imitation is structurally blind to execution gaps

The mechanism, as a flow:

graph LR
  A["Train on the teacher's states"] --> B["Deploy: π_θ drives,<br/>visits π_θ's OWN states"]
  B --> C["One slip → a state the<br/>teacher never visited"]
  C --> D["No training signal there<br/>→ error compounds"]
  D --> B

This isn’t hand-waving — it’s a theorem. Ross, Gordon & Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning” (AISTATS 2011, arXiv:1011.0686), Thm 2.1: a behavior-cloned policy with per-step error ε incurs total cost bounded by J(π̂) ≤ J(π*) + ε·T² — quadratic in horizon T, because a single deviation moves you to states off the expert’s distribution where you have no supervision, and errors accrue at up to unit cost for the rest of the episode. Their DAgger correction — aggregate data from the learner’s own induced state distribution — restores near-linear O(ε·T) regret.

Translate to your setting: an execution failure happens, by definition, in a state π_θ reaches on its own. Off-policy data is — structurally — data about states π_θ doesn’t reach. So off-policy SFT optimizes correctness in the wrong region. On a long agentic CTF trajectory (large T), the T² vs T gap is the whole story.

The fix, stated once

Make the data on-policy: sample from π_θ, then score those samples. Every method that “fixes execution” — rejection-sampling FT, GRPO/RLVR, on-policy distillation — is a different way of doing exactly that. The on-policy distillation line makes the connection explicit: it exists specifically to kill the train/inference mismatch of fixed-dataset KD by sampling from the student during training (GKD, Agarwal et al., arXiv:2306.13649).

The corollary you’ll use constantly

• Failure lives in π_θ’s own distribution (it solves sometimes, fails often) → on-policy method.
• The correct behavior is absent from π_θ entirely (never appears at high N) → nothing on-policy to reinforce → you must inject off-policy (SFT / teacher data / a tool).
• The behavior exists but is mis-ranked → preference method.

That routing is the spine of The decision.

What this axis is not

This is one axis, not a set of independently-toggleable knobs. Crossing on/off-policy against the imitation/preference/reward paradigm split does not yield a free combinatorial grid of methods — signal and policy-source largely determine each other, so each named method (SFT, rejection-sampling FT, DPO, GRPO/RLVR, on-policy distillation) is a fixed preset on this axis, not a pick-two-knobs product. An earlier pass at this material packaged it as an interactive matrix implying free combination, which produced nonsense for some products — that framing is retired; see Contested edges §6 for the correction and Methods overview for the preset list.

A caution on evidence quality while you’re building intuition here: a low-citation 2025 preprint (arXiv:2507.10616, “Scalpel vs. Hammer”) frames the amplify/replace split in exactly this vocabulary (“GRPO amplifies, SFT replaces”) but is a single, contested, 0-citation source — its own authors call it “preliminary.” Don’t anchor on it. The load-bearing evidence for “on-policy reinforces what already fires, off-policy is required to inject what doesn’t” is this chapter’s own DAgger argument above, plus “SFT Memorizes, RL Generalizes” (Chu, Zhai et al., arXiv:2501.17161, ICML 2025, 694 citations) and the elicit-not-expand result (Yue et al., arXiv:2504.13837, NeurIPS 2025 Best Paper Runner-Up) — both verified live, both far better cited than 2507.10616. See Contested edges §1 for the full, honest state of that debate (it’s genuinely contested at the margins — prolonged, KL-controlled RL can expand the boundary; standard-recipe RL elicits).

Where this axis goes next

This single axis is the foundation the rest of the book’s structure is built on:

• The fixed presets above don’t stand alone — frontier recipes run them as an ordered stage sequence, not a menu (The recipe is a sequence, not a pick).
• That sequence isn’t strictly one-shot either — it loops, and whether a re-applied stage continues from checkpoint or restarts from base changes what “on-policy” even means the second time around (Is the recipe a loop?).
• The ordering constraints on which stage-type can safely follow which are a direct consequence of the εT² argument above (Ordering rules: interleaving stages & fixing N problems).
• Off-policy injection has its own failure mode worth knowing before you reach for it — naive off-policy SFT can cause format/behavioral collapse independent of weight magnitude (Data mixing, ratios & not forgetting how to think).
• For “what do I actually run first,” the axis becomes a ranked, proven-first decision, not just a diagnostic (Start here: a proven-first ranking of the methods).