Introduction

This is a living field manual for post-training an agent — written for an engineer who wants to know it, then do it, not experiment blindly. It grows: each session adds or sharpens a chapter.

📄 Offline? Grab the whole book as a single PDF: download.pdf — every chapter, diagrams included, for reading with no network.

The problem it exists to answer

Run a CTF-playing agent against a real benchmark and the pattern is a familiar one: most challenges fail, only a minority solve. You have the harness, the workflow, the budget, and the backing to do anything. The bottleneck is usually not data — it’s knowing which fine-tuning method you want, and therefore what data to build, and why.

Everything here builds toward answering that from first principles.

The shape of this book

Four parts, front to back — Problem → Learnings → Understanding → Conclusion:

1. Problem — this chapter, the 5-minute journey, and the diagnostic chapters (behavioral audit → training signal, diagnosing the gap) establish what’s actually broken and how to know it from evidence, not vibes.
2. Learnings — how models are actually built — the general theory, unattached to your case: the on/off-policy axis, what “data” means, the method family (imitation / preference / reinforcement / agentic RL / PEFT), and how frontier labs sequence, loop, order and batch, and mix data without forgetting these into real recipes.
3. Understanding — applying it to your case — the same theory turned toward your bottleneck: ranking methods by proven-ness, deciding whether you have one problem or many, mapping method → data, and what instrumentation has to exist before a training run.
4. Conclusion — the actual decision, the roadmap and forks still open, and the edges that are genuinely contested (so you don’t mistake open questions for settled ones).

How to read it

• Want the feel first? → The 5-minute journey — the interactive version, embedded.
• Want the theory? → start at Learnings and go in order.
• Want the answer for your case? → jump to Method → Data and The decision.
• Want to know if you actually have an execution gap, and which RL technique fixes it? → Diagnosing the gap, then the RL sweep at RL that creates value.

What’s canonical vs. what’s a teaching scaffold (read this once)

Being honest about provenance, because you’re becoming a researcher and the distinction matters:

• Canonical, universal, you’ll find it in any RL/post-training text: the on-policy vs. off-policy axis, and the three learning paradigms (imitation / preference / reinforcement). These are load-bearing and not up for debate.
• My teaching scaffold: any packaging that presents these as “N knobs you freely toggle.” The axes describe methods; they are not independent dials you combine — each named method is a fixed preset. An earlier interactive matrix implied free combination and produced nonsense for some products. That was the scaffold over-reaching. Corrected here: learn the one axis (on/off-policy) + the fixed method presets, not a combinatorial grid.

The one line to anchor on

Every method is the same move — push probability mass toward good behavior — differing only on whose distribution the data comes from (off- vs on-policy) and whether you also learn from failures.

Keep that; the rest is detail.

The 5-minute journey

Before the theory, feel the shape of it. This is the interactive walk-through — problem → reasoning → decision — embedded live. Scroll it; click the ladder rungs and answer the decision tool at the end.

If the frame is cramped, open it full-screen: assets/journey.html

Everything in that frame is expanded, corrected, and cited in the chapters that follow. The frame is the map; the chapters are the territory.

From behavioral audit to training signal

A trace-level behavioral audit of agentic CTF-solving runs — not just a pass/fail benchmark score — is a behavioral trace. Each of the 5 findings below describes a shape of failure, and each shape implies a different kind of gap: exploration, methodology/reasoning, credit assignment, or eval validity. This page maps observed behavior → gap type → the literature’s actual fix → how you’d check the fix worked, for each of the 5 patterns, against this project’s real setup (rejection-sampling SFT on verifier-passed solves now, GRPO/RLVR when entropy collapses; ground-truth flag verifier; knowledge in tools not weights).

flowchart LR
    A[observed behavior] --> B[gap type]
    B --> C[training signal / method]
    C --> D[verification check]
    D -.re-audit.-> A

None of this relitigates the diagnosis (execution gap, not knowledge gap) — it asks a narrower question per pattern: given this specific failure shape, which lever in the SFT→GRPO graduation actually targets it, and how would you know if it worked?

Pattern 1 — Agents prefer their own tools over the provided tool surface

Observed: raw curl/shell dominates; the bespoke sectools surface is mostly unused.

Gap type [E]: this is not a knowledge gap (the agent isn’t ignorant of the tools — they’re in its context) and not really a reasoning gap. It’s an exploration-of-tool-space problem: tool choice is driven by the model’s pretraining prior (shell/curl is high-likelihood, familiar, “free” under next-token probability) rather than by the tool’s actual value for the subtask. Faghih et al., “Tool Preferences in Agentic LLMs are Unreliable,” arXiv:2505.18135 (2025-05-23) shows this is gameable by description text alone — edited docstrings shift usage >10x with zero change in tool capability. Confidence: established (controlled cross-model study). This is a diagnosis, not a fix — it just rules out “better docstrings” as a ceiling-breaking move.

Training signal:

Designed to fix: pattern 1 — agents defaulting to raw shell/curl over the provided tool surface.

• ToolRL (Qian et al., arXiv:2504.13958, 2025-04-16, established) — don’t SFT-imitate tool traces; decompose the reward into per-call terms so which tool was chosen is its own learnable signal, not folded into one coarse outcome score:

  r_format  = valid_json_schema(call)          # 0/1
  r_tool    = tool_is_appropriate(call, state)  # 0/1, graded by task type
  r_param   = params_correct(call)              # 0/1 or partial
  r_outcome = env_feedback(call)                # sparse, terminal-heavy — your flag verifier
  reward = w1*r_format + w2*r_tool + w3*r_param + w4*r_outcome
  

  Their ablation: reward granularity (per-call beats per-episode) and reward type (graded beats binary) both matter. This is the cheapest lever — you already have the tool registry, you need the auxiliary signal.
• ReTool (Feng et al., arXiv:2504.11536, 2025-04-15, established) — trains the whole trajectory end-to-end so the policy learns when in a reasoning chain to reach for a tool, not just which one given an isolated decision point. Confirms the shape of the project’s SFT→GRPO plan; its specific addition is that real sandbox tool-execution output (not a paraphrase) must be in the rollout context that gets scored — worth auditing whether secagent/daytona feeds live stdout into the trajectory used for reward.
• Tool-Star (Dong et al., arXiv:2505.16410, 2025-05-22, promising, <6mo/low-citation) — naive RL under-explores a large tool inventory because gradient concentrates on whatever already gets used. Its fix: manufacture forced/hinted rollouts that exercise under-used tools before RL, verify with the real environment, fold verifier-passed ones into SFT data. Directly targets tools that never get invoked — and flags a self-reinforcing trap: a rejection-sampling corpus built from the current curl-biased policy will never contain a dead tool succeeding, because the policy never tried it. RL alone has ~zero probability mass to reinforce on those tools.
• Search-R1 (Jin et al., arXiv:2503.09516, 2025-03-12, established) contributes one mandatory engineering detail, domain-general: mask tool/environment output tokens from the policy-gradient loss — you don’t want to reinforce or penalize text the environment produced (shell stdout, HTTP bodies), only the model’s own query/command-generation tokens.

Honest caveat: ToolRL/ReTool/Tool-Star are validated on general agentic tool-use, not offensive-security tool-use specifically — the transplant to sectools is this researcher’s inference, not literature-verified. Random-Crypto/HackSynth, arXiv:2506.02048 — academic domain-specific CTF RL work, cited for context only, not a basis for this page’s claims — reports vanilla GRPO on crypto-CTF attributing generalization gains to improved tool usage; per this project’s standing rule, academic cybersecurity-LLM training/benchmark papers don’t count as frontier evidence, so this is mentioned but not relied on. The actual basis for “RL over tool-choice transfers” stays the general-agentic evidence above (ToolRL/ReTool/Tool-Star) — domain-general RL-for-tool-use results that don’t need a CTF-specific data point to hold.

Verify the fix: track the tool-usage histogram across all 40 sectools entries before/after training — do previously-dead tools get invoked at all on held-out challenges (not just replayed on training challenges)? Cheap, no-training-required companion check: DIVER, arXiv:2509.26209 (2025-09-30) rewards pairwise diversity across a rollout group — if adapted to “which tools/commands were used” rather than token text, a rising group-diversity score on tool choice is a leading indicator before solve-rate moves at all.

Pattern 2 — React-and-guess, no methodology (no wordlists/checklists/PTES sequencing; premature pivoting after failure)

Observed: no visible recon→enum→exploit sequencing; the agent abandons a line of attack after one failed attempt instead of enumerating alternatives methodically.

Gap type [R]: a sequencing prior is missing — the model has PTES-shaped knowledge somewhere in its weights (it can describe methodology if asked) but doesn’t apply it as an ordering constraint during live rollouts. This reads as reasoning/methodology, not exploration-in-the-entropy sense, though the two compound (§ below, and see RAGEN / StarPO’s “Echo Trap”).

Training signal:

Designed to fix: pattern 2 — no methodology / premature pivoting after a single failed attempt.

• Two-stage guide-then-explore RL, re-grounded on general RL theory: Jump-Start Reinforcement Learning (JSRL) — Uchendu et al., arXiv:2204.02372 (2022-04-05, established, 17+ citations) is the domain-general theoretical basis for the same two-stage shape this page used to cite a CTF-specific paper for: a guide-policy (built from offline data/demonstrations/an existing policy) forms a curriculum of starting states, and an exploration-policy is trained forward from those states — naive initialization-then-finetune underperforms this because value-based methods handle a cold-start policy poorly. Mapped onto this project’s actual pipeline:

  # Stage 1 — guide-policy: rejection-sampling SFT on verifier-passed walkthroughs
  # (methodology prior — ordered recon → enum → exploit, not just any verifier-passed trace)
  # Stage 2 — exploration-policy: online GRPO/RLVR in the live sandbox
  # reward = terminal flag-verified {0,1} + intermediate env feedback (command succeeded/failed)
  # JSRL's curriculum knob: anneal how far into the guide-trajectory the exploration-policy starts,
  # rather than always starting cold — cheap to add to the existing rejection-sampling corpus.
  

  DeepSeek-R1’s own disclosed recipe (R1, arXiv:2501.12948) is the frontier-lab confirmation that this shape works at scale: “cold-start SFT data, then RL” outperforms RL-from-scratch specifically because RL-from-scratch on an unguided policy wastes exploration budget relearning basic structure the SFT stage would have given it for free — the same failure mode JSRL formalizes. Academic CTF/pentest work — cited for context only, not a basis: Pentest-R1 (Kong et al., arXiv:2508.07382, evaluated on Cybench + AutoPenBench) reports the identical two-stage shape in the CTF domain specifically and claims both stages are required, order matters, and stage-1 data must be walkthrough-shaped rather than raw verifier-passed traces. Per this project’s standing rule, academic cybersecurity-LLM training/benchmark papers (Pentest-R1, AutoPenBench, and similar) don’t count as frontier evidence for a decision — the load-bearing basis here is JSRL + DeepSeek-R1’s cold-start-then-RL recipe, both domain-general. If Pentest-R1’s specific findings (walkthrough-shaping matters, order matters) turn out to matter operationally, that’s this project’s own thing to verify empirically, not something to inherit from an academic CTF paper.
• Structured attack trees / ATT&CK scaffolding (Nakano et al., arXiv:2509.07939, 2025-09-09, established mechanism, but this is scaffolding not weight-training — flag the distinction) — externally constrain rollout-time reasoning with a deterministic task tree built from MITRE ATT&CK’s kill-chain, filtering unproductive actions. Reports 71.8–78.6% subtask completion vs. 13.5–75.7% for self-guided reasoning at far fewer queries — a large, reproducible gap. The cheapest lever in this whole page: use the ATT&CK tree as a rollout-generation scaffold to harvest higher-quality, more methodical verifier-passed trajectories for the SFT corpus right now, with zero RL infrastructure — and it keeps knowledge in the tree (a prompt/tool artifact), not baked into weights, consistent with the project’s “knowledge in tools not weights” rule.
• PEARL (Wang et al., arXiv:2601.20439, 2026-01-28, promising) — treats the planning step (which tools, in what order) as its own object of RL, rather than only optimizing the final answer. Directly relevant as a formal mechanism for systematic tool-sequencing instead of react-and-guess, though not yet validated outside general multihop tool-use.
• Adjacent, unverified beyond title: classical-planner-hybridized LLM agents (arXiv:2512.11143) — a stronger, harder version of the ATT&CK-tree idea; worth a follow-up only if the tree scaffold proves insufficient.

Honest caveat: the load-bearing basis for the two-stage recipe above is domain-general (JSRL, DeepSeek-R1’s cold-start-then-RL disclosure), not the CTF-specific Pentest-R1 paper — per this project’s standing rule, an academic CTF/pentest paper being “in-domain” doesn’t make it stronger evidence, it makes it out of scope as a basis (no academic cybersecurity-LLM project has produced a frontier model). The ATT&CK-tree scaffold (Nakano et al.) is a separate, non-cybersecurity-specific mechanism (deterministic task-tree filtering built on a public taxonomy, evaluated as scaffolding not weight-training) and isn’t subject to the same caveat. The main open question either way is whether a walkthrough corpus built from this project’s own data generalizes past the small set of canonical challenges it would initially be built from — that’s this project’s own thing to test.

Verify the fix: measure PTES-phase coverage per episode (recon steps taken before first exploit attempt) and the pivot-after-failure rate (does the agent try ≥2 alternatives before abandoning a technique?) on held-out challenges, pre/post. Both are derivable from existing Phoenix spans without new instrumentation.

Pattern 3 — Good guessers until they’re not (many solves hinge on an ungrounded guess; brittle to a wrong first guess)

Observed: many successful runs pivot on a single ungrounded guess at a critical step; when that guess is wrong, the agent rarely recovers.

Gap type [E]/[R]: this is where entropy collapse and reasoning intersect — a policy that has already spent its exploration budget on one high-probability guess has no remaining probability mass on alternatives when that guess fails. Cui et al., “The Entropy Mechanism of RL for Reasoning Language Models,” arXiv:2505.22617 (2025-05-28, high confidence, mechanistic + empirical) is the underlying diagnosis this pattern shares with pattern 4: entropy falls monotonically and predictably (R = -a·exp(H) + b), and the mechanism is the covariance between a token’s probability and its advantage — the model reinforces what’s already likely rather than exploring what’s uncertain.

Training signal:

Designed to fix: pattern 3 — brittle single-guess behavior with no recovery on failure.

• SCoRe — self-correction via RL (Kumar et al., arXiv:2409.12917, 2024-09-19, established, canonical DeepMind paper) — SFT on (wrong→right) correction pairs barely transfers because it’s distribution-mismatched; only multi-turn RL with reward shaped to reward improvement, not just final correctness, produces genuine revision instead of mode-collapsing to “be right turn 1” or no-op-collapsing to “always change the answer”:

  r1 = verifier(attempt_1)
  r2 = verifier(attempt_2)
  reward = r2 + alpha * max(0, r2 - r1)   # bonus specifically for turning a miss into a hit
  

  Naming collision, flag explicitly: a different Sept-2025 paper reuses “SCoRe” for teacher-corrected earliest-error localization + short-horizon RL-continue-from-verified-prefix (Lyu et al., arXiv:2509.14257, promising, 7B-matches-72B claim unreplicated). Cite by arxiv id, not the shared name. If a 100-turn episode fails at a single identifiable wrong-guess turn (which matches this exact failure shape), earliest-error-localized short-horizon RL is much cheaper credit assignment per episode than scoring the whole trajectory as one unit.
• CDE — curiosity-driven exploration (Dai et al., arXiv:2509.09675, 2025-09-11, medium-high, ICLR 2026 poster) — an actor-side perplexity bonus (reward the model for being “surprised” by its own output) reports a calibration collapse finding as a byproduct: the policy becomes confident regardless of correctness, which the perplexity bonus specifically counters. This is the literature-side twin of “commits to an ungrounded guess and doesn’t recover” — cheap to try (no extra network, just log-perplexity of the model’s own rollout) before anything heavier.
• Representation-based exploration (Tuyls et al., arXiv:2510.11686, 2025-10-13, medium-high) — an inference-time-only lever, not training: build a diverse k-of-N pass@k pool from hidden-state dissimilarity instead of random k-of-N. Notable negative result: this is anti-composable with high-temperature sampling — high-temp outputs look “novel” in representation space without being useful. Worth an ablation on whatever temperature the pass@k>1 eval pool currently uses, since it changes nothing about training and answers whether current sample diversity is real strategic variance or just noisier repeats.

Honest caveat: SCoRe (both papers) and CDE are domain-general (math/code) — the CTF/100-turn transfer is inference, not literature-verified. One concrete, cheap check the project can run without any new training: confirm the reward/credit-assignment scheme doesn’t implicitly favor shorter successful trajectories — a 60-turn recovery-from-wrong-guess success should score the same as a 10-turn lucky first guess; if it doesn’t, the reward is actively working against fixing this pattern regardless of which paper’s fix gets adopted.

Verify the fix: on a held-out set, measure whether backtracking-after-a-wrong-guess correlates with eventual success pre/post training (does the trained policy actually try a second technique after the first fails, and does that second attempt land more often?).

Pattern 4 — Uneven PTES phases: strong at chaining inside an exploit, weak at thorough enumeration (failures often stall in exploitation)

Observed: the agent is good at following a known exploit chain once inside it, but weak at the enumeration/reconnaissance breadth that would get it there in the first place; most failures stall during exploitation rather than in an earlier phase.

Gap type [L]/[E]: two compounding gaps. First, entropy collapse narrows the recon repertoire — once RL reinforces whatever enumeration path happened to work once, alternative recon strategies stop being tried (same mechanism as pattern 3, arXiv:2505.22617). Second, a long-horizon credit-assignment problem: a flat terminal-only reward over a ~100-turn episode gives early, correct enumeration steps the same undifferentiated credit as late exploitation steps — so even when enumeration was necessary for the eventual win, nothing in the reward signal reinforces it specifically.

Training signal:

Designed to fix: pattern 4 — weak/uneven enumeration and stalling mid-exploitation.

• DAPO (Yu et al., arXiv:2503.14476, 2025-03-18, established, widely reproduced) + the Entropy Mechanism paper (arXiv:2505.22617) together are the baseline recipe, not an optional add-on, once GRPO starts: clip-higher (decouple the PPO clip range so rare-but-good tokens aren’t capped as hard as likely ones) and dynamic sampling (drop degenerate all-correct/all-wrong prompt groups, which otherwise contribute zero gradient — this matters disproportionately here since a whole-group-zero-reward challenge at ~100 turns/rollout is expensive to keep resampling; consider curriculum-filtering to the 30–60% pass-rate band the project already targets, rather than paying for blind resamples on genuinely-unsolved challenges).

  eps_low, eps_high = 0.20, 0.28              # decoupled clip (vanilla PPO/GRPO: symmetric 0.20/0.20)
  ratio = exp(logp_new - logp_old)
  clipped = clip(ratio, 1 - eps_low, 1 + eps_high)
  loss_pg = -min(ratio * adv, clipped * adv)   # per-token, mean over ALL tokens in batch
  

• GiGPO (Feng et al., arXiv:2505.10978, 2025-05-16, established, NeurIPS 2025 poster) — the most directly transplantable long-horizon idea in this map, and it needs no new infra: no critic, no extra rollouts. It adds a second, step-level advantage on top of GRPO’s trajectory-level one by retroactively hashing (state, step) pairs that recur across rollouts and scoring “what happened next” conditioned on that shared state — pure post-hoc bookkeeping on trajectories already sampled. This is exactly the mechanism that could stop rewarding all 100 turns equally when only the exploitation phase decides the outcome. Failure mode to flag honestly: GiGPO’s state-hashing was validated on benchmarks with hashable states (web pages, grid worlds); a CTF agent’s state is unbounded free text (shell/HTTP output), so state-canonicalization needs a bespoke similarity function — e.g. (tool_name, normalized_target, response_status_class) from the existing tool_call/tool_exec_ms spans — rather than a raw-text hash, or the anchor groups never fire.
• HiPER / hindsight credit assignment (Peng et al., arXiv:2602.16165, 2026-02-18; Tan et al., arXiv:2603.08754, 2026-03-07, both promising, 0 citations, evaluated on WebShop/ALFWorld — domain transfer to cybersecurity is inference, not literature-verified) — explicit hierarchical decomposition: a planner proposes subgoals (recon-done, foothold-gained, priv-esc-done, flag-captured), each independently checkable against environment state, with terminal reward at the flag level and intermediate credit at the subgoal level. The PTES phases already tracked are a natural subgoal taxonomy for this — no new taxonomy needed. If the flag-verifier is extended to also check an intermediate condition (“foothold confirmed” via sandbox state), that stays deterministic and doesn’t violate the ground-truth-verified-reward rule.
• RL-PLUS (Dong et al., arXiv:2508.00222, 2025-07-31, promising, strong math/code ablations, cybersecurity transfer untested) — names “capability boundary collapse” directly: pass@k at large k drops under pure on-policy RLVR even as pass@1 rises, i.e. uneven phases get more uneven as training narrows the policy. Its fix: mix verifier-passed off-policy trajectories (already produced by rejection sampling!) into GRPO via importance-sampling correction, plus an advantage bonus for visiting under-explored-but-successful states. Concretely: don’t discard the rejection-sampling SFT corpus once GRPO starts — feed it back in as off-policy anchor data.
• NuRL (Chen et al., arXiv:2509.25666, 2025-09-30, medium, consistent gains across six benchmarks/three models) — targets prompts with zero reward across every rollout in the group, which vanilla GRPO simply cannot learn from (zero gradient). Self-generates a hint conditioned on the gold answer, re-rolls with the hint injected, trains on the hint-augmented rollout, then drops the hint at inference. This is the strongest candidate for the ~800 currently-unsolved challenges in the portfolio — but the “gold answer” analogue doesn’t port zero-shot: the flag itself isn’t the how-to- get-there knowledge, so a CTF adaptation needs a hint source (walkthrough/verifier metadata, or a successful trajectory from a similar challenge family) that doesn’t yet exist and requires design work.

Verify the fix: instrument mean policy entropy from step 0 of any GRPO run (this alone is diagnostic, not a fix — but without it you cannot tell “the task is hard” from “the policy already collapsed at step 50 and is just getting faster at one script”); track per-PTES-phase solve/stall rates before and after each intervention layer (DAPO → GiGPO → RL-PLUS/NuRL); and run RL-PLUS’s own diagnostic — pass@k at large k on the base model vs. the trained checkpoint — to check whether exploitation-phase gains are holding or narrowing over training.

Pattern 5 — Benchmarks measure pattern-match speed, not thoroughness/methodology/robustness

Observed: a rising solve-rate number doesn’t by itself tell you whether the agent generalized a strategy or pattern-matched something close to a memorized/leaked challenge shape.

Gap type [N]: this is an eval-methodology gap, not a training-loop gap directly — but it’s the project’s own check on whether the SFT→GRPO pipeline is doing what axis [N] (novelty/boundary-expansion) requires, versus merely axis-amplifying what the base model already does.

Training signal (mostly diagnostic, one eval-recipe change, one eval-recipe addition):

Designed to fix: pattern 5 — solve-rate gains that can’t be told apart from elicitation/memorization.

• Yue et al., “Does RL Really Incentivize Reasoning Capacity Beyond the Base Model?,” arXiv:2504.13837 (2025-04-18, established, large systematic study across model families/algorithms) — across math/code/visual-reasoning and 6 popular RLVR algorithms, the base model catches up and overtakes at large pass@k even when RL wins at pass@1: the patterns RL concentrates on were already latent in the base model’s sampling distribution. This paper’s own recommended fix — evaluate pass@k at large k, not just pass@1 — is already this project’s locked methodology (pass@k, k=3/5/10 bands per decisions/2026-06-11-ctf-benchmark-pass-count.md); the missing piece is running the base model at the same pass@k bands as a mandatory control. If base-model pass@10 ≈ trained-model pass@10 on a chunk of challenges, that chunk’s improvement is elicitation, which is fine to attribute to the SFT stage (SFT is explicitly meant to replace/instill, not expand) but would be a red flag if it persists after GRPO, where genuine capability gain is expected.
• Semantics-preserving-transform robustness testing, re-grounded on general (non-cybersecurity) evidence: GSM-Symbolic — Mirzadeh et al. (Apple), arXiv:2410.05229 (2024-10-18, established, frontier-lab work) is the domain-general instance of the same failure mode: regenerating GSM8K-style math questions from symbolic templates — same structure, only surface values/names changed — shows LLM solve rates drop and get noticeably more variable under these semantics-preserving perturbations, and performance degrades further as templates add clauses that shouldn’t change the answer. This is established, cited-widely evidence (independent of any cybersecurity-domain paper) that a rising benchmark number can reflect pattern-match on the specific surface form rather than a robust, generalized method — exactly the risk pattern 5 flags for this project’s own solve-rate numbers. Concrete recipe for this project: build semantics-preserving variants of the existing challenge set (renamed services/users, reordered but logically-identical steps, cosmetic code/config changes that don’t alter the exploit path) and check whether a claimed solve-rate jump transfers to a transformed variant before crediting it to “the agent got better at CTF,” mirroring GSM-Symbolic’s methodology directly. Academic CTF-benchmark work — cited for context only, not a basis: Capture the Flags / Evolve-CTF (Honarvar et al., arXiv:2602.05523) runs the identical idea in the CTF domain specifically (source-transformation families, composed-obfuscation degradation) and would be the more directly-transplantable recipe if it counted as evidence — but per this project’s standing rule it’s an academic domain-specific CTF-benchmark paper, so it doesn’t serve as the basis here; GSM-Symbolic does.

Honest note: this pattern’s fix is mostly an eval-protocol addition, not a new training signal — the “fix” for pattern 5 is really making patterns 1–4’s fixes falsifiable. Both papers above are cheap (no new training) relative to any of the RL infrastructure work in patterns 1–4 and should run before crediting the first GRPO graduation with “fixing execution,” not after.

Verify the fix: run the base-model pass@k-at-large-k control (5.1) alongside every RL-trained checkpoint’s pass@k; separately, generate semantics-preserving-transform variants of a held-out challenge subset and check whether solve-rate gains survive the transform. If gains evaporate on either check, that’s elicitation/memorization, not the execution-reliability improvement the rejection-sampling-SFT diagnosis is banking on.

Summary table

What this changes about the plan, concretely

• Cheapest, no-training-required move first: the ATT&CK attack-tree scaffold (pattern 2) improves rejection-sampling SFT corpus quality today, before any RL infra exists.
• When GRPO starts, DAPO’s clip-higher + dynamic sampling is the baseline, not an optional add-on — it’s simultaneously the fix for patterns 3 and 4’s shared entropy-collapse mechanism.
• GiGPO is the single most transplantable long-horizon idea (pattern 4) — zero new infra, just a state-canonicalization function over the existing tool_call spans.
• Keep the rejection-sampling SFT corpus as off-policy anchor data through GRPO, not a disjoint earlier stage — RL-PLUS’s argument (pattern 4) applies directly since that data already exists.
• Pattern 5’s checks (base-model pass@k control, semantics-preserving-transform families) should run before the first GRPO graduation is credited with anything — they’re the cheapest falsification test available and gate whether patterns 1–4’s fixes actually expanded capability or just re-elicited it.
• Contested / open: whether GiGPO/HiPER-style credit assignment transfers from hashable (WebShop/ALFWorld) state spaces to a CTF agent’s unbounded free-text environment state is this project’s own thing to test, not something published literature has already settled — say so plainly if this page gets cited externally.

Diagnosing the gap — a scientific framework

The question this chapter answers: is there an industry-standard, peer-defensible way to prove a failure is a KNOWLEDGE gap, not an EXECUTION gap, not an EXPLORATION gap? Short answer, upfront: no single accepted instrument exists. There is no ISO-9001 for capability diagnosis. What exists is a converging set of measurement techniques, each independently validated, that — combined into one protocol — give you a defensible, falsifiable, split verdict. That protocol is what this chapter hands you.

Every arXiv id below was verified live against arxiv.org/abs/<id> on 2026-07-02 (project research pass, artifacts/overnight-rl-sweep/research/diagnosis.md). Confidence tags follow that pass: [HIGH] peer-reviewed/heavily reproduced, [MED] coherent preprint not yet contested, [LOW] single small-N preprint.

Bottom line up front

For a large, long-horizon, ground-truth-flag-verified CTF portfolio at a low k=1 solve rate, the honest, defensible answer will not be a single sentence. It will be: X% of the currently-failing challenges are a knowledge gap, Y% are an execution/performance-floor gap fixable by elicitation, Z% are an exploration gap that needs on-policy RL, not more demonstrations — and here is the measurement that sorted each challenge into its bucket. That heterogeneous, per-challenge-subtype verdict is itself the scientifically credible output — collapsing it into “it’s execution not knowledge” is exactly the move a skeptical reviewer will catch you on.

1. Three gap types, defined precisely

Ground the vocabulary in the 60-year-old linguistics/cognitive-science split this whole ML debate re-derives without citing: competence (what the system can in principle produce) vs. performance (what it actually produces under real constraints — prompting, memory, self-verification, time) — Firestone, “Performance vs. competence in human–machine comparisons,” PMC7604508 [HIGH], and its LLM-era instance splitting formal competence (linguistic surface mastery) from functional competence (using it in the world) — Mahowald et al., arXiv:2301.06627 [HIGH].

The exploration gap is the one that’s easy to misdiagnose as a knowledge gap if you only look at a single snapshot: it’s a process failure (the training loop killing coverage that existed a step ago), not a static property of the base model. Section 4 below is the test that tells these two apart.

Finer-grained refinement — Gap taxonomy. That chapter refines this table’s Execution row into two sub-signatures at different granularity — macro/plan-level (R, ranked below a shallow default) and micro/decision-point (P, exposed by a different elicitation channel) — and derives them as one continuous axis, not two boxes (its §0/§2). In this chapter’s coarser vocabulary: Execution = R ∪ P. Its Knowledge gap is the same category as this chapter’s Knowledge row. Exploration here is not a fourth static bucket in either chapter — it’s what an R/P-classified failure becomes when the winning path is sequentially-gated (§2.4 below is exactly that segmentation test).

2. The core instrument: pass@k → Cover@τ → Pass@(k,T)

2.1 pass@k as a coverage probe [E]

The unbiased pass@k estimator (the one this project already uses at eval time, per decisions/2026-06-11-ctf-benchmark-pass-count.md):

def pass_at_k(n, c, k):
    """n = samples generated, c = number correct, k = budget."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

Sampling k completions per problem at large k and plotting the curve is, structurally, a coverage measurement — the probability mass the policy places on any correct completion. The theory for why this works: the Coverage Principle — cross-entropy loss is dominated by tokens irrelevant to correctness, but coverage (mass on high-quality responses) is necessary and sufficient for post-training / test-time scaling to succeed, and estimates faster than loss does. Chen et al., arXiv:2510.15020 [MED]. [E]

2.2 The crossover test — does RL amplify or replace? [E][N]

The core instrument. Run the base model and your trained checkpoint through the same challenge set at k = {1, 4, 16, 64, 256…}. Plot both pass@k curves.

• Base catches up to or exceeds trained pass@k at large k → the training only reweighted an existing distribution (elicitation, not new capability). Yue et al., “Does RL Really Incentivize Reasoning Capacity Beyond the Base Model?,” arXiv:2504.13837 [HIGH] — the founding result. 6 RLVR algorithms tested; all “remain far from optimal in leveraging the base model’s potential.”
• Trained pass@k pulls away and widens as k grows → real capability expansion.

Designed to fix a common failure: benchmarks measuring pattern-match speed, not thoroughness -> report the crossover, not just pass@1. Yue et al.’s own prescription is exactly “report pass@k at large k, with base model as control” — not just pass@1. Report base-model pass@k at the same k as a mandatory control column in every benchmark table you publish.

Contested rebuttal — CoT-Pass@K: pass@k credits a correct final answer even from a wrong chain-of-thought (a lucky guess). Require the reasoning path itself to be correct and the crossover disappears — RLVR shows monotonic gains at every k. Wen et al., arXiv:2506.14245 [MED]. State this as contested when presenting — it directly falsifies the load-bearing assumption of §2.2’s headline result. For your agent, this maps onto a real risk in your own SFT curation: a verifier-passed trajectory can still contain wrong/wasted turns before the winning one — a commonly-observed pattern where a nontrivial fraction of solves hinge on an ungrounded guess at a critical step rather than sound reasoning throughout. Filter on trajectory soundness (backtracking, wasted turns, tool-call validity), not just flag==1, or you reproduce the exact confound this paper diagnoses.

2.3 Cover@τ — punish guessing, reward reliability [E]

pass@k at huge k conflates “genuinely solvable” with “eventually guessable by brute force.” Cover@τ(q) = 1 if ≥ τ·n of n samples on problem q are correct — a reliability threshold, not a “did any sample land” threshold. Dragoi et al., arXiv:2510.08325 [MED]. Relative RLVR-algorithm rankings change under Cover@τ vs pass@1 — some algorithms that look best on pass@1 are worse at genuine reliability.

Designed to fix a common failure: high pass@k that masks guessing rather than genuine reliability (“good guessers until they’re not”) -> Cover@τ as a second axis. A challenge with high pass@64 but near-zero Cover@0.3 is guessing-dominated — rejection-sampling SFT on its lucky wins teaches the model to guess more confidently, not more competently. Report Cover@τ (τ≈0.3) alongside pass@k as a second axis in every benchmark table.

2.4 Pass@(k,T) — the agentic extension, and the single most load-bearing citation in this chapter [L][E][N]

Everything above is validated on static, single-shot reasoning (math). Your agent is T-round tool interaction. Zhai et al., “Does RL Expand the Capability Boundary of LLM Agents? A Pass@(k,T) Analysis,” arXiv:2604.14877 [MED], asks the crossover question with interaction-depth as a second axis:

PASS@(k,T)(q,π) = 1 - C(n - c_T, k) / C(n, k)

— identical to standard pass@k, except c_T counts correct at interaction depth T, not overall.

Their finding flips §2.2 on compositional tasks: on Category C (compositional, sequentially-gated information gathering — structurally identical to enumerate-then-chain vuln discovery), the RL curve pulls above and widens against the base curve as k grows — the opposite of the static-reasoning crossover. On independent-retrieval tasks the effect is small; on pure static reasoning (no tool, negative control) RL is inert, replicating Yue et al.

The critical additional result: matched-data SFT actually regresses the capability boundary on the same compositional tasks (net −4 vs RL’s net +4). This isolates self-directed exploration during RL — not data exposure — as the causal factor for expansion.

Designed to fix a common failure: uneven PTES-phase performance — agents chain well inside an exploit but stall in enumeration/reconnaissance -> compositional segmentation. Category C (“sequential retrieval”) is structurally identical to “enumerate-correctly-at-turn-5-before-turn-40’s-exploit-becomes-visible.” This is the paper that tells you where your exploration gap is likely to live on the portfolio.

The single highest-value experimental design in this chapter: segment your challenge portfolio by whether the winning path is (a) single-shot / not sequentially gated (their Cat A/B analog), or (b) genuinely compositional/sequentially-gated (Cat C analog — your “enumeration-gates-exploitation” weak spot). Run Pass@(k,T) on both segments, before and after rejection-sampling SFT. The falsifiable prediction: on (a), SFT works fine, further RL may plateau (§2.2’s static result holds — an execution gap, cheaply closed). On (b), SFT alone regresses capability and you need on-policy RL specifically — an exploration gap, not an execution gap, and rejection-sampling SFT is the wrong tool for it. This is directly testable this week and determines whether “SFT now, GRPO later” has the ordering right for the compositional subset, or whether it needs RL first.

3. Does base/SFT pass@k predict RL gains, before you spend the compute?

High SFT-stage scores are not reliably predictive of eventual RL performance — sometimes inversely so. What does predict post-RL pass@1: generalization loss on held-out examples and pass@large-k on the post-SFT checkpoint, with up to 2× better R²/Spearman correlation than post-SFT pass@1 alone. Kang et al. (Meta FAIR + Virginia Tech), “Quagmires in SFT-RL Post-Training,” arXiv:2510.01624 [HIGH] — >1M GPU-hours, hundreds of models to 12B, 7 math benchmarks, up to 256 repetitions.

Training-loop delta: add a cheap diagnostic gate between SFT and RL. Before launching a GRPO run, compute pass@64 (or larger) on your rejection-sampling-SFT checkpoint, cold-start, pdq --fresh-retries, on the held-out challenge set. If it’s flat/low, don’t trust the SFT accuracy number as a green light — this predicts a disappointing GRPO run regardless of how good SFT looked.

What changes in your graduation criterion: the project’s stated trigger is “graduate to GRPO/RLVR when policy entropy collapses.” Add a second, independent gate: AND pass@64 on held-out challenges is non-trivial. Entropy collapse tells you SFT has converged; pass@64 tells you there’s still coverage headroom worth converting. Both are needed — collapsed entropy with flat pass@64 means you’ve converged onto a policy with nothing left to reinforce.

4. The routing test — does the correct action ever appear at high N?

This is the book’s existing one-line diagnostic (see The decision), and it deserves the fuller justification here because it is doing real theoretical work, not just intuition:

• Never, at any N, on any checkpoint → knowledge gap. This is the elicit-not-expand result again, at the boundary: RLVR only reweights mass the base model already places somewhere in its distribution — full evidence chain (Yue elicit-not-expand, SFT-memorizes/RL-generalizes, the contested Scalpel-vs-Hammer/OOD-healing pushback) in The frontier recipe is a sequence §3. Inject off-policy (demonstration, teacher, or — cheaper — put the missing fact in a tool, not the weights).
• Sometimes, and pass@(k,T) shows RL (not matched SFT) expanding it (§2.4) → exploration gap. Coverage exists but training-time entropy collapse is what’s suppressing it turn-to-turn; the fix is on-policy RL with explicit exploration preservation, not more demonstrations.
• Sometimes, and a few-shot prompt or a handful of SFT demos recover it (§5) → execution gap / performance floor. Cheap fix, no RL infrastructure needed.

This routing test is directly the mechanism §2.2–§2.4 give you a rigorous way to run, not just eyeball. The crossover is the “does it ever appear at high N” test, made continuous and plottable rather than binary.

5. The elicitation ladder — a borrowed, rigorous “prove it’s X not Y” methodology

The AI-safety capability elicitation literature already had to solve almost exactly this problem — “is the model bad at this, or is the capability locked away where naive prompting can’t reach it?” — because sandbagging/eval-integrity depends on the same distinction. Their protocol, adapted: build (or find) a case where you don’t yet know if the capability is present, then try elicitation techniques in escalating cost order, and whichever one first recovers the behavior tells you what kind of gap it was.

prompt / few-shot  →  SFT on a handful of demos  →  RL against a judge/verifier  →  (still nothing) → knowledge gap
   (cheapest)              (cheap)                    (expensive)

• Password-locked models: fine-tune a model to only show a capability when a password token is present (imitating a weaker model otherwise), then test whether elicitation techniques recover it. Finding: a few high-quality SFT demonstrations are often sufficient to fully elicit a locked capability; RL can also elicit it when demos aren’t available, but less reliably. Greenblatt et al., arXiv:2405.19550 [HIGH].
• Harder organism (circuit-breaking, not just password-locking): prompting/steering fail; only fine-tuning-based elicitation succeeds. Recommendation: “evaluators should use fine-tuning techniques to elicit capabilities when possible” — if prompting can’t get behavior out, don’t conclude the capability is absent; fine-tune first, then conclude. Hofstätter et al., arXiv:2502.02180 [MED].
• Operational definition of real capability: what can be elicited at ≤1% of total training cost (Anthropic RSP’s own definition, directly reusable). van der Weij et al., arXiv:2406.07358 [HIGH, ICLR 2025].
• Order matters: neither SFT nor RL alone reliably elicits held-back performance from a degenerate policy; SFT on weak demonstrations first, then RL, is what fully elicits it — RL-first “almost always leads to reward hacking rather than genuine improvement” starting from a degenerate policy. Ryd et al., arXiv:2604.22082 [MED, 2026].

Designed to fix a common failure: agents preferring raw shell/HTTP over a provided higher-level tool surface, leaving much of that surface unused — tool-selection is known to be unreliable and highly sensitive to how tools are described/exposed (Faghih et al., “Tool Preferences in Agentic LLMs are Unreliable,” arXiv:2505.18135) — -> run the elicitation ladder. This is the cheapest, most directly actionable experiment in the whole chapter. Take a handful of ignored tools and run the ladder: (a) few-shot prompting with 2–3 correct-usage examples recovers usage → pure elicitation/prompting gap, no training needed; (b) SFT on a small demonstrated-usage set recovers it → elicitation via light fine-tuning (matches Greenblatt’s finding); (c) neither works → genuinely a missing-knowledge/affordance problem, and rejection-sampling SFT should specifically upweight trajectories that exercise those tools. This turns “the model prefers curl” from an anecdote into a falsifiable, paper-backed experiment.

5.1 The wrinkle mid-episode: the self-verification cliff [E][R]

Your flag verifier is external, ground-truth, and perfect — exactly the regime where BoN/rejection-sampling/RL should work without a ceiling (Stroebl et al., “Inference Scaling fLaws,” arXiv:2411.17501 [HIGH]: with an imperfect verifier the false-positive floor is non-removable even at infinite compute; a perfect verifier has no such floor). That’s a load-bearing reason the project’s “ground-truth-verified reward, never regex” rule is correct.

But the verifier only fires at submission — at turn 40 of 100, the agent must judge without it whether its current path is worth continuing. That’s exactly the regime multiple papers show degrades, not improves, with capability: models find a correct answer among k samples far more often than they can self-select it, and the self-selection gap widens with generator capability (contested/preliminary — one 2026-06 OpenReview submission, no confirmed arXiv id, treat as [LOW], flag if cited). Corroborating, harder evidence: Best-of-N provably degrades past a reward-hacking threshold even with a competent reward model — scaling samples isn’t monotonically good. Huang et al., arXiv:2503.21878 [HIGH, ICML 2025]. Formal geometry: rejection-sampling and Best-of-N both converge to a ceiling set by the verifier’s ROC curve; more samples cannot buy past it. Dorner et al., arXiv:2507.12399 [MED].

Designed to fix a common failure: agents abandoning a viable path after a single setback rather than persisting (“good guessers until they’re not”) -> post-hoc pivot-point audit. Diagnostic, cheap, run on your existing run-trace corpus: log every point in a trajectory where the agent pivots/abandons a path, and check post-hoc whether the abandoned path was actually unproductive (e.g., did a later successful run on the same challenge use a similar path?). If abandoned paths are disproportionately ones that would have worked, that’s a self-verification-cliff signature — an execution/judgment gap, and the fix is a mid-episode progress signal, not more SFT demonstrations.

6. A two-type failure vocabulary — grounded in general agent/RL evidence, not domain benchmarks

Standing project rule: no conclusion here may rest on academic cybersecurity-LLM training/benchmark papers (CTF-Dojo-style work, pentest-agent papers, CTF-family robustness studies, etc.) — none of that literature has produced a frontier cybersecurity model. The domain-specific pentesting-agent papers below are mentioned for context only, not as a basis for anything in this chapter:

• Deng et al., “What Makes a Good LLM Agent for Real-world Penetration Testing?,” arXiv:2602.17622 — academic, cited for context — not a basis for our decisions. (28 pentesting systems, proposes a Type A “capability gap” / Type B “planning and state-management limitation” taxonomy, and an Evidence-Guided Attack Tree Search system, “Excalibur.”)
• Nakano et al., arXiv:2509.07939 — academic, cited for context — not a basis for our decisions. (Deterministic ATT&CK-derived task tree lifts subtask completion 13.5–16.5% → 71.8–78.6% on the same models — a pentest-benchmark result, not project evidence.)
• Shen et al., “PentestAgent,” arXiv:2411.05185 — academic, cited for context — not a basis for our decisions. (Frames its own motivating failure as a knowledge gap fixed with RAG; illustrative only of how unsettled domain-specific pentest-agent papers are, not evidence either way.)

The two-type vocabulary itself is worth keeping — it just needs a non-domain-specific foundation. Re-grounded on general long-horizon-agent evidence and RL theory:

• Capability/elicitation gaps — missing tools, inadequate prompts, absent demonstrations. Cheaply closed by better scaffolding, tool surface, or light SFT (§5’s elicitation ladder, itself grounded in the AI-safety elicitation literature, not domain-security work).
• Planning/state-management limitations — a structurally different failure mode that does not reliably close with a stronger base model or more knowledge alone. Four independent, non-cybersecurity anchors support treating this as a real, separate axis — two qualitative/theoretical, two now directly quantitative:
  1. Empirical, frontier-model, general-domain: METR’s time-horizon study finds that what separates frontier models on long-horizon tasks is reliability and the ability to adapt to their own mistakes, not per-step knowledge or reasoning quality — and this axis scales on its own trajectory, doubling roughly every 7 months, independent of raw capability jumps. Kwa et al. (METR), “Measuring AI Ability to Complete Long Software Tasks,” arXiv:2503.14499 [HIGH]. This is general evidence (RE-Bench/HCAST software tasks, no cybersecurity framing) that long-horizon degradation is a distinct axis from single-turn competence — exactly the property a “Type B” needs to be a real thing and not just restated knowledge-gap.
  2. Theoretical, classic RL/imitation-learning result: compounding error under covariate shift — a policy trained/evaluated with a small per-step error rate accumulates error quadratically in trajectory length because each mistake pushes the agent into states its training distribution under-covers, and no single-step fix removes this without addressing the sequential structure itself. Ross, Gordon & Bagnell (DAgger), arXiv:1011.0686 [HIGH, 840+ citations]. This is the general-theory reason a state-tracking/replanning failure at turn 40 of 100 can be structurally invariant to swapping in a stronger base model — the problem is the sequential decision process, not the weights.
  3. Direct quantitative test, general-domain, verified 2026-07-02: isolating pure execution (plan and knowledge handed to the model, so only turn-to-turn execution is measured), larger models within the same family do execute more correct turns — but per-step accuracy still degrades as turns accumulate, driven by a self-conditioning effect (the model becomes more likely to err once its own prior mistakes are sitting in context). The paper’s own framing is exactly this chapter’s question: “self-conditioning does not reduce by just scaling the model size” — it is removed only by switching training paradigm to a “thinking”/reasoning-trained model, not by a bigger non-reasoning model of the same family. Sinha, Arun, Goel, Staab & Geiping, “The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs,” arXiv:2509.09677 [MED, preprint]. This is the closest thing in general literature to a direct scale-invariance test of a planning/state-tracking failure — the honest reading is invariant to parameter scale within a training paradigm, not invariant to LLM full stop (reasoning-trained models are a real escape hatch, a bigger base model of the old paradigm is not).
  4. Direct quantitative test, general-domain, verified 2026-07-02: a minimal explicit-lookahead planning module (FLARE) bolted onto a much weaker base model lets LLaMA-8B outperform GPT-4o run with standard step-by-step reasoning on multi-step planning benchmarks — i.e. an eight-billion-parameter model with a planning fix beats a frontier model without one. This is a clean existence proof that the planning axis is separable from and can dominate raw base-model strength. Wang, Wu, Wang, Tang, Li, Yin, Ma, Li, Sun, Chen & Ye, “Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents,” arXiv:2601.22311 [LOW, 0-citation preprint — promising, not yet validated].
  • Corroborating, not separately load-bearing: τ-bench’s own cross-model pass^k leaderboard (already cited above, §6.1) shows the same steep pass^1→pass^8 reliability collapse for both gpt-4o and claude-3.5-sonnet — picking the “better” frontier model of a different family narrows but does not close the multi-trial consistency gap. Yao et al., arXiv:2406.12045 [HIGH]. And a 2026 cross-family diagnostic benchmark (GPT-5 variants + Claude models, 3100+ trajectories) documents the same horizon-dependent degradation pattern recurring across both families rather than being a one-model artifact. Wang, Bai, Sun, Wang, Zhang, Hu, Schroder, Mutlu, Song & Nowak, “The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break,” arXiv:2604.11978 [MED, preprint].

Designed to fix a common failure: agents stalling in exploitation after chaining well through enumeration — a plausible instance of exactly the compounding-error dynamic DAgger formalizes: an early misstep in enumeration pushes the trajectory off-distribution, and the deeper into exploitation the agent gets, the harder recovery becomes. Actionable, cheap: label a sample of your failed trajectories on two axes — (a) missing tool / bad prompt / no demonstration → capability/elicitation gap, cheap fix; (b) over-committed to a low-value branch, exhausted context near a dead end, no real-time replanning after a costly failure → planning/state-management gap. If (b) dominates, the theoretical prediction (DAgger), the empirical general-domain finding (METR), and the two direct quantitative tests above (Sinha et al.’s self-conditioning result; Wang et al.’s weak-model-plus-planning-beats-strong-model result) all say: don’t expect SFT-then-GRPO on raw episode reward alone to close it, and don’t expect swapping in a bigger base model of the same family to close it either — the fix needs to address the sequential/compounding structure (mid-trajectory checkpointing, explicit replanning triggers, a difficulty/progress signal, or a reasoning-trained backbone), not just more knowledge or more parameters. Flag, updated 2026-07-02: the qualitative distinction (capability vs. planning/state) is now well-grounded in four independent general-literature anchors, and two of them (Sinha et al. 2509.09677, Wang et al. 2601.22311) are direct quantitative tests of “does a stronger/bigger base model fix it” in non-cybersecurity settings — both say no, for different mechanisms (self-conditioning invariant to scale; planning-fix on a weak model beats a strong model without one). What general literature does not give you is this project’s own number — no source above measured the specific fraction of this portfolio’s failures that are planning/state-management vs. capability, nor whether it holds at this project’s turn-depths (~100) and challenge structure. Status: qualitative claim grounded; quantitative “X% invariant to base LLM” is worth pursuing for your own portfolio — must be measured on your existing run-trace corpus, not assumed from the general literature or the academic pentest-agent papers above.

6.1 Per-turn fault labeling — operationalize the taxonomy on your own trace corpus [R]

τ-bench provides an auto error-identification tool with a fixed taxonomy (fault assignment × fault type: used_wrong_tool, used_wrong_tool_argument, took_unintended_action, goal_partially_completed). Yao et al., arXiv:2406.12045 [HIGH]. AgentRx extends this to an automated framework that localizes the single critical failure step in a long trajectory, with a cross-domain taxonomy (Misinterpretation of Tool Output 24.1%, Intent-Plan Misalignment 24.1%, Under-specified Intent 27.6%, per their τ-bench column). Barke et al. (Microsoft Research), arXiv:2602.02475 [MED].

Your existing run-trace corpus is exactly the substrate this tooling wants. Label each turn as {reconnaissance-adequate, wrong-tool, wrong-argument, wrong-decision/policy, tool-output-misread, under-specified-plan} and compute: fraction of episodes with a labeled “under-specified-plan” or “wrong-decision” turn before the first “tool-output-misread” — this operationalizes a commonly-observed “no clear methodology” failure into a countable metric that separates planning (execution/skill — a scaffolding or SFT-curriculum fix) from interpretation (closer to reasoning/knowledge) from tool affordance (§5’s elicitation question).

7. A robustness cross-check: is the “gain” generalization or memorization?

Domain-specific aside (context only, not a basis for the method below): Honarvar et al., arXiv:2602.05523 — academic, cited for context — not a basis for our decisions. They build families of semantics-preserving CTF variants and find models robust to shallow transforms but degrading sharply under composed/deeper obfuscation, in the CTF domain specifically; per project rule, an academic CTF-benchmark paper cannot be the basis for the method below.

The methodology stands on its own, general (non-cybersecurity) grounding: semantics-/knowledge-preserving perturbation is an established way to separate genuine generalization from pattern-matching in general LLM evaluation. C-BOD rephrases MMLU questions with a parameterized, meaning-preserving transform and finds an average 2.75% performance drop across 32 SOTA models under modest rephrasing — with higher-performing, larger models showing greater sensitivity, i.e. bigger benchmark numbers can mean more surface-cue reliance, not less. Cohen-Inger et al., “Forget What You Know about LLM Evaluations — LLMs are Like a Chameleon,” arXiv:2502.07445 [MED, EMNLP 2025]. The same logic applied to code generation: rewrite a task’s ground-truth solution into a semantically-different-but-equal-difficulty variant and check whether the model’s answer degrades — a Memorization Risk Index that’s high only when the model reproduces a similar-looking answer and fails the rewritten task. Zhang et al., “Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting,” arXiv:2503.02296 [LOW, brand-new preprint, 0 citations — promising, not yet validated].

Before crediting any pipeline change with “closing an execution gap,” check the gain isn’t an artifact of the fixed set of canonical challenges the harness has seen many times, using the same semantics-preserving-perturbation logic C-BOD and the code-rewriting paper apply outside cybersecurity.

Designed to fix a common failure: solve-rate gains that don’t survive semantics-preserving rephrasing (memorization, not generalization) -> robustness cross-check. Cheap, no new training: generate meaning-preserving transformed variants of a held-out subset (renaming, restructuring, composed obfuscation — the transform families are generic; you don’t need a domain-specific benchmark paper to justify the check) and see whether a solve-rate gain transfers. If it evaporates on transformed variants, you have elicitation/memorization, not the execution-reliability improvement the SFT/GRPO diagnosis is banking on — and per C-BOD’s finding, don’t assume your strongest checkpoints are exempt; they may be the most exposed to this failure mode.

Also check the opposite failure mode post-RL: RL-PLUS names “capability boundary collapse” — pass@k at large k dropping even as pass@1 rises during RLVR, i.e. the on-policy training itself narrowing what the model can still do, not just what it does by default. Their fix mixes in the very off-policy verifier-passed trajectories rejection-sampling SFT already produces, via importance-sampling-corrected updates. Dong et al., arXiv:2508.00222 [MED]. This is the training-time confirmatory check for an exploration gap that got worse, not better, under GRPO — run pass@large-k before and after every GRPO checkpoint, not just pass@1.

8. The honest verdict: no single standard — here is the assembled protocol

No single accepted diagnostic instrument exists that a team runs once for a clean X-not-Y verdict. What the literature convergently offers, ranked by how load-bearing each is for this project:

1. Pass@k / Pass@(k,T) / Cover@τ curve decomposition (§2) — the closest thing to a standard quantitative instrument, but interpretation is contested even among its own authors (§2.2 vs its CoT-Pass@K rebuttal), and its agentic extension shows the diagnosis is task-structure-dependent: same metric, opposite conclusion, depending on whether the task is compositional/sequentially-gated or not.
2. Capability-elicitation methodology (§5) — a rigorous, falsifiable, cost-ordered protocol borrowed from AI-safety evaluation, directly reusable and cheap to run against the tool-avoidance finding.
3. Capability/planning two-type taxonomy + per-turn fault labeling (§6) — a general vocabulary grounded in long-horizon-agent measurement (METR), compounding-error theory (DAgger), and now two direct general-domain quantitative tests of scale-invariance (self-conditioning not fixed by scale, Sinha et al. 2509.09677; weak-model-plus-planning beats strong-model-without, Wang et al. 2601.22311) — not in domain-specific pentest-agent papers; the specific per-corpus “X% invariant to base LLM” number stays flagged as worth pursuing, to be measured on this project’s own corpus, not asserted from general literature.
4. The competence/performance vocabulary (§1) to frame the final answer for a skeptical reviewer: expect and report a split verdict by challenge subtype, not a single number.

The protocol — what to actually run, in order

Report all nine together, segmented by challenge subtype. Refuse to collapse it into one sentence — §2.4 and §6 both predict, and §7’s entropy check explains a mechanism for, the true answer being heterogeneous across the portfolio.

9. The decision, expanded

flowchart TD
  Start["Failing challenge / challenge-subtype<br/>under diagnosis"] --> Seg{"Winning path structure?"}

  Seg -->|"Single-shot / independent recon<br/>(Cat A/B analog)"| PKT_AB["Pass@(k,T): base vs SFT vs RL<br/>(arXiv:2604.14877)"]
  Seg -->|"Sequentially-gated:<br/>enum must succeed before<br/>exploit is even visible (Cat C)"| PKT_C["Pass@(k,T): base vs SFT vs RL<br/>(arXiv:2604.14877)"]

  PKT_AB --> X1{"Base pass@k(large k)<br/>>= trained pass@k?"}
  X1 -->|"Yes — crossover"| Elicit1["ELICITATION only:<br/>run the ladder (§5)<br/>before assuming knowledge gap"]
  X1 -->|"No — trained pulls away"| Exec1["EXECUTION gap:<br/>rejection-sampling SFT → GRPO<br/>is the right ordering"]

  PKT_C --> X2{"Does the correct action<br/>ever appear at large N,<br/>on ANY checkpoint?"}
  X2 -->|"Never"| Know["KNOWLEDGE gap:<br/>inject off-policy<br/>(SFT / teacher / TOOL)"]
  X2 -->|"Yes — but matched-data SFT<br/>REGRESSES it (§2.4 test)"| Explore["EXPLORATION gap:<br/>on-policy RL required,<br/>NOT more demonstrations"]
  X2 -->|"Yes — few-shot prompting<br/>recovers it"| ElicitP["Performance floor:<br/>cheap prompting/scaffold fix<br/>(§5)"]
  X2 -->|"Yes — only light SFT<br/>on demos recovers it"| ElicitS["Elicitation via light SFT<br/>(arXiv:2405.19550)"]

  Elicit1 --> TypeAB["Capability vs. planning/state<br/>label the failures<br/>(arXiv:2503.14499, arXiv:1011.0686)"]
  Exec1 --> TypeAB
  Explore --> TypeAB

  TypeAB -->|"Capability: tool / prompt gap"| FixA["Engineering fix:<br/>tool surface, scaffolding,<br/>walkthrough-shaped SFT data"]
  TypeAB -->|"Planning/state: difficulty-<br/>estimation, compounding error<br/>(test invariance on own data)"| FixB["Architectural fix:<br/>difficulty-gating / attack-tree<br/>search wrapper — NOT more<br/>SFT-then-GRPO on raw reward"]

  classDef your fill:#132b22,stroke:#34d399,color:#eafaf3;
  class Exec1,Explore,TypeAB your;

This is the same routing question as The decision — does the correct action ever appear in π_θ’s own outputs at high N? — expanded with the two tests that make it rigorous for an agentic, T-round, sequentially-gated task instead of a single-shot one: the compositionality segmentation (§2.4) and the matched-data-SFT-regression test that separates a genuine exploration gap from a knowledge gap when coverage exists but training destroys it.

Cross-links

• Gap taxonomy — the fine-grained refinement of this chapter’s three-way split: its K = this chapter’s Knowledge; its R ∪ P (one continuous axis, macro/micro sub-signatures — its §0/§2) = this chapter’s coarse-grained Execution; its “exploration gap” language names the same training-dynamics modifier this chapter’s Exploration row describes (an R/P failure on a sequentially-gated task, not a fourth static category). Canonical for the gap definitions; this chapter stays canonical for the instrumentation (pass@k/Cover@τ/Pass@(k,T)) and the fast three-way routing label.
• The decision — the one-line version of this chapter’s routing test; this chapter is its full justification.
• Contested edges & landmines §1 — “RL can’t create capability” is contested exactly along the lines §2.2/§2.4 draw out (recipe-dependent, not a law).
• Agentic & multi-turn RL — where the exploration-gap fix (on-policy RL, entropy preservation) is implemented once diagnosed.
• Imitation — SFT · distillation · rejection sampling — where the execution-gap and knowledge-gap fixes live.
• memory/research/ (shared pool) — long-horizon.md and exploration.md research notes underlie §2.4/§7’s turn-level and entropy-collapse mechanics respectively; not re-derived here to keep this chapter’s scope to diagnosis, not fix.

Bibliography (all verified live via arxiv.org/abs/<id>, 2026-07-02)

Academic cybersecurity-LLM domain-specific work — mentioned in §6/§7 for context only, per standing project rule NOT a basis for any claim/decomposition/recipe/verdict/number in this chapter (none of these produced a frontier cybersecurity model):

Flagged, not cited as fact: “The Self-Verification Cliff” (OpenReview, 2026-06-17) — no confirmed arXiv id as of this writing. Treat as [LOW], directional-only, per §5.1.

The three gaps — knowledge · prior · policy

The question this chapter answers: your agent doesn’t do the expert thing. On a fresh challenge it fires a scattered curl or dig instead of running a recon methodology; it reaches for subfinder but never amass; and when it does run subfinder it uses the defaults, not your flag-set. Before you spend a single GPU-hour on SFT, DPO, or GRPO, you have to answer one question: which of three gaps is this? Because the three have different, non-substitutable fixes, and picking the wrong lever doesn’t just waste the run — for one of them it actively makes the model worse.

This short chapter is the map. The six chapters after it are the territory, each fully cited.

Bottom line up front

Your symptom decomposes into three candidate gaps, and the whole game is telling them apart:

The load-bearing fact that makes K genuinely distinct from R/P — and this is a theorem, not a heuristic — is the closed-form of every KL-regularized objective. DPO’s optimum is π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β) arXiv:2305.18290, and RLVR is formally bounded to the base model’s support arXiv:2507.14843. Any RL or preference method can only redistribute probability mass the reference policy already assigns somewhere. If π_ref(y*) ≈ 0, the gradient has nothing to grab. That is precisely why “just RL it” fails on a knowledge gap and why the diagnosis must come before the training run.

The headline: your tool/flag failure is almost certainly a P-gap

You told me the failure is universal across frontier models — every one of them reaches for the salient tool and the default flags. That universality is itself the diagnosis: it’s a policy-prior signature, not a weak base model. Generic instruction+agentic post-training optimizes plausible general helpfulness, which sharpens the default distribution toward the single most-salient action. Expert procedure (the full tool-set, the full flag-set, a disciplined sequence) is long-tail behavior that generic RLHF under-rewards. So no lab’s post-training taught it — which reframes your fix from “get a better model” to “supply the data + reward that make expert methodology the default policy.”

Two 2026 mechanistic results make this concrete for exactly your example (both recent/low-citation — promising, not yet broadly validated — but strikingly direct):

• Akgül et al., It’s Sparse Policy Selection, Not Capability Learning arXiv:2605.06241: RL changes only 1–3% of token positions, concentrated at high-entropy decision points, and the token it promotes is always already in the base model’s top-5 logits. Operationalized: if the correct tool/argument token sits in the base model’s top-5 at the decision step, you are in a checkable P-gap and RL/DPO is well-targeted.
• Chen, Looking Is Not Picking arXiv:2606.16364: on real tool-call failures, per-candidate attention shows the model attends to the correct tool 80% of the time (vs 21% chance) yet still calls the wrong one. Prompt-side fixes (reordering/duplicating the tool in the schema) recover ≤23% of failures; decision-readout-layer interventions recover 59–91%. The model sees amass; it mis-picks at the readout. That’s a P-gap, and it tells you where the fix lands (the decision layer, i.e. on-policy preference/RL) — not in the knowledge.

The deep version of this taxonomy and its three-literature genealogy is in the taxonomy chapter.

The diagnosis, on one screen

You don’t guess which gap — you run a cheapest-first probe battery. The full runnable version (with numbers, pseudocode, and readouts) is Diagnosing which gap; here is the skeleton:

flowchart TD
  S["Symptom: agent won't run amass /<br/>won't use expert flags / no opening plan"] --> PK["pass@k sweep: grep 50 rollouts<br/>for the expert action (temp 0.7-1.0)"]
  PK -->|"appears sometimes"| RP["in-support → R or P gap"]
  PK -->|"never, even at k=256<br/>across ≥10 rephrasings"| ORC["in-context ORACLE probe:<br/>put the answer/flags in the prompt,<br/>re-run"]
  ORC -->|"now it does it"| Kd["was surfacing, not absent →<br/>R/P gap after all"]
  ORC -->|"still can't, even handed the answer"| K["KNOWLEDGE gap →<br/>inject (curation chapter)"]
  RP --> TOK["token-level: is the expert token<br/>in the base model's top-5?<br/>(arXiv:2605.06241)"]
  TOK -->|"yes"| P["POLICY gap →<br/>on-policy DPO / GRPO<br/>(1-3% of tokens move)"]
  TOK -->|"in-support but buried"| R["PRIOR gap →<br/>cold-start SFT re-prior"]
  classDef fix fill:#132b22,stroke:#34d399,color:#eafaf3;
  class K,P,R fix;

The two probes that carry the most weight, in plain terms:

pass@k = run the same task N times (say 50) at temperature ~0.8 and ask “does the expert action ever appear?” — score it with the unbiased estimator 1 − C(N−c,k)/C(N,k), never naïve c/k. Ever appears → in-support → R/P. Never, even at k=256 across ≥10 rephrasings → a knowledge-gap candidate. (Caveat: for pure factual recall a small k can false-positive a K-gap arXiv:2605.07153 — push k high, and on a multi-turn agent use Pass@(k,T), not Pass@k.)

in-context oracle probe = put the answer in the prompt ("expert workflow: subfinder -d T -all -recursive AND amass enum -passive -d T") and re-run. If it now executes correctly, the machinery was there and only the default policy failed to deploy it → R/P gap. If it still can’t even when handed the answer → genuine K gap. This is the single cleanest K-vs-P splitter you can run in an afternoon, no training.

The warning you already sensed: trajectory SFT amplifies an unfixed gap

Your instinct — “everyone jumps to trajectory-level [SFT]; maybe that’s been amplifying the problem” — is correct and citable. If you SFT trajectories that use knowledge the base model doesn’t actually have, you teach it to imitate the surface form of expert tool-use without the grounding: new-fact SFT rows are learned slower and, once learned, linearly increase hallucination on previously-known facts arXiv:2405.05904. Layer that on an unfixed K-gap and you manufacture a confident fabricator. The ordering rule that falls out: diagnose first → fix knowledge/prior → trajectory-SFT last. The full evidence chain is Does trajectory SFT amplify an unfixed gap?.

How to read this section

Cross-links

• Diagnosing the gap — a scientific framework — the complementary framing (knowledge / execution / exploration gap). The mapping is exact, not a loose overlap: execution = R ∪ P — the same continuous in-support-but-mis-ranked axis this chapter’s taxonomy defines, at two granularities (R = macro/plan-level, P = micro/decision-point); exploration is not a fourth gap but a training-dynamics modifier that attaches to an R/P failure when its winning path is sequentially-gated rather than single-shot. taxonomy.md is canonical for this mapping — read it before reconciling the two vocabularies.
• The kinds of SFT — it is the data, not the algorithm — the SFT data-shape taxonomy the curation chapters build on; its §4 (synthetic-authoring → confabulation) is the twin of trajectory-amplification.
• The one axis that predicts everything — the on/off-policy distinction underneath every “inject vs amplify” decision here.
• Contested edges & landmines — the elicit-vs-expand debate that most of the diagnosis rests on is genuinely unresolved; treat it as contested.

Bibliography (ids verified live in the survey pass, 2026-07-02)

Standing rule (all chapters in this section): no load-bearing claim rests on an academic cybersecurity-LLM training/benchmark paper — every citation is general ML/RL theory or frontier-lab evidence. The cyber running example is the motivation, not the basis.

The three gaps, defined — knowledge · prior · policy

This is the theory chapter The decision and Diagnosing the gap both assume and neither fully derives. Those two chapters give you the routing tree and the instrument battery; this one gives you the reason the tree is even a legitimate tree — why “knowledge gap” and “execution/ranking gap” are not just a convenient vocabulary but two mathematically distinct regimes with a hard boundary between them.

The question this chapter answers: when your agent reaches for subfinder and never amass, or fires nmap with default flags instead of an expert scan profile, is that because the model genuinely doesn’t know the alternative exists, or because it knows and doesn’t act on it — and how do you tell the difference in a way a skeptical reviewer can’t wave away?

Bottom line up front: there are exactly two structurally different failure regimes, not three, and the field’s “K/R/P” vocabulary names one hard boundary plus one soft one. K (knowledge) is a genuine absence — the reference policy places ≈0 probability mass on the correct behavior, in which case no amount of KL-regularized optimization (RLHF, DPO, GRPO, RLVR — all of them, provably) can get you there, because reweighting mathematically cannot manufacture mass where none exists arXiv:2305.18290. R (prior) and P (policy/preference) are not two separate gaps — they are one continuous axis, “in-support but mis-ranked,” with two diagnostic sub-signatures that happen to have gotten separate names. For your running example: the agent’s macro failure — no opening recon methodology, a scattershot curl/dig instead of an expert plan — is the R-flavored signature (a coherent plan exists somewhere in the sampling distribution, just outranked by a shallow default). The agent’s micro failures — reaching for subfinder and never amass, nmap with stock flags instead of an expert profile — are the P-flavored signature, and there is now direct mechanistic evidence for exactly this claim: the correct tool is attended-to 80% of the time and the correct token sits in the top-5 logits, yet default decoding still emits the wrong one arXiv:2606.16364, arXiv:2605.06241. The tool/flag failure is a P-gap. And critically — this pattern is universal across frontier model families, which is itself evidence about which gap you’re looking at, not just a curiosity (§4.3 below).

A note on vocabulary, before you go further. The decision and Diagnosing the gap use a three-way “knowledge / execution / exploration” (or “knowledge / execution / ranking”) split, aimed at routing you to a training method fast. This chapter’s K/R/P split is a refinement of the same territory at higher resolution, not a competing taxonomy: their “knowledge gap” is this chapter’s K; their “execution gap” and “ranking gap” are both instances of this chapter’s single R/P axis, distinguished by which diagnostic sub-signature fires (§2.4); and their “exploration gap” is what you get when an R-flavored failure turns out to be sequentially-gated rather than single-shot (§6 works through exactly this case for F1). Use the practitioner’s three-way split day-to-day; come back to this chapter when you need to defend why a given failure was routed where it was, or when a failure doesn’t cleanly fit either “execution” or “ranking” and you need the finer axis to say why.

1. Genealogy — three literatures, one convergence point

The K/R/P split feels like it should come from one canonical RL paper. It doesn’t. It’s the convergence point of three independent lines of work that were solving three different problems and only recently discovered they were describing the same boundary.

flowchart TD
  L1["Line 1 — linguistics → LLM eval<br/>competence vs performance<br/>(Chomsky 1965, imported by<br/>Mahowald et al. 2301.06627)"]
  L2["Line 2 — control-as-inference<br/>→ RLHF/DPO theory<br/>(Levine 1805.00909,<br/>Ziegler 1909.08593,<br/>DPO 2305.18290)"]
  L3["Line 3 — AI-safety<br/>capability elicitation<br/>(Greenblatt 2405.19550,<br/>van der Weij 2406.07358)"]

  L1 -->|"gives the VOCABULARY:<br/>'does it know' vs 'does it show'"| CONV["2025-2026 RLVR pass@k<br/>crossover debate —<br/>the empirical battleground"]
  L2 -->|"gives the THEOREM:<br/>KL-regularized RL can only<br/>reweight existing support"| CONV
  L3 -->|"gives the PROOF METHOD:<br/>engineered ground-truth organisms,<br/>watch which intervention recovers it"| CONV

  classDef your fill:#132b22,stroke:#34d399,color:#eafaf3;
  class CONV your;

Line 1 — linguistics → LLM eval methodology. Chomsky’s 1965 competence/performance distinction (pre-arXiv) was imported wholesale into LLM evaluation by Mahowald et al., Dissociating language and thought in large language models arXiv:2301.06627 [HIGH, 200+ citations]: formal linguistic competence (grammar/pattern knowledge — LLMs are surprisingly strong here) is dissociable from functional competence (using language to reason/act in the world — LLMs are spotty, and it’s the axis that needs fine-tuning or external scaffolding to close). Hu & Frank operationalize the same move quantitatively as a “task-demand gap” — measured performance = f(true competence, task-execution-demands) — and show a chunk of what looks like a competence deficit in smaller LMs is actually an artifact of the auxiliary demands a task imposes on top of the core skill being measured arXiv:2404.02418 [MED, COLM 2024]. This line hands you the methodology: before any RL-specific vocabulary existed, there was already a rigorous way to ask “does it know” separately from “does it show.”

Line 2 — control-as-inference / RLHF theory: the load-bearing theorem. Levine’s control-as-inference tutorial arXiv:1805.00909 [HIGH] proves that the optimal soft-RL policy is a reference-weighted Boltzmann posterior — reward doesn’t replace the reference dynamics, it reweights them. Ziegler et al. operationalized this exact posterior as the KL-anchored RLHF objective used to fine-tune LMs arXiv:1909.08593 [HIGH]. Rafailov et al.’s DPO made it algebraic and closed-form: π*(y|x) ∝ π_ref(y|x)·exp(r(y)/β), and the implicit reward is literally β·log[π(y|x)/π_ref(y|x)] arXiv:2305.18290 [HIGH]. This is the theorem the whole taxonomy rests on — §3 below derives it in full and states exactly why it draws a hard line between K and everything else.

Line 3 — AI-safety capability elicitation: the empirical proof method. Greenblatt et al.’s password-locked models arXiv:2405.19550 [HIGH, NeurIPS 2024] trained a model to imitate a weaker model unless a password token is present, then showed the hidden capability is recoverable from a handful of SFT demonstrations, sometimes RL alone. van der Weij et al. named the adversarial framing directly: sandbagging — frontier LMs can be prompted or fine-tuned to strategically underperform, and it generalizes to held-out benchmarks arXiv:2406.07358 [HIGH]. This line gives you the proof method: build (or find) a case where you know the capability is present — you put it there — and watch which class of intervention recovers it. That’s exactly the diagnostic posture this book needs, applied to a model whose true capability state you don’t already know.

Hofstätter et al.’s Elicitation Game sharpens Line 3 in a way that turns out to matter directly for the P definition below: prompting alone fully elicits hidden capability on MCQA/recognition-shaped tasks (both password-locked and circuit-broken model organisms), but fails on open-ended/code-generation tasks — there, only fine-tuning-based elicitation works arXiv:2502.02180 [HIGH, ICML 2025]. That is a format-dependent reliability finding about elicitation technique, but read it the other way and it’s a direct empirical demonstration that a recognition channel and a generation channel really can hold different amounts of the same capability — which is precisely the axis §2’s P-gap definition formalizes. Tool-call synthesis is generation-shaped, not MCQA-shaped, which is the reason a negative prompting-only probe on a tool-use failure is weak evidence of a true ceiling — you need the fine-tuning-based (or at minimum the forced-choice/logprob) probe before trusting a negative result.

These three lines meet at the 2025-2026 RLVR pass@k-crossover debate — “does post-training reweight-only, or does it sometimes manufacture new support?” — which is why §6 below spends real time on it: it’s the empirical battlefield where the theorem from Line 2 gets stress-tested against real training runs.

2. The definitions, grounded in the math

All three definitions are stated in terms of the reference policy π_ref — the checkpoint you’re about to train from (base model, or the SFT/cold-start checkpoint, depending on where in the pipeline you’re diagnosing). This matters: a gap diagnosed against the base model can be a different gap once diagnosed against your SFT checkpoint, because SFT itself moves π_ref.

K — the genuine absence

Allen-Zhu & Li’s Physics-of-Language-Models series gives the sharpest formal grounding available for what “genuinely absent” means, and it comes with a trap built in. A fact can be perfectly stored in the weights (probeable via linear probing or gradient-based extraction) yet 0% extractable via QA unless the pretraining corpus contained diverse paraphrases of it — storage and extractability are different properties arXiv:2309.14316 [HIGH, seminal]. Storage itself is capacity-bounded at roughly 2 bits/parameter, an information-theoretic ceiling that further tightens with data quality arXiv:2404.05405 [HIGH, ICLR 2025 Spotlight] — this is a hard limit on how much niche knowledge fine-tuning can inject into a small model, full stop, independent of training recipe. Kandpal et al. tie extractability directly to pretraining frequency: QA accuracy on a fact correlates causally with how many pretraining documents mentioned the relevant entities, and retrieval-augmentation closes the gap far more cheaply than parameter scaling arXiv:2211.08411 [HIGH, seminal].

The trap: even perfectly stored and extractable knowledge fails at manipulation — classification, comparison, inverse-search — and inverse search specifically sits near 0% regardless of prompting unless chain-of-thought is used at both train and inference time arXiv:2309.14402 [HIGH, seminal]. This looks exactly like a K-gap from the outside (the model “can’t do it,” 0% no matter how you ask) but is structurally closer to an R-gap — the raw fact has non-trivial log π_ref, it’s the composition that’s mis-ranked or missing a reasoning scaffold. Don’t classify a failed inverse-direction query as K without checking the forward direction first.

R — in-support, mis-ranked

The support-constraint theorem (derived in full in §3) is what makes R a fixable-by-reweighting category rather than a euphemism for K. DPO’s closed form is the seminal statement arXiv:2305.18290 [HIGH]; Wu et al. formalize the identical constraint specifically for RLVR — verifiable-reward RL cannot sample a completion with zero initial probability under the reference policy, full stop, algebraically arXiv:2507.14843 [MED]; Ni et al. add the GRPO-specific mechanism, proving the group-normalized-advantage update is a provably conservative reweighting operation — bounded in how far it can shift mass per step, not just directionally reweight-only arXiv:2510.15990 [MED]. Together these three are why “reweighting within existing support is what KL-regularized RL/preference optimization is mathematically built to do” is not a hand-wave — it’s a proven property of the objective class.

P — the sharper, channel-divergent form

Kadavath et al.’s Anthropic result is the seminal proof that a “judgment” channel can diverge from a “generation” channel: models trained to output P(True)/P(IK) (probability they know the answer) before attempting a task are well-calibrated on MCQ/self-eval even when their own default open-ended generation gets it wrong arXiv:2207.05221 [HIGH, seminal]. Cao et al.’s 2026 follow-up makes it current and concrete: models carry a usable pre-generation confidence signal that predicts eventual success, but default decoding simply doesn’t consult it arXiv:2605.14186 [LOW — very recent, promising, not yet validated]. Hofstätter et al.’s MCQA-vs-open-ended elicitation split, above, is the format-level version of the same claim arXiv:2502.02180 [HIGH]. This is the P signature: not “is y* in the sampling distribution somewhere” (R already answers yes) but “does a different elicitation channel recover it when default generation doesn’t.”

A worked, illustrative walkthrough — not measured data

To make the three rows of the table concrete before the theorem derivation, here’s how the same decision — “which recon tool to invoke first” — would read out under each regime, as a walkthrough of what the numbers would look like, not as data from any cited paper:

K-gap read (illustrative):
  log pi_ref("amass enum -active -d target.com" | context)  ~  numerical floor
  MCQ forced-choice(subfinder vs amass)                       ~  chance (no separation)
  -> amass's *expert invocation syntax* is essentially absent from the distribution

R-gap read (illustrative) — this is F1, the missing recon PLAN:
  log pi_ref(scattered curl/dig opening | context)   >   log pi_ref(structured PTES-style opening | context)
  but sampling k=64 surfaces a coherent methodology in a non-trivial fraction of rollouts
  -> the plan is in-support, just outranked at k=1

P-gap read (illustrative) — this is F2/F3, subfinder-not-amass and default-not-expert-flags:
  log pi_ref("amass" | context)  is within the top-5 tokens at the decision point
  MCQ forced-choice(subfinder vs amass)  strongly prefers amass
  default open-ended generation  still emits subfinder
  -> the readout step, not the distribution, is where the failure lives

The point of laying it out this way is that K, R, and P aren’t three qualitatively different kinds of brokenness — they’re three different readouts of the same underlying quantity, log π_ref(y*|x) and its rank, at increasing resolution: does it exist at all, is it ranked below a default, and does a different channel expose a preference the default channel hides. §3 derives why that quantity is the one that matters.

Two ways to misclassify a gap, and why they’re not symmetric

The definitions above are precise, but two specific failure modes account for most of the misclassifications in practice, and they pull in opposite directions:

Both traps share a structure: a cheap, low-k or single-channel probe returns a false “K” verdict because the probe’s resolution wasn’t fine enough to find mass that’s genuinely there but deeply buried or structurally locked behind a missing reasoning step. This is the concrete argument for why the ladder in Diagnosis escalates through multiple probes before concluding K rather than accepting the first negative result — a single null probe is never sufficient on its own to place a failure in the off-policy-injection branch of The decision.

R and P are one continuous axis, not two boxes

The field has not converged on where R stops and P starts. Treat them as one continuous “in-support-but-mis-ranked” axis with two diagnostic sub-signatures, and use both running-example failure modes as the worked illustration, because they sit at genuinely different points on that axis:

• F1 — no opening recon methodology (macro, R-flavored). The agent fires a scattered curl http://target/ / dig target instead of a structured PTES-style recon sequence. This is a plan-level failure — there’s no single wrong token, there’s a wrong overall shape. The diagnostic signature is distributional plateau: sample the base/reference policy at k=16, 64, 256 on the same opening move, and a coherent methodology shows up somewhere in the tail, just consistently outranked by the shallow default at k=1. That’s R in its purest form — the correct plan is in-support, it’s mis-ranked against a more probable generic-troubleshooting default.
• F2/F3 — reaches for subfinder never amass, default not expert flags (micro, P-flavored). This is a single-decision-point failure — one token (or a short span) at one specific place in the trajectory. The diagnostic signature is explicit recognition/generation divergence: force a choice between subfinder and amass via MCQ or forced-logprob at that exact decision point, and the model picks correctly at a rate wildly higher than its own default-generation rate. §4 below gives the direct mechanistic evidence for this exact claim.

Both are “in-support but mis-ranked.” What differs is the granularity of the readout — a whole-plan ranking problem (R) versus a single-token decision-readout problem (P) — and that granularity difference is precisely what determines which cheap probe catches it first (pass@k sweep for F1-shaped failures; forced-choice/logprob-rank check for F2/F3-shaped failures). The full probe battery that operationalizes this — in order, cheapest first — lives in Diagnosis; this chapter’s job is only to establish that the axis is real and continuous, not to hand you the runbook.

flowchart LR
  subgraph SUPPORT["Where does π_ref(y*|x) sit?"]
    Y0["≈0 across every rephrasing,<br/>every sampling budget"]
    Y1["finite, non-trivial,<br/>ranked below a shallow default"]
    Y2["finite, and RECOGNIZED under a<br/>different elicitation channel<br/>(MCQ / oracle / forced-logprob)"]
  end

  Y0 --> K["K — KNOWLEDGE GAP<br/>nothing for KL-regularized<br/>RL/preference to reweight onto"]
  Y1 --> RP["R/P — ONE continuous axis:<br/>'in-support but mis-ranked'"]
  Y2 --> RP

  RP -->|"sub-signature 1: distributional<br/>plateau (pass@k rises w/ k)<br/>— your F1: no opening plan"| R["lean R —<br/>plan-level ranking fix"]
  RP -->|"sub-signature 2: explicit<br/>recognition/generation divergence<br/>— your F2/F3: subfinder-not-amass"| P["lean P —<br/>token-level readout fix"]

  classDef your fill:#132b22,stroke:#34d399,color:#eafaf3;
  class R,P your;

3. The load-bearing theorem — why K vs R/P is a real, checkable line

This is the derivation, not just the citation. It’s what makes the taxonomy falsifiable rather than a vibe.

Step 1 — control-as-inference. Frame RL as probabilistic inference over an “optimality” variable: the optimal policy under a KL-regularized objective is a Boltzmann-weighted posterior over the reference dynamics — reward doesn’t replace π_ref, it tilts it arXiv:1805.00909 [HIGH].

Step 2 — operationalized as the RLHF objective. maximize E_y~π[r(y)] − β·KL(π ‖ π_ref) has the exact closed-form solution π*(y|x) ∝ π_ref(y|x)·exp(r(y)/β) arXiv:1909.08593 [HIGH]. Every KL-anchored post-training objective in current use — PPO/RLHF, GRPO/RLVR, DPO and its whole family — is solving for a version of this exact posterior, whether or not it ever instantiates a reward model.

Step 3 — DPO makes it algebraic. Invert the closed form: r(y) = β·log[π*(y|x)/π_ref(y|x)] + β·log Z(x). The implicit reward is literally the log-ratio between the trained and reference policy. Fitting a Bradley-Terry loss directly on this log-ratio — the partition function Z(x) cancels between a chosen/rejected pair sharing the same prompt — gives the DPO loss, no separate reward model, no RL loop, but mathematically the identical constrained optimum arXiv:2305.18290 [HIGH].

# The Boltzmann posterior every KL-regularized post-training objective converges to:
# pi_star(y|x) is proportional to pi_ref(y|x) * exp(r(y) / beta)

# DPO inverts this for the implicit reward — no reward model needed:
# r(y) = beta * log(pi_star(y|x) / pi_ref(y|x)) + beta * log(Z(x))

# The load-bearing consequence — this is the whole chapter in five lines:
if pi_ref(y_star, x) == 0:               # or underflow-indistinguishable from the float floor
    pi_star(y_star, x) == 0              # for ANY finite r(y_star) — reward cannot rescue zero mass
    # KL(pi_star || pi_ref) -> +inf the instant pi_star assigns positive mass where
    # pi_ref assigns exactly zero. The objective doesn't merely fail to reach y_star —
    # it is mathematically FORBIDDEN from doing so while remaining KL-finite.

This is the reason K vs R/P is a real, checkable category and not a matter of taste. If π_ref(y*|x) ≈ 0, there is nothing to push on. No amount of preference optimization, no amount of verifiable-reward RL, no clever reward shaping changes this — it’s a structural property of the objective class, not an optimization difficulty you could someday out-compute.

RLVR-specific formalization, not just DPO’s. Wu et al.’s Invisible Leash proves the identical support constraint holds for RLVR (GRPO-style binary-correctness-reward RL): it cannot sample a completion with zero initial probability under the reference policy arXiv:2507.14843 [MED]. Ni et al. sharpen this specifically for GRPO’s group-normalized advantage estimator: the update is provably a conservative reweighting operation, bounded per-step, not just directionally constrained arXiv:2510.15990 [MED]. So the theorem isn’t specific to DPO’s closed form — it’s a property of the whole KL-regularized family, RLVR included, which is exactly the family your GRPO stage lives in (Reinforcement — PPO · GRPO · RLVR).

What to actually run before you spend RL compute. Teacher-force the target completion y* through the reference checkpoint and read log π_ref(y*|x) and its rank among alternatives at the decision-token position — this is a single forward pass, no sampling, no training run. Near the numerical floor across ≥10 rephrasings of the prompt → you are in K-gap territory and no RL budget will fix it; go curate data. Finite and non-trivial → you’re in R/P territory, and every method in the KL-regularized family is mathematically available to you. This is the cheapest possible instance of the probe battery in Diagnosis — run it before anything else.

From the closed form to a training loss — DPO makes the theorem operational

It’s worth seeing the last algebraic step explicitly, because it’s the step that turns “reward is a reweighting of π_ref” from a fact about the optimum into a loss you can actually run. Take the inverted closed form from Step 3 and plug it into a Bradley-Terry preference model over a (chosen, rejected) pair sharing one prompt:

# Implicit reward per DPO's inversion of the Boltzmann posterior:
r(y | x) = beta * log(pi_theta(y | x) / pi_ref(y | x))        # + beta*log Z(x), which cancels below

# Bradley-Terry probability that y_chosen is preferred to y_rejected:
P(chosen > rejected) = sigmoid(r(chosen) - r(rejected))
                      = sigmoid(beta * [ log(pi_theta(chosen)/pi_ref(chosen))
                                        - log(pi_theta(rejected)/pi_ref(rejected)) ])

# DPO loss — no reward model, no RL loop, Z(x) cancelled algebraically:
loss = -log_sigmoid(beta * (lr_chosen - lr_rejected))
# where lr_y = log pi_theta(y|x) - log pi_ref(y|x)   for y in {chosen, rejected}

Notice what this loss can and cannot do to pi_theta(chosen). It can raise lr_chosen relative to lr_rejected — that’s a ratio against pi_ref, and the ratio is only informative where pi_ref(chosen) > 0 to begin with. If pi_ref(chosen) ≈ 0 (the K-gap condition), log(pi_theta(chosen)/pi_ref(chosen)) is either undefined or an enormous number driven entirely by the denominator underflowing — the loss has no stable signal to climb, and in practice this is exactly the regime where DPO training on a K-gap pair produces degenerate, off-distribution completions rather than the intended fix (a known, separate failure mode called likelihood displacement — pushing chosen up can drag unrelated high-probability completions down when the pair isn’t genuinely in-support; see Interventions per gap for the sanity gate this motivates). The loss is doing exactly what §3’s theorem predicts: real, useful gradient where pi_ref already has mass; nothing coherent to climb where it doesn’t.

The same frame applied to GRPO — where the “1-3% of tokens” claim comes from

GRPO drops the separate reward model and the pairwise comparison, but it’s solving the identical constrained problem with samples instead of an algebraic inversion:

# GRPO — sample N completions per prompt from the CURRENT policy (on-policy, by construction):
resps = [pi_theta.generate(prompt) for _ in range(N)]
rewards = [verifier(r) for r in resps]                    # ground-truth, e.g. flag correctness
advantage = [r - mean(rewards) for r in rewards]           # the group mean IS the baseline — no critic
# PPO-style clipped update pushes log-probability of high-advantage tokens up,
# low-advantage tokens down — but the update is bounded by the PPO-clip range
# AND by how much mass pi_ref (this step's rollout distribution) already assigned nearby.

Because every rollout in the batch is sampled from π_θ itself, GRPO can only ever reweight tokens it already sampled with non-trivial probability — which is exactly Ni et al.’s formal result that the group-normalized advantage update is a provably conservative reweighting operation, bounded per step arXiv:2510.15990 [MED]. It’s also the direct mechanism behind Akgül et al.’s empirical “1-3% of token positions, always top-5” finding arXiv:2605.06241 [LOW, promising] — the update can only touch positions where a sample actually landed with meaningful probability, and by construction that’s a small, high-entropy subset of all positions, drawn from a token’s existing short-list of likely continuations. §4 develops this into the mechanistic case for the running example.

4. Mechanistic evidence for the running example — the tool/flag failure is a P-gap

Two 2026 papers give near-direct mechanistic confirmation for exactly F2/F3’s shape: “agent reaches for one salient tool, uses default not expert args.”

4.1 Sparse, top-5-bounded correction

Akgül et al. find that only 1–3% of token positions change under RL, concentrated at high-entropy decision points — and, critically, the token RL promotes is always already in the base model’s top-5 logit alternatives arXiv:2605.06241 [LOW — very recent, low citation count, promising, not yet validated]. This gives you a literal, checkable number: if the correct tool/arg token sits in the base model’s top-5 at the decision point, you are in a checkable P-gap regime, and you should expect any fix (RL, DPO, or a lighter steering intervention) to touch a tiny fraction of the model’s behavior — this is not a “relearn the tool surface” problem, it’s a “nudge one decision” problem.

4.2 Attends-correctly-but-picks-wrong

Chen’s Looking Is Not Picking is more direct still, and the methodology is worth walking through because it’s what makes the claim mechanistic rather than just behavioral. On real BFCL tool-call failures — cases where the model, given a schema of candidate tools and a task, calls the wrong one — the paper decomposes the forward pass into two separable stages: (1) an attention/localization stage, measured as per-candidate attention mass over the tool-schema tokens, and (2) a decision/readout stage, the final projection that turns the model’s internal state into the emitted tool-call token. Per-candidate attention shows the model attends to the correct tool 80% of the time (vs. 21% chance under a uniform baseline) — it is under-attended in only ~10% of failures — yet the model still calls the wrong one on the majority of these cases arXiv:2606.16364 [LOW — very recent, single-author preprint, promising, not yet validated]. That decomposition is the whole argument: if the failure were localization (the model never “looked at” the right tool), that would look a lot like a K/R-flavored breadth problem — the correct option simply isn’t salient enough to reach. Instead the correct option is salient — attended to at 4× chance rate — and the failure is downstream of that, in the mapping from “attended-to” to “emitted.”

The intervention data confirms the decomposition rather than just illustrating it. Prompt-side fixes — reordering the tool schema so the gold tool appears earlier, or duplicating it to increase its salience — target the localization stage and recover ≤23% of failures, consistent with localization already being mostly fine (there’s not much headroom left to buy there). Interventions that instead target the decision-readout layer — the late projection/classification step, intervened on directly rather than through the prompt — recover 59–91% of failures.

That 23%-vs-59–91% spread is about as direct a proof as exists in the current literature that “picks the salient-not-expert tool” is a readout/decision failure (P), not a visibility/knowledge failure (K). It proves it by showing where the fix actually lands.

Put together: F2/F3 is not “the model doesn’t know amass exists” — it’s attending to the right region of its own knowledge and still emitting the wrong token at the readout step. That is the textbook P-gap signature, and it means the fix lever is a decision-level nudge (on-policy DPO on tool-choice pairs, or a light GRPO pass rewarding the expert choice), not a data-injection campaign to teach the model what amass is.

4.3 Cross-model universality as evidence, not just an observation

The running example states this failure is universal across frontier models — every family in a typical benchmark roster reaches for the same salient tool and the same default flags. That universality is itself diagnostic, and it’s worth stating the logic explicitly rather than treating it as a curiosity.

If this were a genuine K-gap, you’d expect it to vary by pretraining corpus and architecture — different labs curate different data mixes, so an idiosyncratic capacity/storage limitation should show up idiosyncratically. Instead the failure is shared across independently-trained frontier families. The more parsimonious explanation, consistent with Kandpal et al.’s frequency-correlation finding that extraction accuracy tracks how often entities co-occur in pretraining text arXiv:2211.08411 [HIGH], is a shared prior baked into the overlapping web-scale pretraining distribution every frontier lab trains on: subfinder-style one-shot recon tools and default-flag invocations are simply far more represented in blog posts, Stack-Overflow-style answers, and tool READMEs than amass’s fuller feature set or an expert scan profile, and every lab’s corpus draws from broadly the same internet. This is a reasoned extension of the theorem plus the frequency-correlation evidence, not a separate paper’s direct finding — flag it as such — but it is a genuinely useful, falsifiable heuristic: a failure that’s universal across independently-trained frontier families is evidence for a shared distributional prior (R/P), not evidence for a per-model capacity ceiling (K), and it’s a cheap first read to run before committing to the full probe battery.

What this changes about your intervention. If the diagnosis holds, the fix is not “teach the model what amass is” (it already knows — every frontier family does) — it’s “change which of the two already-known options the readout step selects, at the specific decision point where they compete.” That’s a data-curation problem for on-policy preference pairs (chosen = amass invocation, rejected = subfinder-only, sampled from the model’s own rollouts at that decision point), not a knowledge-injection problem. The concrete recipe for building that pair set lives in Interventions per gap; this chapter’s job is only to establish why that’s the right category of fix.

5. The generation–verification gap — an orthogonal axis

K/R/P all answer one question: can the model produce the right answer? There’s a second, genuinely orthogonal question that matters independently for any pipeline that leans on the model’s own outputs to bootstrap further training (rejection sampling, STaR-style loops, RLAIF): can the model tell right from wrong once something has been produced?

Song et al. formalize this as the generation-verification gap: pass@N coverage minus self-verification accuracy, on the same problem set arXiv:2412.02674 [HIGH, ICLR 2025 Oral]. This quantity scales monotonically with pretraining compute and bounds how much any self-improvement loop can gain — a model can have excellent K/R/P properties (the right answer is reliably produced somewhere in its sample distribution) and still have a wide generation-verification gap (it can’t reliably pick the right one out of its own samples), and the two properties can diverge in either direction.

This matters for your running example at a specific, practical seam: the flag verifier is external, ground-truth, and only fires at submission time. Mid-episode — at turn 40 of 100 — the agent has to judge without the verifier whether its current path is worth continuing. That’s a self-verification judgment, not a generation judgment, and it’s governed by this orthogonal axis, not by K/R/P. A model can have zero K-gap and zero P-gap on “which tool to use next” and still make a bad mid-episode continue/abandon call because its self-verification is weak relative to its generation.

Contested, and worth stating as contested rather than settled. The intuition that “verification is easier than generation” is rooted in scalable-oversight/debate theory — the founding argument that a weaker verifier can still adjudicate a stronger generator’s claims if the claims are checkable arXiv:1805.00899 [HIGH, seminal]. It is not universal. West et al.’s Generative AI Paradox documents cases where models are worse verifiers than generators on the same items, driven by an acceptance bias toward the model’s own (or plausible-looking) output rather than genuine discrimination arXiv:2311.00059 [MED]. Treat the direction of the generation-verification gap as task-dependent, not a law:

Don’t assume a model that solves your challenges reliably is equally reliable at telling you, mid-trajectory, whether it’s on a dead path — that’s a different channel than the one measured by solve rate, and per the table above its reliability isn’t guaranteed by the solve rate being high. The self-verification-cliff mechanism this connects to on the agentic-execution side is developed fully in Diagnosing the gap §5.1; this section’s job is only to place it correctly as orthogonal to K/R/P, not to re-derive the mid-episode diagnostic.

6. The contested debate — does RL ever expand the boundary, or only reweight?

Everything in §3 says KL-regularized optimization can only reweight existing support. The empirical debate over how narrow that reweighting really is in practice — whether it’s ever indistinguishable from genuine capability expansion — is the single most contested thread in this literature, and it matters directly for whether F1 (the recon-methodology gap, R-flavored) is cheaply fixable by rejection-sampling SFT + light RL, or needs a heavier on-policy exploration recipe.

• Yue et al. — the founding result arXiv:2504.13837 [HIGH, NeurIPS 2025 oral]. At large sampling budgets (pass@256+), un-RL’d base models match or beat RLVR-trained checkpoints on static math/code — RLVR raises pass@1 by resampling paths already in the base model’s support, it does not expand the reasoning boundary. This is the empirical anchor for the theorem in §3.
• ProRL — the direct counter, for one specific recipe arXiv:2505.24864 [MED, NVIDIA]. Prolonged RL with explicit KL-divergence control, periodic reference-policy resets, and diverse tasks uncovers reasoning strategies inaccessible to the base model under any sampling budget tried. This doesn’t contradict §3’s theorem — it’s evidence that “reference policy” is not static across a long, reset-punctuated training run, so the support boundary itself can shift over the course of prolonged training even though no single KL-regularized step can violate it.
• CoT-Pass@K — a metric-level rebuttal arXiv:2506.14245 (cited in the source survey narrative; not independently re-confirmed against arxiv.org in this ledger pass — treat as a live but unconfirmed thread). Argues naive pass@k credits a correct final answer even from a broken reasoning chain (a lucky guess), and once you require the reasoning trace itself to be correct, the crossover disappears — RLVR shows monotonic gains at every k. If this holds, it directly falsifies the load-bearing assumption behind Yue et al.’s headline reading.
• Two-stage reconciliation arXiv:2510.04028 (same unconfirmed-in-ledger caveat). Proposes the two camps are sampling two different phases of one dynamic: an early “exploitation” phase looks like pure reshuffling (the Yue-camp signature), while a later “exploration” phase — reached only if training survives entropy collapse long enough, which needs ProRL-style KL-control/reset machinery — can genuinely promote rarely-sampled optimal tokens into the accessible distribution.
• Agentic Pass@(k,T) — the caveat most relevant to your own harness arXiv:2604.14877 [MED]. The static single-turn “RL only elicits” result may not transfer to tool-use agents: T rounds of environment interaction can reveal compositional strategies that flat resampling (more k, same T) cannot recover. On their Category C (compositional, sequentially-gated retrieval — structurally identical to “enumeration must land at turn 5 before the exploit is even visible at turn 40”), the RL pass-curve pulls above and widens against the base curve as k grows — the opposite of the static-reasoning crossover — while matched-data SFT on the same subset actually regresses it (net −4 vs. RL’s net +4). This isolates self-directed exploration during on-policy rollout, not data exposure, as the causal ingredient for expansion on compositional tasks.
• Recall specifically arXiv:2605.07153 [MED]. In a deduplicated, zero-shot, closed-book factual-recall setting, RL on a binary correctness reward does yield ~27% relative recall gains — by moving correct tokens from an astronomically low-probability tail into a reliable greedy-decode slot. This complicates “no pass@k gain ⇒ K-gap” specifically for pure recall (as distinct from multi-step reasoning): a small k can misclassify a rare-but-present fact as a true K-gap when it’s actually a very-low-rank R-gap that a longer RL run would fix.

A few adjacent threads round out the picture without being individually load-bearing here: a compositional-generalization result claims RL can teach f∘g composition never itself sampled from the base model, but a rebuttal shows the same signature can arise from mere length generalization rather than true composition, so treat that specific pairing as unresolved; a boundary-aware curriculum-RL line is a further active attempt to engineer past the static-reasoning crossover. Neither is independently confirmed in ledger-A.md for this pass, and neither changes the net read below — they’re additional evidence that the debate is active, not additional weight on either side.

The engineer’s net read. “RL only reweights, never teaches” is real, but narrower than usually quoted. It’s most solid for static, single-turn math/code reasoning under short/vanilla RLVR — exactly the regime Yue et al. tested. It gets progressively weaker for (a) pure factual recall, (b) multi-turn agentic/tool-use settings, and (c) prolonged RL runs engineered with KL control, reference resets, and curriculum. Don’t conclude K-gap from a k=16 pass@k null result on an agentic task — push k as high as budget allows, and if the task is agentic, run Pass@(k,T), not flat Pass@(k).

Why this matters for F1 specifically, and why it matters less for F2/F3. The mechanistic P-gap evidence in §4 (top-5-bounded token correction, 80%-attended-but-mis-picked) is largely agnostic to this debate — a mis-ranked-but-already-attended token doesn’t require capability expansion to fix, only reweighting, so F2/F3 are cheap, well-targeted fixes regardless of which side of the elicit-vs-expand debate turns out to be right. F1 (the recon-methodology gap) is exactly where the debate’s resolution is operationally load-bearing: if the agent’s recon planning is a static, single-turn ranking problem, treat it as the Yue-camp regime (rejection-sampling SFT should work fine). If it’s genuinely sequentially-gated — the recon sequence has to unfold correctly across several tool calls before the right next move is even visible — you’re in Zhai et al.’s Category-C regime, and matched-data SFT alone is predicted to regress it; you need on-policy RL with real exploration, not more demonstrations. This is exactly the segmentation test The decision and Diagnosing the gap §2.4 already build a runbook around — this chapter supplies the theoretical reason that segmentation is the right first move, not an arbitrary methodological preference.

7. Synthesis — the running example, fully classified

Pulling §2 through §6 together against the three named failures, one table:

The universality argument from §4.3 already makes the K-row unlikely for F1–F3 specifically — a genuinely per-model capacity ceiling wouldn’t reproduce identically across independently pretrained frontier families. That’s a strong prior, not a substitute for running the actual probes; the theorem in §3 is exactly what makes “run the probe, trust the number” a legitimate move instead of an argument from authority.

The one-sentence version of this whole chapter. K asks “does log π_ref(y*|x) exist at all”; R asks “is it ranked below a shallow default”; P asks “does a different elicitation channel expose a preference the default channel hides” — and the last two are one axis, not two, so stop trying to draw a hard line between them and instead ask which sub-signature (plateau vs. divergence) your specific failure shows, because that determines which cheap probe catches it first.

Cross-links

• The decision — the routing tree this chapter’s K/R/P definitions and theorem sit underneath; read that first for the practical branch, this chapter for why the branch is legitimate.
• Diagnosing the gap — a scientific framework — the complementary knowledge/execution/exploration framing and the full pass@k → Cover@τ → Pass@(k,T) instrument battery; §2.4 there is the direct operationalization of §6’s contested debate, and §5.1 develops the self-verification-cliff mechanism this chapter’s §5 only places on the map.
• Diagnosis — the runnable probe battery (forced-logprob check, pass@k sweep, in-context oracle injection, token-rank check, spurious-reward control) that turns this chapter’s definitions into an executable decision procedure.
• Knowledge-gap data curation — the data-row recipe for a confirmed K-gap, and why jumping straight to trajectory-level SFT on an unconfirmed one causally amplifies it rather than fixing it.
• Interventions per gap — the matched training-row recipes once a gap is confirmed: paraphrase-rich SFT for K, on-policy DPO/GRPO pairs for R/P.
• Trajectory-SFT amplification — the fuller causal argument for why naive trajectory-level SFT on an unfixed K-gap makes hallucination worse, not better.
• Preference — RLHF · DPO · KTO — the mechanism behind the R/P-gap fix lever this chapter derives the legitimacy of but doesn’t itself teach.
• The one axis that predicts everything — the on/off-policy genealogy this chapter’s K-vs-R/P line is a refinement of: K needs off-policy injection because there’s nothing on-policy to reinforce; R/P are on-policy-fixable by construction.

Bibliography

All ids below are drawn from artifacts/three-gap-survey/section-A.md and ledger-A.md. Confidence follows the ledger where the id was independently confirmed against arxiv.org; two ids used in §6’s contested debate were not in the ledger’s confirmed table for this pass and are flagged accordingly — cited because the chapter’s brief specifically required presenting that exact debate, not presented as independently re-verified.

Confidence calibration for this chapter: the theorem (§3) rests entirely on HIGH-confidence, well-established citations (Levine, Ziegler, DPO) plus two MED-confidence formal extensions to RLVR/GRPO specifically — treat the general KL-regularization argument as settled, the RLVR-specific and GRPO-specific formalizations as solid-but-newer. The mechanistic P-gap evidence for the running example (§4.1–4.2) is explicitly LOW / promising, not yet validated — both papers are very recent, low-citation preprints; they are the best mechanistic evidence currently available for “tool-selection failure is a readout problem,” not a settled consensus. §6’s contested debate is presented as contested on purpose — two of its six citations were not independently re-confirmed in this ledger pass and are flagged inline; do not treat the “net read” as more settled than the six bullet points underneath it.

Diagnosing which gap — the runnable probe battery

The question this chapter answers: given one failing behavior — the agent never runs amass, always runs subfinder with default flags, and never front-loads a recon methodology before it starts poking things — what is the cheapest sequence of runnable probes that sorts this into a Knowledge (K), Prior/Policy (R/P), or Exploration-collapse gap, with numbers, not vibes?

BLUF. Run nine probes, cheapest-first, each costing at most what the previous one bought you: (0) a free grep of your existing corpus, (1) one forced-logprob forward pass, (2) a pass@k sampling sweep, (3) in-context oracle injection — the single sharpest K-vs-P splitter in the whole battery — (4) a generation-vs-recognition split, (5) a token-level top-5 rank check, (6) a spurious-reward control gate before you trust any RLVR result, (7) Pass@(k,T) segmented by compositional structure, (8) a minimal-parameter/few-shot-SFT elicitation probe, and (9) the escalating elicitation ladder that synthesizes 0–8 into one verdict. No individual probe is trusted alone — every family below ships with a documented confound, and the real confidence signal is agreement across three or more independent probe families, not a single clean plot. Close with a purpose-built bottleneck CTF that turns the whole battery into one number you can read off a solve-rate band.

Every arXiv id below is CONFIRMED in artifacts/three-gap-survey/ledger-A.md or ledger-B.md (project research pass, verified live against arxiv.org/abs/<id>). Confidence tags: [HIGH] peer-reviewed/heavily reproduced, [MED] coherent preprint not yet contested, [LOW] single small-N or very-recent preprint — “promising, not yet validated.” Ids flagged UNCERTAIN or absent from either ledger are not used here, per the standing project rule against inventing or citing unverified ids.

0. This chapter vs. Diagnosing the gap — a scientific framework: same territory, two vocabularies

The framework chapter sorts a failure into Knowledge / Execution / Exploration, grounded in the linguistics competence/performance split (Firestone PMC7604508 [HIGH]; Mahowald et al., arXiv:2301.06627 [HIGH]) and built around one core instrument (pass@k → Cover@τ → Pass@(k,T)). taxonomy.md sorts the same failure into the finer-grained K / R / P — Knowledge, Prior, Policy — and its own §0 is the canonical statement of how the two vocabularies map onto each other: “their ‘knowledge gap’ is this chapter’s K; their ‘execution gap’ and ‘ranking gap’ are both instances of this chapter’s single R/P axis… and their ‘exploration gap’ is what you get when an R-flavored failure turns out to be sequentially-gated rather than single-shot.” This chapter doesn’t re-derive that mapping — it takes it as given and adds the missing piece: a runnable test (Probe 7, §9) for the one part of the mapping that isn’t a static lookup — whether a given R/P-flavored failure is also sequentially-gated, and therefore an Exploration-collapse case rather than a plain R/P-gap.

The correspondence — restated compactly here, canonical derivation in taxonomy.md §0:

Practically: this chapter is the how — the runnable protocol with concrete numbers, thresholds, and pseudocode, operationalizing taxonomy.md §0’s mapping rather than re-deriving it. Framework.md is the why — the theory of why pass@k, Cover@τ, and Pass@(k,T) are valid instruments at all, including the contested edges (§2.2’s crossover vs. its CoT-Pass@K rebuttal). Read taxonomy.md §0 for the mapping and framework.md for the instrument derivations; this chapter assumes both and tells you what to actually type into a terminal.

1. The running example: three decision points, one battery

Three co-occurring symptoms, universal across the project’s benchmark roster (5 model families — xai, gemini, deepseek, glm, qwen, none privileged, per lessons/evals/pd-bench-benchmark-family-roster.md):

• F1 — no opening methodology. Turn 1 dives straight into a scan/exploit attempt instead of a structured recon-first plan.
• F2 — subfinder, not amass. Subdomain enumeration reaches for the tool with the narrower default passive-source list.
• F3 — default, not expert, flags. Whatever tool gets called, it’s invoked with its bare defaults, not the flag combination (-passive -config <file> for amass; --min-rate 5000 -p- -sV -sC for nmap) that a human operator would reach for.

The reason to run the same battery on all three: each is a decision point — a place in the trajectory where the model chooses among alternatives — just at different granularity. F2 and F3 are single-token/single-argument decisions (clean, cheap to probe up to k=256). F1 is a structural decision about the shape of the opening move, closer to framework.md §6’s planning/state-management axis — grounded there in the METR long-horizon study and DAgger’s compounding-error theory, not re-cited here — probeable with the same instruments, but its verdict is less often clean, and a confirmed non-K reading on F1 should route to framework.md §6.1’s per-turn fault labeling for the deeper cut, not stop here.

Run this first, for free: before any of the probes below, confirm the symptom is real and not a sampling artifact — pull 10-20 raw transcripts and read them. A “gap” that turns out to be one bad temperature setting or a truncated context window is not a research question.

2. Probe 0 — free triage before you spend anything

Two zero-cost checks that route you into the ladder; neither is a verdict.

Corpus grep. Your run-trace corpus already exists — events.jsonl per run, preamble carrying meta/tool_schemas/system_prompt/user_message, per this project’s harness observability contract (lessons/security-agent/harness-observability-contract-2026-06.md, always-loaded handbook rule #17). Grep it before generating a single new token:

grep -l '"amass"' runs/*/events.jsonl | wc -l     # e.g. 2 / 50 existing rollouts
grep -l '"subfinder"' runs/*/events.jsonl | wc -l # e.g. 47 / 50
grep -c '"-passive"' runs/*/events.jsonl          # expert-flag string, near 0 across the corpus

Readout: rare-or-absent at existing N tells you where to look, nothing more. Family 1’s own caveat applies immediately: absence in 50 already-collected rollouts at whatever temperature they were sampled at is not evidence of a K-gap — under-sampling collapses pass@k toward pass@1 and manufactures false K-gap readings (section-B, Family 1). Route to Probe 1.

Verbalized confidence, free. Ask the policy, before it acts: “List every subdomain-enumeration approach you could use here, and rate 0-100 your confidence you’re not missing a stronger passive-source option.” Kadavath et al. arXiv:2207.05221 [HIGH] established that models are reasonably calibrated on this kind of pre-generation self-report even when open-ended generation is wrong — a high self-reported completeness score that turns out to be false (the model never mentions amass) is itself a P-gap-flavored signature worth carrying forward, not a verdict on its own.

3. Probe 1 — forced-logprob + recognition (one forward pass)

What you run. Teacher-force the exact expert string through the policy at the decision token — no sampling, no gradient:

target = "amass enum -passive -d target.com -config ~/.config/amass/config.ini"
logps  = teacher_force(policy, prefix=trajectory_up_to_tool_call, target=target)
nll_mean   = -mean(logps)                       # whole-target perplexity
min_k_stat = -mean(sorted(logps)[: int(0.15 * len(logps))])   # worst 15% of tokens — Shi et al.

Min-K% Prob — the mean of the lowest-probability slice of tokens rather than the whole-sequence mean — catches a single catastrophically bad token (e.g. the -passive flag) hiding inside an otherwise fluent, plausible-looking command. Shi et al., arXiv:2310.16789 [HIGH], 700+ citations, the mainstream instrument for exactly this.

Two mandatory adjustments, or you’ll manufacture a false K-gap:

• Sweep 3-5 paraphrasings of the system/task prompt and take the minimum perplexity across them (equivalently, max logprob) — a single unlucky framing of the task is only a lower bound on what the model “knows,” per Jiang et al.’s prompt-sensitivity finding, arXiv:1911.12543 [HIGH].
• Normalize for surface-form competition if more than one valid rendering of the target exists (-passive vs --passive-only; amass enum vs amass intel) — raw target-string probability underestimates knowledge when synonymous forms split probability mass. Holtzman et al., arXiv:2104.08315 [HIGH].

In parallel, run the forced-choice recognition twin: present a 4-option MCQ (“which command maximizes passive subdomain source coverage: A) subfinder -d target.com B) amass enum -passive -config ... C) … D) …”), score selection accuracy across the same paraphrasings.

Readout:

Don’t commit to trajectory-level SFT on the strength of a single-prompt logprob read. Gekhman et al., arXiv:2405.05904 [HIGH, EMNLP 2024]: SFT rows the pre-training policy classifies as genuinely novel (via k-sampling) are fit dramatically slower, and once fit, linearly increase hallucination on unrelated, previously-known facts. Ghosal et al., arXiv:2406.14785 [MED]: SFT on lesser-known facts teaches the model to ignore the conditioning subject/context and emit a generic default — i.e. naive SFT on an unconfirmed K-gap can strengthen the “reach for the salient tool” default rather than fix it. This is the reason Probe 1 is a screen, not a commit signal — proceed to the oracle-injection step (§5) before writing a single SFT row.

4. Probe 2 — the pass@k sweep, and what the SHAPE tells you

What you run. The unbiased combinatorial estimator — never naive c/k — first defined for exactly this purpose by Chen et al.’s Codex/HumanEval paper, arXiv:2107.03374 [HIGH]:

def pass_at_k(n, c, k):
    """n = samples generated, c = number correct, k = budget. Numerically stable form."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

Sample n ≥ 4·k_max completions per decision point at temp 0.7-1.0 (never greedy — under-sampling collapses pass@k toward pass@1 and manufactures a false ceiling), across ≥10 rephrasings of the task prompt, same format for every checkpoint you compare. Score c = number of rollouts where the target action (the amass invocation with the expert flag set, specifically, not “any working subdomain enumeration”) actually fires. Sweep k = {1, 4, 16, 64, 256} and plot pass@k vs. log(k).

Cost reality check: n ≥ 4·k_max at k_max=256 means n≥1024 — expensive for a full T≈100-turn agentic rollout. This is affordable for F2/F3 because they’re single-decision-point resamples (re-roll from the same trajectory prefix up to the tool call, not the whole episode) — cheap, parallel, no environment cost. Reserve full-episode resampling at large k for after you’ve narrowed the search with the cheap single-decision version; full-trajectory Pass@(k,T) sweeps (§9) are reserved for smaller k, larger T instead.

What the SHAPE means:

• Flat near-zero across all k up to 256, across all 10 rephrasings → K-gap candidate, proceed to oracle injection (§5) before finalizing — per Family 1’s convergent caution, a null pass@k result on a pure recall-flavored decision like “which tool has broader source coverage” can still be a false K-gap at small k; Yang et al. show RL on binary correctness reward yields real recall gains in a closely analogous zero-shot factual-recall setting by moving a correct token out of an astronomically low-probability tail — cited via section-A’s ladder as [arXiv:2605.07153] but note this specific id is present only in ledger-A’s confirmation and is not independently re-verified in ledger-B; treat as [LOW], corroborating not load-bearing.
• Rises sharply and plateaus well above pass@1’s floor by k≈16-64 (e.g. pass@1 = 3%, pass@64 = 55%) → R/P-gap: the correct action is in-support, just not the default top choice. Proceed to §5.
• If you have both a base and an SFT/RL-trained checkpoint: plot both curves together — this is framework.md §2.2’s crossover test, not re-derived here. Trained wins at k=1, base catches up/exceeds by k=64-256 → elicitation-only (Yue et al., arXiv:2504.13837 [HIGH]); contested by requiring reasoning-path correctness, not just final-answer match (Wen et al., arXiv:2506.14245 [MED]) and by prolonged-RL-with-KL-control counter-evidence (Liu et al., ProRL, arXiv:2505.24864 [HIGH]). State the crossover as contested when you report it — see framework.md §2.2 and Contested edges §1.
• Discrete/low-cardinality caution. A flag string is exactly the kind of discrete, low-cardinality answer space where pass@k at large k is dominated by guessing rather than genuine reliability (Cover@τ, framework.md §2.3, not re-derived here — Dragoi et al., arXiv:2510.08325 [MED]). Report Cover@τ (τ≈0.3) alongside any pass@64+ number you’d otherwise be tempted to read as a clean win.

Mandatory control for any few-shot variant of this probe: if you’re also sweeping in-context demonstration count (dose-response per Many-Shot ICL, Agarwal et al., arXiv:2404.11018 [HIGH]), always run a shuffled-label control alongside the correct-demo condition — Min et al., arXiv:2202.12837 [HIGH]: randomizing demonstration labels barely hurts accuracy across 12 models, meaning most naive few-shot lift is format/label-space calibration, not a learned mapping. Only the delta between correct-demo and shuffled-demo few-shot is real evidence of an R/P-gap; if correct-demo ≈ shuffled-demo, the zero-shot gap was a schema-calibration artifact, fixable with a system-prompt nudge, not training data. Zhao et al., arXiv:2405.19874 [MED-HIGH]: ICL underperforms fine-tuning as an elicitation technique and the gap widens at higher capability — a negative few-shot-only probe on a frontier-scale model is not, by itself, license to conclude K-gap.

5. Probe 3 — in-context oracle injection: the K-vs-P splitter

This is the single sharpest instrument in the whole battery, and the direct answer to “does putting the answer in context and re-running settle it.”

What you run. Take the exact prompt that produces F2/F3, and append the missing knowledge verbatim into the system prompt or tool description:

Addendum to system prompt:
"For subdomain enumeration, prefer `amass enum -passive -d {target} -config amass.ini`
over `subfinder` — amass's default passive source set includes {source X}, which
subfinder's default list omits. Chain to `httpx` for liveness confirmation afterward."

Re-run N=32-50 rollouts, cold start, same challenge, same temperature as your Probe 2 baseline.

Readout — the split itself:

• Fixes it near-zero-shot (correct-tool-use rate jumps from ~2% to 70%+) → CONFIRMED K-gap. The policy machinery to use the information was already there; only the declarative fact was missing. Route to data curation (./knowledge-curation.md), not raw trajectory-level SFT — see Probe 1’s caution above on why that specific move backfires on an unconfirmed K-gap.
• Doesn’t fix it, even with the answer handed over verbatim → CONFIRMED P-gap — a readout/decision-layer failure, not a knowledge failure. This is exactly the mechanistic signature Chen documents directly on tool-call failures: per-candidate attention finds the correct tool 80% of the time (vs. 21% chance) yet the model still calls the wrong one; prompt-side fixes (reordering/duplicating the gold tool in the schema — which is structurally what oracle injection is) recover ≤23% of failures, while decision-readout-layer interventions recover 59-91% (Chen, arXiv:2606.16364 [LOW, single-author preprint — promising, not yet validated]; drawn from section-A’s ladder). Route to Probe 5 / RL-DPO fixes, not more demonstrations.

Teacher Flip Rate — three privileged-context levels, not one. Build a graduated version of the oracle instead of a binary present/absent:

• L1 — abstract hint. Self-generated by the same model, conditioned on the gold trajectory, asked to produce “a hint containing the core knowledge needed” — no distribution shift from an external author.
• L2 — concrete-but-partial. amass --help excerpt, or one worked example — tests the K-gap question specifically.
• L3 — full oracle. The exact gold command in context (what you just ran above).

Sample G=8-16 rollouts unprivileged, keep only the wrong ones, re-sample the same wrong prompts at each level (inference only, no weight update). TFR(level) = fraction that flip to correct. Mandatory control: also compute TFR against a semantically-empty, length-matched context (“think step by step, be thorough”) — if TFR is elevated there too, your verifier has a leak and you should distrust the whole probe. This is the OPSA framework’s Teacher Flip Rate, Fu et al., arXiv:2605.15239 [MED, recent].

Readout table:

Two caveats before you trust a positive TFR: (1) revealing the full gold answer as a hint can actually underperform an abstract hint — Chen, Peng et al.’s NuRL, arXiv:2509.25666 [MED, ICLR 2026 accepted poster], found full-info hints cause shortcut-copying rather than genuine reactivation, so don’t skip straight to L3 and assume it’s the most informative signal. (2) a hint that creates learning signal under hinted rollouts does not guarantee the deployed hint-free policy improves — Xia et al.’s HiLL, arXiv:2604.00698 [LOW, promising], is the mandatory downstream control: always re-validate any fix completely hint-free before declaring victory.

Run this on F1 too. For the “no opening methodology” symptom, the oracle is a structured checklist: “Standard operating order: (1) passive recon, (2) active recon, (3) enumerate, (4) exploit, (5) verify.” If injecting the checklist alone fixes turn-1 compliance, that’s a K-gap (the model didn’t know to front-load recon as a declarative procedure) — curate data. If it still skips steps even with the checklist explicitly in context, that smells like the attention/decision-readout class of failure, or a genuine planning/state-management limitation — route to framework.md §6 and its per-turn fault taxonomy (§6.1) for the deeper cut; this chapter’s single-decision-point probes aren’t built to fully resolve a structural planning failure.

6. Probe 4 — generation-vs-recognition (MCQ) split

Why this is a separate probe from Probe 1’s recognition check: Probe 1 asks whether the correct answer is ranked competitively. This probe asks whether the model’s own judgment channel (pick the best of several options, one of which is its own default) disagrees with its generation channel (what it actually does by default). West et al.’s Generative AI Paradox names the theoretical basis: generation and discrimination/understanding are dissociable capacities, and the dissociation can run in either direction — arXiv:2311.00059 [HIGH].

What you run. Build the candidate set from the model’s own rollouts, not hand-written strawmen — this matters, because a distractor set the model would never actually produce tells you nothing about its failure mode:

state s = trajectory prefix up to the enumeration decision
C = { a_expert:  "amass enum -passive -d {t} -config amass.ini"  (verified-correct),
      a_shallow: <the model's OWN greedy pass@1 rollout at s>,
      a_dist_1, a_dist_2: <the model's OWN other sampled rollouts at s> }

Present s + C in randomized position order (position bias is real — Zheng et al., arXiv:2306.05685 [HIGH], 432 citations), ask the model to (a) select the best option and (b) justify by citing the differentiating feature. Run a discriminative control variant too (single-token choice / logprob-over-options, no justification) — if it disagrees with the generative select-then-justify answer, the justification may be post-hoc rationalization, not the real reason.

Readout:

This is the Elicitation Game’s exact signature transplanted onto a tool-choice: Hofstätter et al., arXiv:2502.02180 [MED, ICML 2025 poster], found prompting/steering suffice to elicit hidden capability on MCQA-shaped probes but fail on generative/code-shaped tasks — a null result from Probe 4’s recognition side is much more trustworthy than a null result from raw open-ended prompting alone (§7’s ladder makes this ordering explicit).

7. Probe 5 — token-level top-5 logprob-rank check

What you run. At the exact generation step where the wrong tool name gets chosen — the token position where subfinder beats amass, or the default flag string beats the expert one — pull the base model’s top-5 logprobs at that position:

logits_at_decision = policy.forward(prefix_up_to_decision_token)
top5 = torch.topk(logits_at_decision, 5)
rank_of_amass_token = (logits_at_decision.argsort(descending=True) == amass_token_id).nonzero()

Readout:

Cross-check against Probe 3’s mechanistic finding: if attention already localizes the correct tool 80% of the time (per Chen, arXiv:2606.16364 [LOW]) but the readout still picks wrong, and the token IS in the top-5 here, you have two independent signals converging on the same P-gap verdict — exactly the “agreement across 3+ families” bar this chapter’s BLUF asks for.

8. Probe 6 — the spurious-reward control, before you trust any RLVR result

Why this probe exists: a GRPO run that appears to fix F2/F3 is not, by itself, evidence the reward was doing discriminative work. It might just be amplifying whatever the sampling temperature already surfaces more of.

What you run. Train the identical GRPO setup twice, a few hundred steps each, on the same prompts:

• (a) Real reward: +1 if the trajectory used amass with the expert flag set AND the flag verified correct, 0 otherwise.
• (b) Shuffled/random reward: the same prompts, but the reward for each rollout is assigned by a coin flip independent of what the rollout actually did.

Readout:

• (b) also moves the target metric substantially (amass-usage-rate climbs under the random-reward run too) → you were mostly amplifying a pre-existing high-prior behavior, not doing genuine discriminative reward-shaping. This is a real, confirmed finding, not a hypothetical — Shao, Li, Xin, Geng et al.’s Spurious Rewards paper, arXiv:2506.10947 [MED — the finding is confirmed but was demonstrated on a specific model family; verify it replicates on your own roster before generalizing across all five families]. Invest in exploration/coverage instrumentation, not more reward engineering.
• (a) works, (b) doesn’t → the reward is doing real discriminative work on an in-support-but-mis-ranked behavior — clean P-gap, RL is well-targeted.
• Neither moves the metric → K-gap; stop spending RL compute, go curate data.

Run this as a gate, not a postmortem. Before committing a full GRPO budget to “fix” F2/F3, run both arms for ~300 steps first — cheap relative to a full training run, and it directly determines whether the rest of your RL compute is well spent.

9. Probe 7 — Pass@(k,T) for the agentic/compositional case

This is where the Exploration-collapse dynamic actually gets diagnosed, and it’s the one probe in this battery that requires framework.md’s full derivation (§2.4) — not re-derived here, only operationalized.

Segment the portfolio first. F2/F3 in isolation (does turn-1 use amass with expert flags — a single decision, not gated by anything upstream) is the simple/independent-retrieval segment. The compositional case is where the flag is reachable only because amass’s passive enumeration surfaces a non-obvious subdomain that the agent must then correctly interpret and pivot to — a chained, sequentially-gated decision, structurally identical to Zhai et al.’s “Category C,” arXiv:2604.14877 [MED].

What you run. Same estimator as pass@k, with a second axis T = max tool-calls/turns:

PASS@(k,T)(q, π) = 1 - C(n - c_T, k) / C(n, k)     # c_T = correct at interaction depth T specifically

Sweep T ∈ {1, 2, 4, 8, 16, 32} at fixed k=1 first — does giving the agent more turns/self-correction rounds alone, no training change, recover the compositional path? Then sweep k ∈ {1, 4, 16, 64} at your deployment T. Run this separately on the simple and compositional segments, before and after your rejection-sampling-SFT checkpoint.

The falsifiable prediction, per section-B’s synthesis of Zhai et al.: on the simple segment, expect the static crossover to hold (SFT works, further RL may plateau — a clean R/P-gap, cheaply closed, per Yue et al. arXiv:2504.13837 [HIGH]). On the compositional segment, expect matched-data SFT on the exact winning trajectories to regress the capability boundary while on-policy RL widens it — a net −4 vs +4 pattern in Zhai et al.’s own numbers, isolating self-directed exploration during RL, not data exposure, as the causal factor.

Readout:

One more control before you trust a “regression” reading as genuine exploration-collapse: the pass@k-minus-pass@1 gap mechanically shrinks as a policy’s sampling distribution concentrates during training — independent of any real change in underlying capability. Yu, arXiv:2511.16231 [LOW, single preprint]. Corroborate the regression reading with an entropy/diversity check on the same checkpoints (framework.md §7’s RL-PLUS “capability boundary collapse,” not re-derived here) before concluding the SFT itself caused the regression, rather than the training run simply narrowing its own sampling distribution for unrelated reasons.

10. Probe 8 — the minimal-parameter / few-shot-SFT elicitation probe

What this probe answers that the others don’t: a quantitative recovery-cost number, not just a direction.

What you run — two variants, pick based on weight access:

• Minimal-parameter unfreeze. Freeze the model; unfreeze a random parameter subset starting at ~100 parameters, scale up to ~10,000; fine-tune briefly on the task’s own verifiable labels (no curated demonstrations needed — the labels come from your own flag verifier).
• Few-shot LoRA. Fine-tune a rank-1 to rank-4 LoRA on 5-20 demonstration rows — self-distilled from the model’s own best-of-N successful rollouts if no external teacher exists (this is a diagnostic use of light SFT, not the production data-curation pipeline — that pipeline is kinds-of-sft.md’s territory, deferred here).

Readout:

This mirrors an unpublished-but-confirmed-on-OpenReview result: 10-100 randomly chosen trained parameters recovering roughly half the full-fine-tune gap (NeurIPS 2025 poster, OpenReview id Dkgx2pS4Ww — no arXiv mirror exists for this one; cite cautiously per the ledger’s own note). More load-bearing: Greenblatt et al.’s password-locked calibration methodology, arXiv:2405.19550 [HIGH] — a handful of high-quality SFT demonstrations is often sufficient to fully elicit a locked capability.

Build ground truth before trusting this probe on your real ambiguous case. Take a capability you already know the base model has (verified via Probe 2’s high-k pass@k). SFT a clone: with a password token, train on 20-50 oracle-quality trajectories (full expert behavior); without the password, train it to imitate the observed shallow default. Sweep re-elicitation budget {4, 16, 64, 256, 1024 rows} on this clone, and separately run a noise-injection pre-check — Tice et al., arXiv:2412.01784 [MED, NeurIPS 2025 poster]: sandbagging/suppressed clones show a non-monotonic accuracy bump under mild Gaussian weight noise (σ ~ 0.001-0.01 relative to per-layer std), vs. monotonic degradation for a genuinely absent capability. Record how much budget it took to recover the known-present capability, then run the identical protocol on your real F2/F3 case and compare recovery cost against that calibration curve.

11. Probe 9 — the elicitation ladder: synthesizing 0-8 into one verdict

The AI-safety capability-elicitation literature solved almost exactly this problem for a different reason (sandbagging/eval-integrity), and its protocol is directly reusable — framework.md §5 covers the full genealogy (Greenblatt, Hofstätter, van der Weij, Ryd), not re-derived here. The ladder itself, made concrete for F2/F3:

Probe 3 (oracle in context)  →  few-shot (2-3 examples)  →  Probe 8 (light SFT, 10-50 rows)  →  Probe 6-gated GRPO  →  (still nothing) → confirmed K-gap
      (free)                       (cheap)                        (cheap-medium)                     (expensive)

• Step A — inject the answer (Probe 3). Fixed near-zero-shot → stop, K-gap confirmed, route to data curation.
• Step B — few-shot, 2-3 correct-usage examples, shuffled-label controlled (Probe 2’s control). Fixed → pure elicitation/prompting gap, no training needed at all.
• Step C — light SFT on 10-50 rows (Probe 8). Fixed → P/R-gap via light elicitation, matches Greenblatt’s finding directly.
• Step D — Probe-6-gated GRPO (never skip the spurious-reward control). Fixed and durable across checkpoints (re-check Pass@(k,T), Probe 7, for regression on the compositional segment) → confirmed P/R-gap, real discriminative fix.
• Step E — nothing works, even at large parameter/data budgets, even with the full oracle in context. Before concluding a hard ceiling, rule out deliberate or contextual suppression — van der Weij et al., arXiv:2406.07358 [HIGH]: frontier LMs can be prompted or fine-tuned to strategically underperform on recognizable-as-eval inputs while performing normally elsewhere. If ruled out, this is the genuine ceiling — a K-gap that survives even oracle injection is architecturally different from one that a light nudge fixes.

Order matters at Step D. Neither SFT nor RL alone reliably elicits held-back performance from a degenerate policy; SFT-on-weak-demonstrations-first, then RL, is what fully elicits it — RL-first from a degenerate starting point “almost always leads to reward hacking rather than genuine improvement.” Ryd et al., arXiv:2604.22082 [MED, 2026]. This is the reason the ladder above puts light SFT before GRPO even when both are on the table.

12. The purpose-built bottleneck CTF — a controlled instrument, not just an observational probe

Every probe above reads an existing decision point. This section builds one on purpose, so the verdict isn’t confounded by whichever real challenges happen to be in the portfolio.

Design. Author a challenge (a recommendation for challenge-builder’s seat, not something this chapter builds) where the flag is reachable only via the full methodology — no partial-credit path:

• A target with a subdomain (e.g. internal-api.<target>) that is discoverable only through a passive certificate-transparency-adjacent source that amass’s default source configuration queries and subfinder’s default source list omits — verify this asymmetry against each tool’s actual current default source list before building on it, source lists drift.
• The flag lives behind that specific vhost, reachable only after the correct enumeration → correct pivot → correct probe chain.
• The task prompt is deliberately light: “Find the flag.” No hint toward amass, no hint toward expert flags — leaking Probe-3-style oracle information into the baseline prompt would contaminate the very solve-rate you’re trying to read.

Run it. Pass@k sweep, k = {1, 4, 16, 64, 256}, across the full 5-family roster, same estimator as §4.

Read the solve-rate band — this reuses the project’s own locked GRPO go/no-go rule (llmresearch-handbook.md rule #7: baseline must hit 30-60%; below 5% no signal; above 80% no headroom) as a diagnostic read, not just a training gate:

Run this as a portfolio-level check, not a single-challenge anecdote. τ-bench’s own leaderboard shows exactly this kind of number can mislead if read only at pass@1: GPT-4o’s 61% pass@1 on retail tool-use collapses to under 25% pass^8 (all-k-succeed), Yao et al., arXiv:2406.12045 [HIGH]. A single high-k solve on a single bottleneck challenge is not the same claim as reliable multi-trial coverage across the portfolio — report the full curve, not one point on it.

13. The routing flowchart

flowchart TD
  Sym["Symptom: agent skips methodology (F1),<br/>calls subfinder not amass (F2),<br/>uses default not expert flags (F3)"] --> P0["Probe 0: grep run-trace corpus<br/>for amass / expert-flag strings<br/>across N existing rollouts"]

  P0 --> Freq{"Rare or absent<br/>at existing N?"}
  Freq -->|"Common already"| NotAGap["Not a gap at this decision point —<br/>check a different turn/challenge"]
  Freq -->|"Rare / absent"| P1["Probe 1: teacher-force the expert<br/>command, read logprob + rank;<br/>MCQ recognition, ge3 paraphrases"]

  P1 --> LP{"Logprob near floor AND<br/>MCQ recognition also fails,<br/>across paraphrasings?"}
  LP -->|"Yes"| P2a["Probe 2: pass@k sweep,<br/>k=1..256, ge10 rephrasings"]
  LP -->|"No — finite logprob<br/>or MCQ succeeds"| P3["Probe 3: in-context oracle<br/>injection + Teacher Flip Rate"]

  P2a --> Shape{"Correct action ever<br/>appears, any k, any phrasing?"}
  Shape -->|"Never, even k=256"| P3
  Shape -->|"Rises, plateaus above<br/>pass@1's floor"| P3

  P3 --> Fix{"Injecting the expert<br/>command / flags fixes it<br/>near-zero-shot?"}
  Fix -->|"No — still fails even<br/>with the answer handed over"| Know["KNOWLEDGE gap:<br/>curate paraphrase-rich SFT rows<br/>-&gt; knowledge-curation.md"]
  Fix -->|"Yes, but TFR(L1) low,<br/>only TFR(L3) is high"| P4["Probe 4: generation-vs-<br/>recognition (MCQ) split"]
  Fix -->|"Yes, and TFR(L1)<br/>already high too"| P5["Probe 5: token-level<br/>top-5 logprob-rank check"]

  P4 --> Disc{"Discrimination accuracy<br/>much greater than<br/>generation accuracy?"}
  Disc -->|"Yes"| P5
  Disc -->|"No, both low"| Know

  P5 --> Rank{"Expert token in the<br/>base model's top-5?"}
  Rank -->|"Yes"| P6["Probe 6: spurious-reward<br/>control before trusting RLVR"]
  Rank -->|"No, not even top-20"| Know

  P6 --> Spur{"Random reward ALSO<br/>moves the metric?"}
  Spur -->|"Yes — false positive"| ReDiag["Distrust the RLVR signal;<br/>re-run Probes 1-5"]
  Spur -->|"No — real discriminative work"| P7["Probe 7: Pass@(k,T),<br/>simple vs compositional segment"]

  P7 --> Comp{"Matched-data SFT REGRESSES<br/>the compositional segment's<br/>boundary (entropy-checked)?"}
  Comp -->|"Yes"| Explore["EXPLORATION-COLLAPSE dynamic:<br/>on-policy RL with diversity<br/>preservation, not more SFT<br/>-&gt; agentic-rl.md"]
  Comp -->|"No regression"| RP["PRIOR / POLICY gap:<br/>light SFT (Probe 8) or DPO/GRPO<br/>on this exact contrast<br/>-&gt; intervention-per-gap.md"]

  classDef verdict fill:#132b22,stroke:#34d399,color:#eafaf3;
  class Know,Explore,RP verdict;

14. The protocol, in order

Refuse to collapse this into one sentence. Per framework.md §8’s own conclusion — restated here because it’s exactly as true at the single-decision-point granularity as at the challenge-portfolio granularity — report the verdict per decision point, segmented by whether it’s a simple or compositional structure. A skeptical reviewer will catch “F2/F3 is an execution gap” stated as a flat claim; “F2 is a confirmed P-gap (Probe 3 fixed it, Probe 5 confirms top-5, Probe 6 confirms real discriminative work) and F1 is unresolved pending framework.md §6.1’s fault labeling” is the defensible version.

Cross-links

• Diagnosing the gap — a scientific framework — the theory this chapter operationalizes: full pass@k/Cover@τ/Pass@(k,T) derivations (§2), the elicitation-ladder genealogy (§5), the planning/state-management axis for F1-style symptoms (§6), and the RL-PLUS entropy-collapse mechanism this chapter’s Probe 7 leans on (§7).
• ./taxonomy.md — the precise K/R/P definitions this chapter assumes; §0 is the canonical K/R/P ↔ knowledge/execution/exploration mapping this chapter’s §0 points to rather than re-derives.
• ./intervention-per-gap.md — what to actually build once a probe returns a verdict: the R/P-gap DPO/GRPO row shape, the K-gap SFT-curation recipe.
• ./knowledge-curation.md — the full K-gap data-row recipe (paraphrase volume, directional pairs, entity-connective text) once Probe 3 confirms a K-gap; not re-derived here.
• ./trajectory-amplification.md — the forgetting/hallucination-amplification mechanics behind Probe 1’s caution against premature SFT on an unconfirmed K-gap.
• The decision — the one-line version of the routing question both this chapter and framework.md expand on.
• Contested edges & landmines §1, §7 — the crossover-test’s own contested status, referenced in §4 rather than re-argued.
• Agentic & multi-turn RL — where the Exploration-collapse fix (on-policy RL, entropy preservation) is implemented once Probe 7 diagnoses it.
• The kinds of SFT — the production data-curation pipeline Probe 8’s diagnostic light-SFT is explicitly not a substitute for.
• Foundations: the one axis that predicts everything — the on/off-policy distinction underlying why oracle-injection (§5) and DPO/GRPO pairs must stay on-policy to be trustworthy.

Bibliography

Cited but not counted as arXiv: OpenReview id Dkgx2pS4Ww (Donoway, Joren, Somani, Sleight, Michael et al., Quantifying Elicitation of Latent Capabilities in Language Models, NeurIPS 2025 poster) — §10, no confirmed arXiv mirror, cite cautiously.

Confidence calibration, restated: every id above is CONFIRMED in ledger-A.md or ledger-B.md; none is invented. Where section-A’s narrative referenced an id absent from either ledger (Verifier Gain, RankAlign, and two OpenReview-only ids with no ledger row), that material was dropped from this chapter rather than cited unverified — the load-bearing generation-vs-recognition claim in §6 rests on West et al. (confirmed, HIGH) instead.

What knowledge data actually looks like (once a K-gap is confirmed)

The question this chapter answers: once diagnosis has ruled in a knowledge gap — the model never emits the right fact/tool/flag at any sampling budget, and handing it the answer in-context fixes it zero-shot — what shape of file do you actually build? Not “paste the README into the system prompt and fine-tune on that.” Not “generate one help-page dump per tool.” Not “make it memorize a tool list.” BLUF, upfront: a fact that appears exactly once, in exactly one wording, in your training data is storable (the model can echo it back near-verbatim) but not extractable (it cannot answer a differently-phrased question about it) — feeding the README once is close to a null intervention, and no amount of downstream trajectory-level SFT repairs that gap once it’s baked in. Extraction has to be purchased at data-construction time: paraphrase volume, bidirectional coverage, and — for the specific “wrong tool, default not expert flags” failure this book keeps coming back to — execution-verified tool-CALL rows, not declarative prose. This chapter is the graded ladder from weakest to strongest, a worked training row at every rung, and the full production schema at the end.

Scope note, standing rule: every citation below is general ML/knowledge-injection literature. Nothing here leans on an academic cybersecurity-LLM paper — this book’s rule throughout, restated because this is exactly the chapter where it’s tempting to reach for a domain-specific “how we built our pentest corpus” paper instead of the general mechanism.

1. The load-bearing principle: one exposure ≈ zero extraction

Ground truth, from a controlled synthetic-biography corpus where the true answer is knowable and every exposure is countable: Allen-Zhu & Li’s Physics of Language Models Part 3.1 (arXiv:2309.14316) trains models on a biography dataset (bioS(N)) and measures closed-book QA accuracy against how many distinct wordings of each fact appeared during training. A fact seen in one wording: ~0-10% QA accuracy, independent of model size, training duration, or how much downstream instruction-tuning you throw at it afterward. Add ~5 diverse rewrites of the same fact and accuracy jumps to ~97%. Even sentence-permutation alone, with zero new wording — literally shuffling the clauses of the same sentence — already lifts extraction from 4.4% to 70%. The mechanism (linear-probe evidence in the same paper): without augmentation, the fact is encoded only at the local context hidden state around where it appeared, never bound to the entity-name token itself, so it can’t be retrieved by a query that doesn’t reproduce that exact local context.

The corollary that matters for this whole chapter: the paper’s own experiments show no amount of downstream instruction fine-tuning repairs an under-augmented base representation. SFT can teach the model how to answer, but if the fact was never bound to the entity token during pretraining/CPT, SFT has nothing to point at. This is the mechanistic answer to “does trajectory-level SFT amplify an unfixed knowledge gap” from the storage side: SFT inherits the storage format, it does not repair it (full amplification argument in Does trajectory SFT amplify an unfixed K-gap?).

The budget for how many exposures “enough” actually is: Part 3.3 (arXiv:2404.05405) measures raw storage capacity at ~2 bits of factual-tuple knowledge per parameter, achieved at roughly 1000 exposures per fact; drop to 100 exposures and capacity roughly halves to ~1 bit/param; junk-diluted corpora (1 useful token to 7 junk) cut capacity ~20×, partly recovered (~2×) by a consistent domain-marker prefix. (A differently-measured, higher raw-memorization ceiling of ~3.6 bits/param exists — Morris et al. — but that’s a random-bitstring membership-inference regime, the theoretical ceiling for arbitrary data, not the practical ceiling for genuinely extractable, augmented, structured knowledge; the two numbers aren’t in tension, they’re measuring different things.)

Why some facts are missing in the first place, before you even get to fixing it: Kandpal et al. (arXiv:2211.08411) show real-corpus QA accuracy correlates causally with how many pretraining documents mention the entities involved. amass is a smaller, less-frequently-discussed tool than subfinder on the open web relative to its actual capability — that’s plausibly why “amass covers passive-DNS/certificate-transparency sources subfinder’s defaults miss” ends up in a model’s blind spot: not an architectural limit, a training-corpus frequency artifact. That’s the diagnosis-side story; this chapter is the fix.

Build this: before writing a single training row, budget for hundreds, not one, exposures per fact — different wordings, different formats, different directions. If your curation plan has one row per fact, stop; you’re about to reproduce the 2309.14316 null result.

2. The graded ladder — six rungs, weakest to strongest

Six shapes of data, in increasing extraction reliability, each with a worked training row for the running example: “amass covers a passive-DNS/certificate-transparency source subfinder’s default flags miss, and subfinder has an expert flag-set (-active, -all, -rl, -oJ, …) the agent never reaches for.”

graph LR
  R0["Rung 0<br/>Raw doc<br/>(README / --help)"] -->|"~0% extraction<br/>2309.14316"| R1["Rung 1<br/>Paraphrase volume<br/>WRAP-style"]
  R1 -->|"fixes non-directional<br/>recall, NOT direction"| R2["Rung 2<br/>Forward+reverse<br/>synthetic QA"]
  R2 -->|"declarative fact ≠<br/>correct tool CALL"| R3["Rung 3<br/>Doc→tool-CALL<br/>rows (RAT + verified)"]
  R3 -->|"single facts, not<br/>chained reasoning"| R4["Rung 4<br/>Entity-pair<br/>connective text"]
  R4 -->|"external synthesis<br/>can drift/omit"| R5["Rung 5<br/>Self-play<br/>Self-QA"]

  style R0 fill:#3a1a1a,stroke:#f87171,color:#fee2e2
  style R3 fill:#132b22,stroke:#34d399,color:#eafaf3

2.0 Rung 0 — raw docs are source material, never a training target

The README, --help output, man page, OpenAPI spec: these are the input to a curation pipeline, never trained on directly and unaugmented. Ovadia et al.’s RAG-vs-FT study (arXiv:2312.05934) is the floor to beat: unsupervised fine-tuning on raw domain text consistently loses to simple retrieval-augmented generation on knowledge-injection tasks — training on the raw doc doesn’t even reliably clear the bar a zero-training RAG pipeline sets for free. Cheng et al.’s AdaptLLM (arXiv:2309.09530) diagnoses why directly: continued pretraining on raw domain text does inject some knowledge, but “drastically hurts prompting ability for QA” — the model gets worse at being asked things, even about the material it just trained on, unless every raw chunk is paired with task-formatted continuations before CPT.

// NOT a training row — CPT/curation SOURCE MATERIAL ONLY, never the target
{
  "source_id": "subfinder-help-v2.7.1",
  "raw_text": "$ subfinder -h\nFlags:\nINPUT:\n   -d, -domain string[]   domains to find subdomains for\nSOURCE:\n   -s, -sources string[]  specific sources to use for discovery\n   -recursive             use only recursive subdomain enumeration sources\n   -all                   use all sources for enumeration (slow)\nFILTER:\n   -active                display active subdomains only\nOUTPUT:\n   -oJ, -json             write output in JSONL(ines) format\n   -rl, -rate-limit int   maximum number of http requests to send per second"
}

This never appears as an assistant target. It’s the thing every downstream rung is derived from.

2.1 Rung 1 — paraphrase volume (WRAP-style)

Maini et al.’s WRAP (arXiv:2401.16380) paraphrases noisy web text into several styles (including explicit “QA format”) and joint-trains real+synthetic: ~3× pretraining speedup, >10% perplexity gain, >2% zero-shot QA gain at fixed compute. This is the direct fix for the 2309.14316 null result — turn one raw doc into several differently-worded statements of the same fact.

Rephrase count is not free of diminishing returns: a very recent Stanford result (arXiv:2603.18534, promising, not yet independently validated) finds rephrase-count efficiency saturates around ~32 rewrites per document, and that packing those 32 rewrites into one long megadocument (rather than 32 separate short training examples) pushes efficiency further (~1.48× → ~1.8×). Practical read: don’t stop at 5, and don’t naively keep generating past ~32 either — and prefer packing multiple rewrites into a single training sequence over one-rewrite-per-row.

// Rung 1 — same fact, 4 of ~20-32 total rewrites, task-formatted per AdaptLLM (2309.09530)
[
  {"messages": [{"role": "user", "content": "What's the -all flag on subfinder for?"},
    {"role": "assistant", "content": "-all tells subfinder to query every configured source, not just the fast/default subset — slower, broader coverage."}]},
  {"messages": [{"role": "user", "content": "True or false: subfinder's default run queries every available source."},
    {"role": "assistant", "content": "False. The default subset is a faster subset; -all opts into every configured source."}]},
  {"messages": [{"role": "user", "content": "To get subfinder's broadest possible source coverage, which flag do you add?"},
    {"role": "assistant", "content": "-all — it enables every configured source instead of the default fast subset."}]},
  {"messages": [{"role": "user", "content": "I ran subfinder with no flags and got fewer subdomains than expected. Why, and how do I fix it?"},
    {"role": "assistant", "content": "Default subfinder uses a fast subset of sources, not all of them. Add -all to query every configured source for broader (slower) coverage."}]}
]

Still not fixing directionality — see rung 2.

2.2 Rung 2 — doc → synthetic QA, forward AND reverse

Paraphrase volume fixes non-directional recall. It does not fix a structurally separate failure: the Reversal Curse. Berglund et al. (arXiv:2309.12288) train on “A is B” and show P(correct | “B is ?”) is statistically indistinguishable from a random-name baseline — and, critically, the authors’ own tested augmentations do not fix this. Kitouni et al.’s Factorization Curse (arXiv:2406.05183) generalizes this: it’s a failure to learn the same joint distribution under different token-order factorizations, and it survives scale, reversed-token training, and naive bidirectional attention. Lin et al. (arXiv:2410.18808) narrow it further and give the actionable lever: generalization to the reverse direction is gated by training-document structure (“[Name] is [Description]” reverses one way, not the other) — and works fine when both directions are given jointly in one training example. (A very recent mechanistic follow-up, arXiv:2604.04943, promising, not yet independently validated, shows bidirectional-objective tricks like masked reconstruction do raise reversal accuracy behaviorally — but by storing forward and reverse as two separately-indexed representations, not one shared direction-agnostic fact. You’re not getting free generalization, you’re paying for a second copy.)

Practical rule: write both directions explicitly, as separate rows, in the same batch.

// Rung 2 — forward and reverse, SAME fact, explicit both ways (2309.12288 fix)
{"messages": [
  {"role": "user", "content": "Does amass or subfinder pull from certificate-transparency logs by default?"},
  {"role": "assistant", "content": "amass — its default enumeration includes CT-log and passive-DNS sources. subfinder's default source set is narrower and can miss what CT logs surface unless you widen it with -all."}
]}
{"messages": [
  {"role": "user", "content": "Which tool should I reach for if I specifically need certificate-transparency coverage that subfinder's defaults don't include?"},
  {"role": "assistant", "content": "amass — run it in passive mode (amass enum -passive -d <target>) to pull CT-log and passive-DNS sources subfinder's default set doesn't cover."}
]}

2.3 Rung 3 — doc → tool-CALL examples: the highest-value rung for “wrong flags”

This is the rung that directly answers “the agent reaches for one salient tool, uses default not expert arguments” — declarative QA about a flag is not the same skill as emitting the call. Patil et al.’s Gorilla (arXiv:2305.15334) is the founding recipe: scrape real API docs, Self-Instruct-generate NL questions per doc entry, grade candidate calls by AST-subtree match against the schema — and, the single most important mechanism here, Retriever-Aware Training (RAT): the doc schema stays in-context at train time, not just at data-generation time. The learned skill becomes “read this schema, emit this call,” not “this exact instruction maps to this memorized call” — which is what lets it generalize to an updated flag set with no retraining, and is why raw doc-only fine-tuning (rung 0) hallucinates calls that Gorilla-style RAT training doesn’t. Hsieh et al. (arXiv:2308.00675) independently confirm the context-side of this: zero-shot prompting with only the tool docs in context matches or beats few-shot demonstrations once the tool count grows — reinforcing “the schema belongs in context, not memorized as a fact.”

Qin et al.’s ToolLLM (arXiv:2307.16789) scales this to 16,464 real API docs and adds the piece that matters for multi-step agentic data specifically: curate the trajectory, not just the terminal call, via a depth-first search decision tree (DFSDT) so failed/incomplete reasoning branches get pruned before they become training data.

The 2024-2025 cluster hardens the verification step into a formal pipeline. Liu et al.’s APIGen (arXiv:2406.18518) runs candidate calls through 3-stage verification — format check, execution check, semantic check — before a row survives; a 7B model trained on the output beats GPT-4 variants on BFCL. Liu et al.’s ToolACE (arXiv:2409.00920) self-evolves a synthetic pool of 26,507 diverse API specs under a dual rule-based+model-based verifier, deliberately covering argument-shapes real scraped docs don’t have. Two very recent papers name the “default not expert flags” symptom almost literally: Yin et al.’s Magnet (arXiv:2503.07826) builds a graph over function signatures and walks reliable multi-tool signature-paths — the concrete fix for “reaches for one tool, not the complete expert set.” Greenstein et al. (arXiv:2601.17829, very recent, promising, single paper, not yet independently replicated) is a direct critique of the entire prior lineage: every pipeline from Gorilla through ToolACE optimized query/function/turn-count diversity but left argument-VALUE coverage underexplored — does the corpus actually range over -rate-limit’s valid integer domain, or does every row use the canonical default? This is the most literal match to “uses default not expert arguments” in the current literature, and it’s genuinely unresolved.

// Rung 3 — verified tool-CALL row. Schema in context (RAT), never the loss target.
{
  "system": "Tools available: subfinder(domain, sources?, active?, all?, rate_limit?), amass(domain, mode: 'passive'|'active', sources?). Find subdomains for the target, preferring the source set with the broadest coverage.",
  "messages": [
    {"role": "user", "content": "Enumerate subdomains for corp-target.com — I need CT-log and passive-DNS coverage, not just the fast default set."},
    {
      "role": "assistant",
      "content": "subfinder's default source set is narrower than what's needed here; amass's passive mode explicitly covers CT-log and passive-DNS sources. Running amass first.",
      "tool_calls": [{"id": "call_1", "type": "function",
        "function": {"name": "amass", "arguments": "{\"domain\": \"corp-target.com\", \"mode\": \"passive\"}"}}]
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "{\"subdomains\": [\"vpn.corp-target.com\", \"staging-api.corp-target.com\", \"...\"], \"source_count\": 14}"},
    {
      "role": "assistant",
      "content": "Widening with subfinder -all to cross-check against amass's results and catch anything CT-log/passive-DNS misses.",
      "tool_calls": [{"id": "call_2", "type": "function",
        "function": {"name": "subfinder", "arguments": "{\"domain\": \"corp-target.com\", \"all\": true, \"rate_limit\": 20}"}}]
    },
    {"role": "tool", "tool_call_id": "call_2", "content": "{\"subdomains\": [\"vpn.corp-target.com\", \"dev.corp-target.com\"], \"source_count\": 22}"}
  ]
}

This row was executed and verified (APIGen’s format → execution → semantic checks) before it entered the set — never hand-authored, the same confabulation risk The kinds of SFT §4 already names for trajectory data generally. Loss masked to assistant turns only — same rule as The kinds of SFT §5.1: the model must learn to produce the call, not to predict the tool’s output.

2.4 Rung 4 — entity-pair connective text (chaining/manipulation questions)

Even perfectly extractable single facts fail at manipulation — classification, comparison, inverse search across two facts — without explicit training on the relationship. Physics 3.2 (arXiv:2309.14402) shows this precisely: knowledge that’s individually stored and extractable can still fail at “which of these two applies here” unless chain-of-thought is used at both train and inference time, or the relationship itself was trained explicitly. This is exactly the “does amass’s output feed usefully into subfinder’s next pass, or should I just use one or the other” question — a chaining question about two entities, not a fact about either alone.

Yang et al.’s EntiGraph (arXiv:2409.07431) is the concrete recipe for small corpora specifically — the tool/API-doc regime, not internet-scale: extract salient entities from the source docs, then prompt an LLM (kept grounded in the source, to prevent confabulation) to generate diverse text explicitly connecting entity pairs that never co-occur in one source sentence. On a small source corpus (1.3M tokens) this produced 455M synthetic tokens (~350×) with log-linear QA-accuracy scaling as synthetic volume grows — and, the operationally important claim, the resulting parametric knowledge composes with RAG at inference rather than competing with it, so this isn’t an either/or against keeping the schema retrievable too.

// Rung 4 — entity-pair connective text: amass and subfinder never co-occur in either tool's own docs
{"messages": [
  {"role": "user", "content": "When would I run both amass and subfinder on the same target, in what order, and why?"},
  {"role": "assistant", "content": "Run amass in passive mode first — it pulls certificate-transparency and passive-DNS sources subfinder's default set misses, giving broader initial coverage. Then run subfinder -all as a second pass over the same target: its source list partially overlaps but isn't identical, so the union catches names either tool alone would miss. Running subfinder alone with default flags is the failure mode: narrower source set, and no cross-check against amass's CT-log coverage."}
]}

Cheap A/B before building EntiGraph-style tooling. Whether the entity-graph structure is actually the load-bearing ingredient, or whether large-volume prompt-engineered paraphrase without any relational structure gets you most of the way there for less engineering, is genuinely open: Tang, Wang, Wang & Lyu’s SPA (arXiv:2603.22213, very recent, ~0 citations, promising) reports a simple, heavily-prompted paraphrase baseline is “tough to beat,” including against structured approaches. Worth a cheap pilot on a handful of facts before committing engineering time to full entity-pair extraction.

2.5 Rung 5 — self-play Self-QA

Park, Zhang & Tanaka’s “New News” (arXiv:2505.01812) closes the fine-tune-vs-in-context gap with a self-play recipe: feed the model the raw fact in-context, have the model itself generate paraphrases/implications/QA pairs about it, then train on the model’s own output rather than an externally-authored rewrite. This sidesteps a real fidelity risk the paraphrase-generation literature has flagged directly: Yu & Xiong’s RePro (arXiv:2510.10681, promising, single-lab) measures that a naively-prompted rephraser (even a 70B one) measurably omits or contradicts source facts — a concrete warning against blind rung-1/rung-2 generation without a fidelity check. Self-play with the fact held in-context during generation, or an explicit faithfulness-scored rephraser, is the mitigation. One budget-relevant finding worth carrying into whichever rung does the generating: Niklaus et al.’s 1T-token study (arXiv:2604.13977, very recent) finds structured rephrase formats (tables/FAQ/tutorials — closer to how a tool doc is already shaped) consistently beat prose-style rephrase, and that generator model size beyond ~1B parameters gives no additional benefit — you don’t need your biggest available model to do rungs 1, 2, or 5’s generation work.

// Rung 5 — self-play: raw doc in-context, model generates its own QA, THAT becomes the training row
// Generation step (not trained on): model sees raw --help text in context, asked "generate 5 QA pairs
// covering this flag's purpose, its opposite/default behavior, and when you'd reach for it."
// Training step: the model's OWN output, filtered for faithfulness against the source, becomes the row —
// same shape as the Rung 1/2 examples above, but self-generated rather than externally authored.

2.6 CPT sizing: LR schedule, mix ratio, token multiplier

When rungs 0-1 are run as an actual continued-pretraining stage (not just an SFT-format QA pass), three additional hyperparameters determine whether the run helps or just destabilizes the model:

• Token multiplier. §1’s ~1000-exposures/fact budget (arXiv:2404.05405) implies a small raw source corpus (a handful of --help outputs and READMEs) cannot supply nearly enough exposure on its own — expect to expand the raw corpus 10-100× via rungs 1-4 synthesis before a fact clears the extraction bar, not the 3-5× a quick single paraphrase pass might produce.
• LR schedule. Ibrahim et al. (arXiv:2403.08763) give the mechanics that keep continued pretraining from catastrophically forgetting: re-warm the LR to a meaningful fraction of the original peak (never continue training at the fully-annealed end-of-pretraining LR), run a fresh decay sized to the actual CPT token budget, and replay a non-zero slice of general-distribution data every batch. Validated 405M-10B params; no single universal replay ratio is established — treat it as a per-corpus sweep, not a constant to copy.
• Mix ratio. Kang et al.’s >1000-LLM controlled study (arXiv:2510.01631) is the scale-corrective for rung 1’s WRAP recipe specifically: pure rephrased-synthetic pretraining is not faster than natural data at scale; the real gain comes from a ~30% synthetic / 70% natural mix, and fully-generated “textbook-style” synthetic data shows model-collapse-like degradation at scale that rephrase-type synthetic does not. For a tool-doc corpus this is more a caution than a direct prescription (the corpus is domain-narrow by construction, not a general-pretraining replacement) — but it argues against ever going 100% synthetic even within the narrow domain: keep the raw-doc-derived rungs 0-2 rows in the final mix alongside rungs 3-5, don’t fully replace them.

3. QA-format beats doc-format at the SFT stage

A frontier-scale (Gemini-1.5) ablation directly settles the “should I keep training on doc-shaped text at the SFT stage, or convert everything to QA” question: Zhao, Awasthi & Haghtalab (arXiv:2503.05919) find QA-formatted training rows generalize knowledge far better than document/article-style rows at the fine-tuning stage — the “task customization vs. knowledge injection” framing some prior work treated as separate problems turns out to be largely a data-format artifact once you control for it. The paper also flags two gotchas worth carrying into curation: numeric facts (rate limits, thread counts, timeouts) are harder to retain than categorical ones (flag names, source names) — budget more rephrasings for numeric args — and fine-tuned knowledge often fails to compose into multi-step reasoning even when the individual fact is retained cleanly, which is exactly why rung 4’s connective text is a separate rung, not automatic once rungs 1-3 are done.

Practical consequence for the doc-format-heavy stages (rung 0-1): convert to QA before spending rung 1’s paraphrase budget — don’t paraphrase in doc-prose form and hope the SFT stage handles the format shift for free.

4. Pre-screen before you spend curation budget

Not every candidate fact is worth the full rung-3/4 treatment — some are already known, some are genuinely absent, and running the expensive pipeline on both wastes budget and, per §5, actively risks damage on the already-known ones. Two complementary pre-screens:

Cheap, zero-generation: Gottesman & Geva’s KEEN probe (arXiv:2406.12673) trains a linear probe on the hidden state at the entity-mention position — zero tokens generated — and predicts downstream QA accuracy for that fact. Use it to triage which facts/tools are worth the expensive rung-3/4 pipeline before spending it. Caveat, load-bearing: Cheang et al. (arXiv:2510.09033) show internal-state probes like this mostly track the model’s recall confidence, not ground-truth correctness — a confidently-recalled but wrong association about amass’s flag set would score as “known” by KEEN alone. Pair it with a ground-truth check; never trust the probe by itself.

The ground-truth check itself — SliCK categorization, per Gekhman et al. (arXiv:2405.05904): score each candidate fact via 1 greedy (T=0) decode + 8-16 sampled (T=0.7-1.0) decodes against the gold answer, then bucket:

Run this on every atomic fact/argument/flag a tool-call trajectory row depends on, not just the trajectory’s final outcome — a trajectory can look “known” because the challenge was solved in the demo while individual intermediate steps reference Unknown facts, and training on the full trajectory teaches the model to project HighlyKnown-style confidence onto those Unknown steps too (the mirroring mechanism, §5).

5. The confabulation guard

Two convergent, causally-demonstrated results say the same thing from different angles: supervising a confident target on a fact the model doesn’t actually know teaches it to be confidently wrong elsewhere, not just on that fact.

• Gekhman et al. (arXiv:2405.05904) — SFT rows requiring genuinely new (“Unknown”) facts are fit markedly slower by gradient descent than knowledge-consistent rows, and once they are fit, they increase hallucination on other, unrelated, previously-known facts, approximately linearly in their fraction of the training mix. This is the direct causal mechanism behind “does trajectory-level SFT amplify an unfixed knowledge gap” — yes, and the damage isn’t contained to the new fact.
• Kang, Wallace & Levine (arXiv:2403.05612) — the shape of supervision given on unfamiliar training rows becomes the model’s default template for unfamiliar inputs generally, and this transfers to unseen unfamiliar queries at inference. If your Unknown rows are supervised with a confident, specific, made-up-looking answer, the model learns “when unsure, sound confident and specific” as a general policy — a training-induced version of the exact confabulation pattern The kinds of SFT §4 names for synthetically-authored tool trajectories, but arriving via the knowledge axis instead of the execution axis.
• Kaplan, Gekhman et al.’s 2026 follow-up (arXiv:2604.15574, very recent, single-lab, promising) localizes the mechanism further: SFT-induced hallucination is driven mainly by representational interference — a new fact about amass corrupts existing knowledge about semantically overlapping entities (other recon tools, subfinder itself) — not pure capacity limitation. The highest-risk rows are new facts about entities that sit close in representation space to well-known ones, which is precisely the amass/subfinder pair. Ghosal, Hashimoto & Raghunathan (arXiv:2406.14785) give the behavioral symptom: fine-tuning on lesser-known-but-true facts teaches the model to ignore the specific entity token and emit a generic plausible response instead, degrading downstream factuality 5-10% even when the correct value is still technically encoded in weights. Zucchet et al. (arXiv:2503.21676) independently find hallucination emerges simultaneously with new-fact learning in phases, not as a later side-effect — the damage isn’t a delayed cost, it happens as the fact is being learned. Dang et al. (arXiv:2511.02626) refine the risk model one more notch: it’s the unfamiliarity of an entire knowledge type, not the proportion of unfamiliar rows in the mix, that predicts damage — a curriculum implication.

The guard, concretely, per WeaklyKnown/Unknown row:

1. Relabel the target, not the row’s existence. Per Kang et al. and R-Tuning (arXiv:2311.09677), instead of a confident factual assertion, the supervised target becomes an explicit hedge (“I’m not certain of amass’s exact default source list — check amass enum -h before assuming”) or a tool-lookup deferral (emit a --help/doc-fetch call instead of a declarative answer). Wu et al.’s uncertainty-aware fine-tuning (arXiv:2502.11962) gives the other legitimate option: drop the row entirely rather than relabel, if a hedge target itself risks teaching over-hedging on genuinely knowable facts.
2. Add a self-distillation / KL regularizer against the pre-SFT policy, computed on a replay set of already-HighlyKnown prompts:

# per Kaplan/Gekhman 2604.15574 — guard against new-fact rows eroding adjacent known facts
loss = sft_loss(new_and_task_rows) \
     + lam * kl_divergence(policy(replay_prompts), frozen_pre_sft_policy(replay_prompts))
# replay_prompts = a held-out HighlyKnown set (per §4's SliCK bucket), NOT the new-fact rows themselves
# frozen_pre_sft_policy = a snapshot taken before this SFT run starts, never updated

3. Gate on a canary set before shipping. Hold out HighlyKnown pre-SFT facts (including some totally unrelated to the new material), re-run the SliCK probe post-SFT with the identical protocol. A drop here is the real hallucination signal — a required gate, not an optional ablation.

Build this: the confabulation guard isn’t optional polish on top of the data ladder — it’s the difference between “K-gap fixed” and “K-gap fixed, R-gap introduced on five adjacent tools you didn’t touch.” Run it on every Unknown/WeaklyKnown row, every time.

6. CPT-then-knowledge-SFT vs RAG — and where LoRA fits

The decision that actually matters before you write any row: does this knowledge belong in weights at all, or is a retrieval/tool-lookup path cheaper and safer? The literature is genuinely split, and it’s split along a specific, checkable axis — data format, not a law about fine-tuning itself.

• RAG wins, in the naive-FT regime. Ovadia et al. (arXiv:2312.05934): unsupervised fine-tuning on raw text consistently loses to RAG on knowledge-injection benchmarks. This is rung 0’s floor, restated as a decision rule: if your curation plan stops at “raw doc + fine-tune,” don’t bother — put the doc in a retrieval index instead.
• A heavily-augmented synthetic pipeline can beat RAG. Han et al.’s Synthetic Mixed Training (arXiv:2603.23562, very recent, promising) combines synthetic QA with synthetic docs (not QA alone) and reports beating a RAG oracle on parametric knowledge acquisition — but this is the rung 1-5 pipeline in this chapter, not naive rung-0 fine-tuning. The Zhao/Awasthi/Haghtalab QA-format result (§3, arXiv:2503.05919) is the likely reconciling mechanism, though neither paper states it explicitly: it’s not “fine-tuning vs RAG,” it’s “unsupervised raw-text fine-tuning vs RAG” that RAG wins, and “properly augmented QA-format fine-tuning vs RAG” is a much closer, workload-dependent fight.
• The theoretical argument for why tool-facts specifically should default toward tools, not weights. Houliston et al. (arXiv:2508.20755, promising, <1yr) prove in-weight factual memorization is provably parameter-bounded (this is the 2-bit/param ceiling from §1, restated as a design constraint) while in-tool/retrieval learning is provably unbounded. For a pretrained agent, teaching general tool-use rules beats fine-tuning specific facts into weights — this book’s “knowledge in tools, not weights” project rule, independently re-derived. The practical instantiation: the tool schema/doc block goes in every training row’s context (rung 3’s RAT design), never as the loss target; the loss stays masked to the decision (which tool, which of the valid args), never to memorizing the argument space as free-standing fact.
• CodeUpdateArena is the cleanest real-analogue triangulation. Liu et al. (arXiv:2407.06249) test the closest available proxy for “a tool’s flags changed”: fine-tuning a code LLM on an API update’s raw documentation text does not transfer into applying the update during generation; prepending the identical text in-context (RAG) does work; fine-tuning on curated usage examples (rung 3’s shape — exercising the update inside a solved problem) shows real improvement over docs-only fine-tuning. This triangulates cleanly across three independent lines: Physics 3.1’s augmentation requirement (§1), Gorilla’s RAT design (§2.3), and this chapter’s whole argument that doc-format alone is close to inert.

Where LoRA fits — contested, lean weak-at-injection. Salnikov’s direct test (arXiv:2502.14502, “How much knowledge can you pack into a LoRA adapter without harming the LLM?”) measures LoRA’s knowledge-injection ceiling directly and finds it materially below full fine-tuning’s — a sharper, injection-specific instrument than the general PEFT-forgetting trade-off The kinds of SFT §6 already covers (LoRA learns less, forgets less, per that chapter’s own citation ledger). Contested, per this survey’s own ledger: whether LoRA is categorically weak at knowledge injection, or whether that’s specific to naive low-rank adapters and newer subspace-targeted PEFT variants close the gap, is unresolved — treat “LoRA can’t inject knowledge” as an overstatement of a real but narrower effect, not an exclusion rule. If the constraint is “don’t overwrite unrelated capability while injecting this,” LoRA remains the right lever for the behavioral rungs (3-4) even if it underperforms full-FT for raw declarative-fact density.

The decision, compressed:

7. The full worked K-gap data row (production schema)

Once §4 has confirmed Unknown/WeaklyKnown and §6 has decided “this belongs in weights,” the per-fact row assembles every rung above into one object — this is the same shape the survey’s own operational recipes converge on independently across themes A/C/E; this chapter’s contribution is the why and the worked content at each field, not a novel schema:

{
  "fact_id": "amass-passive-ct-coverage-vs-subfinder-default",
  "source": "amass README §Passive Enumeration; subfinder --help SOURCE block",  // rung 0, never trained raw
  "slick_bucket": "Unknown",                                                    // §4 gate result
  "keen_score": 0.14,                                                           // §4 cheap pre-screen, corroborated by slick_bucket per 2510.09033's caveat
  "rephrasings": [                                                              // rung 1, ~20-32 per 2603.18534
    "amass enum -passive pulls CT-log and passive-DNS sources subfinder's default set doesn't include.",
    "subfinder's default flags query a fast source subset; -all widens it but still isn't identical to amass's passive-mode coverage.",
    "... (18-30 more diverse rewrites, packed into a megadocument per 2603.18534) ..."
  ],
  "directional_pairs": [                                                        // rung 2, per Reversal Curse 2309.12288
    {"forward": "What does amass's passive mode cover that subfinder's defaults miss?",
     "answer": "CT-log and passive-DNS sources."},
    {"reverse": "Which tool/mode do I need for CT-log coverage subfinder's defaults don't provide?",
     "answer": "amass, in passive mode (amass enum -passive)."}
  ],
  "tool_call_rows": [ /* rung 3 — see §2.3's full worked example; execution+semantic verified, APIGen-style */ ],
  "entity_pair_connectives": [ /* rung 4 — see §2.4's worked example, amass→subfinder chaining */ ],
  "unfamiliarity_flag": true,                                                   // from slick_bucket, corroborated not solely-probe-derived
  "supervision_target_mode": "confident",                                       // "confident" only because rungs 1-4 above cleared the bar;
                                                                                 // if still Unknown after curation, flip to "hedge" per §5
  "kl_regularizer": {"replay_set": "highly_known_canary_v3", "lambda": 0.1},    // §5's confabulation guard, always attached for Unknown-origin rows
  "post_sft_canary_check_required": true                                        // §5's gate — block ship without re-running SliCK on the canary set
}

8. Directly answering the three questions

• “Do I feed the README?” Once, raw, unaugmented — no. §1’s null result (arXiv:2309.14316) and §6’s RAG-vs-FT floor (arXiv:2312.05934) both say this buys close to nothing over a zero-training retrieval index, and can actively hurt QA-ability per AdaptLLM’s finding (§2.0, arXiv:2309.09530). The README is rung 0: source material for the pipeline, never a training target — if you’re not going to build the pipeline, put it in RAG instead of training on it.
• “Do I generate help pages?” Yes — but not as one help-page-shaped document per tool. Convert to §3’s QA format, generate the §2.1 paraphrase volume (~20-32/fact), the §2.2 forward+reverse pairs, and — the highest-value step for “wrong tool, wrong flags” specifically — the §2.3 execution-verified tool-CALL rows with the schema kept in context (RAT), not memorized as declarative text. Help-page content becomes many differently-shaped rows, most of which aren’t prose at all.
• “Do I make it memorize a tool list?” Only as many diversified rows, across many contexts and both query directions, covering the full argument-value range — never as one verbatim list. A single memorized list is exactly rung 0’s failure mode restated (one exposure, one wording) and specifically fails the “expert flag-set” symptom this chapter opened with: Greenstein et al. (§2.3, arXiv:2601.17829) show the entire prior tool-synthesis literature under-covered argument values, not just tool/query diversity — a memorized list of flag names with no rows exercising their actual valid ranges reproduces that exact gap.

9. Not a shortcut: knowledge editing (ROME/MEMIT) on tool facts

The obvious-looking cheap alternative to this whole ladder — “just patch the fact into the weights with a closed-form edit, skip the data pipeline” — is worth naming and ruling out explicitly, because it fails specifically on the tool/API-fact analogue rather than being generically weak. Yang et al.’s “Mirage of Model Editing” (arXiv:2502.11177) shows ROME/MEMIT’s reported ~96.8% editing success is inflated by teacher-forced evaluation (the ground-truth content and length leak into the “test”); under honest free-generation evaluation, success drops to 38.5%, and sequential edits collapse catastrophically by ~1000 edits. Chhetri, Siddique & Farooq (arXiv:2511.03182) run the direct empirical test on the closest available proxy for “a tool’s flag changed”: applying ROME/MEMIT/PMET/GRACE to code LLMs under controlled API-deprecation edits drops syntactic validity up to 86 percentage points and functional correctness up to 45pp, and correct adoption of the intended change occurs in only ~6% of passing generations — most “passes” are workarounds that avoid the edited fact entirely, not evidence the edit worked.

Reading this pattern (near-perfect on the probability/cloze probe, near-total failure on free generation) is itself diagnostically useful, independent of whether you’d ever ship an edit: if even a causally-traced, surgical weight edit can’t produce correct generation behavior for a fact, that’s evidence the underlying gap has an execution/policy shape layered on top of the knowledge shape, not pure knowledge absence — worth a note back to diagnosis before concluding the ladder above is even the right tool. Treat editing as a throwaway dev-time patch at most, never a production pipeline for “as tool flags change, patch them in” — the ladder in §2, however slower, is what a shipped fix runs on.

Cross-links

• Diagnosing a K-gap — the confirmation step this chapter assumes already passed: never start rung 1 without first clearing the in-context-oracle probe (fixed near-zero-shot when the fact is handed over) and the never-appears-at-any-N pass@k check.
• Diagnosing the gap — a scientific framework — the fuller instrument set (Pass@k / Pass@(k,T) / Cover@τ, the elicitation ladder) that chapter runs to produce the confirmed-K-gap verdict this chapter starts from.
• The kinds of SFT — it is the data, not the algorithm — general SFT data-shape taxonomy: §5.2 covers Knowledge/Q&A rows at survey depth, §6 covers data-selection/curation generally, §4 names the synthetic-trajectory confabulation risk §5 above extends to the knowledge axis. This chapter is the deep, K-gap-specific expansion of that §5.2 cell.
• Does trajectory SFT amplify an unfixed K-gap? — the full causal chain behind §1’s corollary and §5’s guard; this chapter cites the load-bearing papers, that one is the dedicated treatment.
• Generic data-selection methods — IFD/LESS/DEITA-style selection applies within whichever rung’s row pool you’ve generated here, orthogonal to which rung you picked.
• The decision — the one-line routing tree this gaps/ series hangs off of; “knowledge gap → inject off-policy” is the branch this chapter fully unpacks.
• Contested edges & landmines — the broader LoRA-vs-full-FT and RAG-vs-FT disputes referenced in §6.

Bibliography

Confidence calibration: every id above verified live against arxiv.org/abs/<id> in the source survey pass (artifacts/three-gap-survey/ledger-{A,C,E}.md, 2026-07-02) — none recalled from training memory, and none used here beyond what those ledgers confirm. High-confidence, widely-replicated: the storage/extraction split (2309.14316, 2404.05405), the Reversal Curse (2309.12288), the RAT tool-call mechanism (2305.15334), and the new-fact-SFT-raises-hallucination causal chain (2405.05904, 2403.05612). Flagged “very recent / promising, not yet independently validated” throughout: 2603.18534 (megadoc rephrase saturation), 2601.17829 (argument-value diversity gap), 2603.23562 (synthetic-beats-RAG), 2604.04943 (bidirectional dual-representation), 2604.15574 (representational-interference mechanism), 2510.10681 (RePro faithfulness). LoRA-weak-at-injection (§6) is stated as contested on purpose — see Contested edges & landmines’s broader PEFT-forgetting thread, and The kinds of SFT §6’s own citation of the same trade-off from the forgetting angle rather than the injection angle. Explicitly not grounded on any academic cybersecurity-LLM paper, per this chapter’s brief — the amass/subfinder example throughout is illustrative scaffolding, not a claim about either tool’s actual documented flag set.

Does jumping to trajectory SFT amplify an unfixed gap? (yes — here’s the chain)

The three gaps — overview teased this in three sentences off a single citation. This chapter is the full evidence chain. The kinds of SFT §4 named the mechanism at the level of one row — a synthetically-authored tool result is fiction that only looks like grounding, and training on it teaches the model to imitate the shape of a tool result rather than react to the real one. This chapter answers the harder question underneath that: even a genuinely executed, verifier-filtered trajectory — Axis B and Axis C of that taxonomy both clean, no synthetic authoring anywhere — can still be built on turns the base policy never actually had the knowledge or the prior to produce reliably. Does SFT-ing on that trajectory anyway make the fabrication problem worse, not better? Data mixing & forgetting already showed off-policy trajectory data can silently erase an existing behavior (CoT-emission); this chapter is the mirror case — off-policy trajectory data silently installing a bad one.

The question this chapter answers

The question this chapter answers: the user’s own words were “everyone jumps to the trajectory [SFT] safety but these are the things we have to fix [first] — maybe this has been amplifying the problem at the trajectory level.” Stated precisely: if the harness runs SFT on full agentic run-logs before diagnosing whether the model’s failures are a knowledge/prior gap (K/R) rather than a policy/ranking gap (P) — does that trajectory-level SFT actively amplify the unfixed gap, rather than sit neutrally on top of it?

BLUF: yes — and it’s over-determined, not a hunch. Nine independent 2024–2026 results, from three separate research lineages that don’t cite each other as their primary motivation, converge on the same mechanism. None of them ran the experiment on your setup — CTF trajectories, a security-tool action space, a ~10–30B dense base — so the honest label is strongly-inferred, not proven by one controlled trajectory-level ablation. As of a May-2026 preprint there is now one direct agentic-setting test that corroborates the mechanism on a rhyming failure mode (§4 below); until more replicate it, treat this chapter’s verdict the way the rest of this book treats a well-supported but unreplicated claim — actionable, not gospel.

1. The hypothesis, restated — and the running example

Restated as a testable claim: SFT on a trajectory row is not a neutral operation with respect to a K/R gap that row happens to touch. If the demonstrated action at some turn required knowledge or a prior the base policy doesn’t reliably have, training on that row doesn’t just fail to help — it teaches the model a new, confident default for what to do the next time it’s in an unfamiliar-feeling state, and that default is “emit something that looks like the demonstrated action,” not “flag the uncertainty.”

Running example, held constant through this chapter: you SFT on a verified-flag-passing trajectory where turn 6 is amass enum -active -brute -d target.com. Suppose the base policy doesn’t actually know why -active and -brute are the right flags here versus -passive — it produced this call once, in one run, possibly because a stronger teacher demonstrated it, or because the model got lucky at temperature. The row is executed, real, verifier-passed — clean on every axis kinds-of-sft.md names. It still teaches confabulation, for reasons that have nothing to do with whether the row was synthetically authored.

2. Why this has to be a chain, not one citation

None of the individual results below was designed to answer “does SFT on an agentic trajectory amplify an unfixed K/R gap.” They were mostly run on single-turn factual QA (does the model know a fact) or short math/code reasoning. The chain works because each result independently rules out an escape hatch the previous one leaves open:

1. Maybe hallucination from new-fact SFT is a one-off artifact of the specific fact being trained → §3 shows it’s a general, linear, measured effect.
2. Maybe it only shows up after many epochs of overfitting → §4 shows it emerges as the fact is learned, not as a late-stage failure.
3. Maybe it only matters if a large fraction of your corpus is unfamiliar → §5 shows it’s the concentration within one type, not the overall percentage, that drives damage — a small, targeted corpus of trajectories about one tool family is exactly the danger shape.
4. Maybe this is confined to single QA rows and doesn’t apply to a 40-turn agentic trajectory → §6–§7 show the mechanism is worse, not the same, once you add a horizon: DAgger’s compounding-error theorem plus the trajectory-specific “fork in the road” result.
5. Maybe SFT doesn’t even touch real capability — LIMA says it’s “mostly format” → §8 shows why that reading, taken correctly, doesn’t rescue trajectory SFT; it explains why the fabrication default is cheap to install, not why it’s harmless.
6. Maybe a later RL stage cleans this up automatically → §9–§11 show RL’s relationship to what SFT installs is asymmetric, sometimes actively can’t reach it, and — critically — SFT overtraining makes the later RL stage’s job structurally harder, not easier.
7. Maybe this is armchair theory with no frontier-lab corroboration → §12 shows DeepSeek-R1’s own published recipe needed a second SFT pass gated behind RL for exactly this reason.
8. Maybe none of this generalizes past factual QA into the agentic/tool-use setting at all → §13 is the first direct test in that setting, and it corroborates a rhyming failure mode.

Every rung removes one plausible objection. That’s what “over-determined” means here — not that one paper proved it, but that the space of alternative explanations keeps shrinking.

3. Link 1 — new-fact SFT is learned slower, and once learned, poisons other known facts

Gekhman et al., “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?” (arXiv:2405.05904, EMNLP 2024, Technion/Google) is the base rate this whole chain sits on. Their SliCK protocol buckets each candidate SFT row by how well the base model already knows it (sample it 10–16 times across varied few-shot framings; HighlyKnown / MaybeKnown / WeaklyKnown / Unknown). Two findings, both measured, not argued:

• Rows in the Unknown bucket are fit by gradient descent markedly slower than rows the model already half-knows — the loss curve visibly lags.
• Once those Unknown rows are fit, hallucination on other, unrelated, already-known facts rises roughly linearly with the number of Unknown rows in the fine-tuning set.

This is the base-rate cost of an unfixed K-gap sitting in your trajectory corpus. It’s not that the amass row itself gets confabulated more — it’s that fitting that one row the model doesn’t actually know measurably degrades the model’s reliability on facts it already had right, elsewhere in the corpus. A trajectory-SFT set built without a per-row familiarity check pays this tax silently across the whole set, not just on the rows that look risky.

(Note on verification: this id shows one WRONG_ID flag in the underlying survey’s ledger alongside four independent CONFIRMED passes matching author/venue/date; treated as an isolated tool-search miss, not a real mismatch — see the bibliography note.)

4. Link 2 — hallucination doesn’t lag the new fact, it emerges with it

Zucchet, Bornschein, Chan, Lampinen, Pascanu & De (Google DeepMind), “How do language models learn facts? Dynamics, curricula and hallucinations” (arXiv:2503.21676) sharpens §3’s timing: hallucination isn’t a late-training side effect that shows up after the new fact is over-learned — it emerges in the same phase as the new-fact acquisition, and continued fine-tuning without curriculum management actively corrupts existing memories, not just fails to add new ones.

For a trajectory pipeline, this closes off the “we’ll just stop training early, before the damage phase” escape hatch — there is no clean early-training regime where the new-fact row is learned and nearby existing knowledge is still intact. The corruption is concurrent with the learning, not a downstream overfitting artifact you can checkpoint your way around by watching a single held-out loss curve. (Confidence: medium — a DeepMind dynamics study, verified live, not yet independently replicated at time of writing; the qualitative timing claim, not exact numbers, is what this chapter leans on.)

5. Link 3 — it’s the type of unfamiliarity that predicts damage, not the percentage

Dang, Hu, Lai, Gao, Zhang & Huang (Nanjing University/Huawei), “Understanding New-Knowledge-Induced Factual Hallucinations in LLMs” (arXiv:2511.02626, ACL Findings 2026) is the finding that reframes “just keep the unfamiliar fraction under X%” as the wrong lever. What predicts damage is concentration of unfamiliarity within one knowledge type, not the overall proportion of Unknown rows in the corpus — and mechanistically, learning the new knowledge measurably weakens attention to the question’s key entities, with the disruption propagating to lexically-similar contexts.

Translate this directly to the running example: a trajectory corpus doesn’t need to be mostly unfamiliar to be dangerous. If every amass-flag row in your corpus shares the same underlying unfamiliarity — the model never really learned what -active/-brute/-passive individually do — that’s concentrated unfamiliarity within one knowledge type (recon-tool argument semantics), even if the rest of a 5,000-row corpus is perfectly clean. The damage signature is exactly the “grabs the one salient tool, uses default args, universal across models” pattern: attention to the specific argument entities weakens, and the model substitutes a generic, lexically-similar-but-wrong default.

(Note: this is genuinely contested against §3 — the source survey itself flags Gekhman’s “linear-in- fraction” reading and Dang’s “concentration-within-type” reading as two measured effects that disagree on the dominant causal variable; don’t treat “keep Unknown rows under X%” as a validated universal threshold, and don’t treat “type concentration” as the sole driver either. Both are real; which dominates in your data isn’t settled by either paper alone.)

6. Link 4 — the mechanism: what you demonstrate at an uncertain point becomes the model’s new default

Kang, Wallace, Tomlin, Kumar & Levine (UC Berkeley/DeepMind), “Unfamiliar Finetuning Examples Control How Language Models Hallucinate” (arXiv:2403.05612) is the mechanistic result that makes §3–§5 actionable rather than just alarming: on unfamiliar test-time inputs, a model’s hallucinated output mirrors the aggregate label distribution it was shown for similarly-unfamiliar SFT rows during training. The hallucinated answer isn’t noise — it’s the loss-minimizing generalization of your own curation choice at the moment you wrote that row.

This is the direct causal story behind “confident fabrication of tool behavior/outputs” in the running example. If your amass-flag trajectory row demonstrates a confident, specific, un-hedged invocation at a decision point the demonstrator (teacher model, human, or your own policy at temperature) was itself uncertain about, Kang’s mechanism predicts the model doesn’t learn “sometimes guess -active -brute” — it learns “when I’m in a state that feels like this one, emit a confident specific answer,” and that policy generalizes to structurally similar turn-N states across different challenges, not just to a repeat of this exact one.

The fix this result licenses is a data-curation move, not a training-recipe change: relabel rows a per-row familiarity probe flags Weakly-Known/Unknown so the target is an explicit hedge/verify-first/ tool-lookup action instead of a confident specific one — “run amass --help and check which mode fits” rather than a bare, un-hedged -active -brute -d target.com. Whatever you demonstrate at the uncertain point becomes the deployed default; demonstrating verification-seeking rather than confident-guessing directly controls what that default is. (Full recipe: Knowledge curation.)

7. Link 5 — why trajectories make this categorically worse than single QA rows

Everything above was measured on single-turn factual QA. A 40-turn CTF trajectory is not a bigger version of the same problem — it’s a structurally different one, for two independent reasons.

7.1 DAgger’s compounding-error theorem

Ross, Gordon & Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning” (AISTATS 2011, arXiv:1011.0686), Thm 2.1: a policy trained by naive behavioral cloning on a fixed, off-policy demonstrator distribution incurs total cost bounded by J(π̂) ≤ J(π*) + ε·T² — quadratic in horizon T — because the moment the learner’s own (even small) deviation pushes it off the demonstrator’s training-state distribution, there is no training signal there for the rest of the episode. Their correction, DAgger, aggregates data from the learner’s own induced state distribution and restores near-linear O(ε·T) regret.

graph LR
  A["Single QA row:<br/>one wrong answer,<br/>bounded damage"] -.->|"horizon T=1"| B["Cost ~ ε"]
  C["40-turn trajectory:<br/>one wrong turn early,<br/>zero signal for recovery"] -.->|"horizon T=40,<br/>off-policy demo"| D["Cost ~ ε·T²<br/>(DAgger bound,<br/>1011.0686)"]
  style D fill:#3a1414,stroke:#e74c3c,color:#fbeaea

A single mis-demonstrated fact in a QA row costs you one wrong answer. A single mis-demonstrated turn in a 40-turn trajectory — say turn 6’s amass invocation, if it was itself the demonstrator guessing — costs you the entire rest of the episode’s worth of training signal once the model’s own drift departs from the exact state the demonstration assumed, because off-policy trajectory data is, by construction, data about states the model itself won’t visit once it starts generating.

7.2 The token-level sibling, and the trajectory-specific “fork” result

The same phenomenon restated at the token level, for purely-supervised sequence models: Bengio, Vinyals, Jaitly & Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks” (arXiv:1506.03099) — exposure bias, the mismatch between teacher-forced training and free-running inference. A recent survey states this plainly for the LLM case: static off-policy SFT “is an instance of exposure bias” scaling “roughly with the square of sequence length” (Song & Zheng, A Survey of On-Policy Distillation for Large Language Models, arXiv:2604.00626).

On top of exposure bias, trajectories have more places to go wrong than QA data does, structurally. Nguyen, Shojaee et al., “Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road” (arXiv:2605.17026) show that wherever training data commits to one canonical path through a decision point that had multiple valid strategies — a “fork” — cross-entropy forces hard commitment there, and measured pass@k shrinkage tracks fork prevalence in the data, not data volume. A trajectory is a chain of forks: which tool first, which flag style, verify-before-acting or assume-and-proceed. Single-turn QA data has comparatively few forks per row; trajectory data is made of them. Corroborating measurements: pass@1 rises monotonically through SFT while pass@k crashes rapidly, and weight-interpolating back toward a pre-crash checkpoint recovers most of the lost coverage (Dang et al., WiSE-FT, arXiv:2504.10478); for CoT-distillation lineages specifically, most semantic diversity is lost at the SFT step itself — more than at any later DPO/RL step — and it’s baked into the weights by training-data composition, not fixable by decoding-time tricks (Karouzos, Tan & Aletras, arXiv:2604.16027).

Why this matters for the diagnosis, not just the mechanism: if a challenge’s winning path is sequentially gated (enumeration must land before exploitation is even reachable), a trajectory-SFT row for it doesn’t just risk one bad fact — it locks in one canonical fork through a decision tree with several defensible branches, at exactly the depth where the model’s own on-policy drift is likeliest to have already left the demonstrated state. This is the same “enumeration-gates-exploitation” shape Diagnosing the gap §2.4 already flags as the place execution and exploration gaps are hardest to tell apart — here it’s the place trajectory SFT does the most damage per row.

8. Link 6 — doesn’t SFT “only teach format”? Why the Superficial Alignment Hypothesis doesn’t rescue this

If LIMA’s strong claim were the whole story, this chapter’s verdict would dissolve: Zhou et al., “LIMA: Less Is More for Alignment” (arXiv:2305.11206) argues most of a model’s knowledge comes from pretraining and SFT mainly teaches format/style — which sample of its own behavior to surface. Read naively, that would mean trajectory SFT can’t inject a genuine knowledge gap at all; it just reshapes presentation.

Two things block that escape hatch. First, the strong-form hypothesis is itself contested in the general literature — Raghavendra, Nath & Hendryx, “Revisiting the Superficial Alignment Hypothesis” (arXiv:2410.03717) find post-training performance scales as a power law in the number of SFT examples on math/coding/multihop-QA well past LIMA’s ~1,000-example regime — style-only alignment does not saturate task performance for reasoning-heavy domains, and tool-use trajectory data is squarely reasoning-heavy, not stylistic preference data. Second, and more important even if you grant SAH’s weaker, correct reading: “format” is exactly where the danger lives. “Always commit to a confident specific action rather than hedge” is a format/policy choice in the SAH sense — a coarse, low-information-content behavioral switch, not new declarative knowledge — and Data mixing & forgetting already established that this class of switch is learnable (and un-learnable) from remarkably little data. LIMA’s own finding — quality format shift from ~1,000 curated rows — is a two-edged fact: it means the harmful direction (confident-guessing-as-default, per §6’s mechanism) is exactly as cheap to install as the beneficial one. SAH doesn’t get trajectory SFT off the hook; it explains why the fabrication default is inexpensive to bake in.

9. Link 7 — training destroys the alternative paths that would have hedged the fabrication

Once a corpus has installed a confident-default at an uncertain decision point (§6), what happens to the other, more honest paths through that same decision point — the ones where a stronger policy would have checked --help first, or flagged low confidence? Two results say: they get trained away, on the SFT step specifically, faster than anywhere else in the pipeline.

• Reasoning-Trace Collapse — Twist, Yannakoudakis & Zhang (King’s College London), arXiv:2605.21127: fine-tuning on ordinary instruction-response data containing no reasoning trace induces a model to stop emitting the deliberation that would have surfaced uncertainty — it minimizes loss by treating “no hedge, no check, straight to the confident answer” as the target behavior, and answer-only accuracy monitoring hides this until it’s severe (a model can be right on the final token while never emitting the verification step that made it reliable). (Confidence: medium — brand-new 2026 preprint, multi-model, not yet independently replicated.)
• On-Policy Self-Distillation Reduces Output Diversity — Nicolicioiu, Pezeshki & Courville, arXiv:2606.26091: even the gentler, self-distillation-based correction to naive off-policy SFT tilts the policy by a pointwise conditional-mutual-information term that amplifies pre-existing probability gaps, flattening pass@k more than an ideal on-policy RL update would — i.e. the failure mode isn’t confined to the crudest off-policy SFT recipe; even the more careful on-policy variants can narrow the distribution of paths the model still considers.

Put together with §7.2’s fork-collapse result, the picture is: trajectory SFT doesn’t just install one bad default at an uncertain decision point — it simultaneously narrows the alternative, hedging paths that would have provided a fallback. The confident-fabrication behavior isn’t competing against a healthy distribution of more cautious behaviors after training; it’s increasingly the only behavior left.

10. Link 8 — the trajectory’s own stated reasoning may not even be the real cause

A subtler compounding factor, orthogonal to the tool-call itself: if your trajectory rows include a reasoning/rationale block before the action (§5.4 of kinds-of-sft.md), that reasoning is not guaranteed to be causally what produced the action. Turpin, Michael, Perez & Bowman, “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” (arXiv:2305.04388) show LM-generated CoT can be systematically unfaithful — a plausible-sounding justification generated after the fact, not the actual driver of the answer.

For trajectory SFT specifically: if a demonstrator’s stated reasoning for choosing -active -brute was itself post-hoc rationalization (plausible-sounding, not actually load-bearing) rather than genuine tool-semantics understanding, training on that row teaches the model to produce equally plausible-sounding, equally ungrounded justifications for its own future fabricated calls. This compounds §6’s mechanism one layer up: the model doesn’t just learn a confident action default at unfamiliar states, it learns a confident explanatory style to go with it — which makes the fabrication harder to catch by reading the model’s own stated reasoning, exactly the failure mode a human reviewer skimming trajectory logs would miss.

11. Link 9 — the SFT→RL handoff propagates, and can harden, whatever the chain above installed

Even granting a subsequent RL stage, four results say “RL will clean it up” is not a safe assumption — and one says the opposite can happen.

• PEAR — Zhang, Xu, Wang, Chen & Peng, “Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning” (arXiv:2602.01058): SFT-checkpoint quality measured on SFT’s own held-out loss does not predict post-RL performance — a checkpoint that looks better by that metric, because it fit more off-policy expert data, can underperform post-RL relative to a weaker-looking one, because the off-policy behavior-policy distribution diverges from what RL’s on-policy target needs to build on.
• Quagmires in SFT-RL Post-Training — Kang, Kuchnik, Padthe, Vlastelica, Jia, Wu & Ardalani (FAIR/Meta + Virginia Tech), arXiv:2510.01624 (>1M GPU-hours, hundreds of models to 12B): high SFT-stage scores are not reliably predictive of eventual RL gains — sometimes inversely so. What does predict post-RL pass@1, with roughly 2× better R²/Spearman correlation than post-SFT pass@1 alone: generalization loss on held-out examples and pass@large-k on the post-SFT checkpoint. A trajectory-SFT checkpoint that fabricates confidently on unfamiliar states will look fine on its own training-distribution accuracy while quietly failing this held-out signal.
• RL’s Razor — Shenfeld, Damani, Hübotter & Agrawal (MIT, ICLR 2026 poster), arXiv:2509.04259: forgetting after fine-tuning is quantitatively predicted by E_{x~new-task}[KL(π_base ‖ π_finetuned)]. Off-policy SFT minimizes forward KL (mode-covering — it matches the demonstrated trajectory’s exact phrasing even where a path closer to the base model’s own distribution would have worked, and in doing so can overwrite an existing good low-probability mode), whereas on-policy RL is implicitly biased toward the KL-minimal solution among all reward-maximizing policies. Corroborated independently by Chen, Razin, Narasimhan & Chen (Princeton), “Retaining by Doing” (arXiv:2510.18874): SFT’s forward-KL mode-covering can overwrite an existing behavior mode; RL’s reverse-KL mode-seeking tends to add a new mode without disturbing the old one’s shape — and “approximately on-policy” trajectory data recovers most of RL’s forgetting-resistance cheaply, i.e. the fix is upstream of RL, in how the SFT data was sourced.
• The non-decoupling theorem — Niu, Bai, Han & Zhang (Huawei), “On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training” (arXiv:2601.07389): formally, SFT and RL cannot be cleanly separated in either insertion order — SFT-then-RL provably increases the SFT loss (some of what SFT taught erodes), and RL-then-SFT provably lowers RL’s achieved reward. There is no ordering where the gains from one stage are safe from the next — which is the formal reason a one-time “gate trajectory SFT, then move on” plan is insufficient; the erosion (or lack of it) has to be re-measured after every stage, not assumed from the ordering alone.
• When RL Fails after SFT — Liu, Liu, Wan, Fu & Pan (HKUST), arXiv:2606.09932: excessive SFT produces over-confident (low-entropy) token distributions and sharper loss landscapes that are measurably harder for a subsequent RL stage to reshape — a genuine plasticity-loss mechanism. This traces to the general Primacy Bias precedent in deep RL (Nikishin et al., ICML 2022, arXiv:2205.07802: early experience locks in and resists later correction unless part of the network is reset). Read together with §6–§9: the earlier a confident-fabrication default gets baked in by trajectory SFT, and the more aggressively SFT converges on it, the harder a later RL stage will find it to dislodge — “SFT now, RL fixes it later” inverts the actual difficulty gradient.

12. The corroboration: DeepSeek-R1 needed a second SFT stage after RL

DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” (arXiv:2501.12948) is not primarily a paper about hallucination or trajectory SFT — but its own disclosed pipeline is a frontier lab conceding exactly this chapter’s point in practice. R1-Zero (pure RL, zero SFT) develops strong reasoning but with poor readability/language-mixing; the added cold-start SFT stage exists specifically to fix format/legibility, not to teach reasoning — and critically, that isn’t where the pipeline stops. After RLVR, DeepSeek runs rejection sampling on the RL checkpoint to build ~600K new, verified SFT rows, then a second SFT/RL pass. A lab with effectively unlimited compute and the strongest incentive in the industry to ship a one-shot cold-start-SFT-then-RL recipe did not trust a single trajectory-adjacent SFT pass — it gated with RL, re-verified with rejection sampling, and iterated. That’s the “Stage-0 diagnosis, Stage-3 re-verification” discipline this chapter ends on, independently arrived at by the team that produced the best-known open frontier RL recipe.

13. The closest thing to a direct test — and it maps onto the running example almost exactly

Everything in §3–§11 was measured on single-turn factual QA or short reasoning traces, not multi-turn agentic tool use. That gap is real, and the honest caveat is in §14. But one very recent result closes most of it directly. Gu et al., “What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?” (arXiv:2602.01611) ran the agentic-setting version of this exact question across 16 AgentBench/AgentGym environments: trajectory-SFT’d agents substantially amplify reliance on the training-time tool/interface surface form — they collapse under semantics-preserving interface rewrites (renamed arguments, reordered parameters, aliased tool names, swapped JSON key order), while non-trajectory-SFT’d (few-shot-prompted) baselines stay stable under the identical rewrites.

This is the runnable version of the running example. Take the trajectory-SFT’d checkpoint and rewrite the amass tool schema with semantics-preserving changes only — -active/-brute renamed to --enable-active-recon/--bruteforce-subdomains, or the argument order shuffled. The required action is unchanged; only its surface form moved. If pass@1 collapses under that rewrite while a prompted-but-not- trajectory-SFT’d baseline stays flat, that’s direct confirmation the model locked in an interface shortcut — surface-form mimicry — rather than the underlying tool semantics, precisely because the knowledge/prior wasn’t secure before trajectory SFT ran. This is Protocol 7 (interface-perturbation probe) in Diagnosing which gap — the cheapest test that would falsify or confirm this chapter’s verdict on your own checkpoint.

flowchart TD
  A["Single-fact SFT rows:<br/>learned slower, poisons<br/>known facts linearly<br/>(2405.05904)"] --> B["Hallucination emerges<br/>WITH the new fact,<br/>not after (2503.21676)"]
  B --> C["Driven by concentration<br/>within one knowledge TYPE,<br/>not overall % (2511.02626)"]
  C --> D["Mechanism: unfamiliar-row<br/>demo becomes the model's<br/>new DEFAULT (2403.05612)"]
  D --> E["Trajectories make this WORSE:<br/>DAgger compounding O(εT²)<br/>+ fork-in-the-road collapse<br/>(1011.0686, 2605.17026)"]
  E --> F["SAH doesn't rescue it:<br/>'format' IS the confident-<br/>default switch (2305.11206)"]
  F --> G["Training destroys the<br/>hedging alternatives<br/>(2605.21127, 2606.26091)"]
  G --> H["SFT->RL handoff can HARDEN,<br/>not clean up, the default<br/>(2602.01058, 2510.01624,<br/>2509.04259, 2606.09932)"]
  H --> I["DeepSeek-R1's own recipe:<br/>gate + re-verify, twice<br/>(2501.12948)"]
  I --> J["Direct agentic test:<br/>interface-rewrite collapse<br/>(2602.01611)"]
  J --> V["VERDICT: yes, amplifies —<br/>strongly-inferred chain,<br/>one direct corroboration"]

  classDef verdict fill:#132b22,stroke:#34d399,color:#eafaf3;
  class V verdict;

14. Honest counter-considerations — what would weaken this verdict

Three genuine open questions, stated the way the rest of this book states contested ground:

• Does the QA-literature mechanism actually transfer to agentic tool-use, or is it a strong analogy? §3–§6, §9–§10’s load-bearing evidence was measured on single-turn factual QA. §13’s Interface Reliance study is the first direct agentic-setting test, and it corroborates a rhyming but distinct failure mode — surface-interface shortcutting, not factual hallucination per se. Treat the QA mechanisms as a strong, well-evidenced analogy to the agentic case, not a fully validated transfer, until more direct agentic ablations replicate it.
• Is the O(T²ε) compounding bound a hard law, or avoidable in a “recoverable” environment? Foster, Block & Misra, “Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning” (arXiv:2407.15007) show the quadratic-in-horizon bound is avoidable under bounded-coverage/well-specified-policy-class/self-correcting-MDP assumptions. A CTF harness with informative tool stderr and a shell the agent can re-probe may be more “recoverable” than the worst-case bound assumes — whether it’s recoverable enough to matter is genuinely open and task-dependent, not something to assume either way without measuring it on your own trace corpus.
• What’s the actual mechanism — interference or storage strength? Kaplan/Gekhman et al.’s follow-up, “Why Fine-Tuning Encourages Hallucinations and How to Fix It” (arXiv:2604.15574), find representation interference among overlapping weights dominates over raw capacity limits, proposing self-distillation SFT (KL-regularize against a demonstration-conditioned self-teacher) as the fix. Ghosal, Hashimoto & Raghunathan, “Understanding Finetuning for Factual Knowledge Extraction” (arXiv:2406.14785) instead find it’s fact-storage strength — even “known” facts poorly stored get the subject entity ignored and a generic response substituted. Both are measured, real effects; they haven’t been unified, and which one dominates in your action space isn’t settled by either paper alone.
• The rebuttal to the sharpest weight-level framing. §11’s “SFT broadly replaces, RL narrowly amplifies” claim from Rajani et al., “Scalpel vs. Hammer” (arXiv:2507.10616) is a single, 0-citation preprint whose own authors call it a “preliminary indication” — and it’s directly contested by Jin, Luan, Lyu et al. (Mila), “RL Fine-Tuning Heals OOD Forgetting in SFT” (arXiv:2509.12235), who show OOD performance during SFT actually peaks early then declines with continued training, and RL does not exceed that early peak — it only restores it, from a bounded checkpoint range. Neither settles whether RL does something SFT categorically cannot, or is compensating for a controllable SFT overfitting/checkpoint-selection failure. This chapter treats “Scalpel vs. Hammer” as supporting color, per Contested edges §1 — not load-bearing on its own.

None of these three weaken the core mechanism (§3–§6, §9–§10); they bound how confidently you can port the exact numbers, and how much you should trust the O(T²ε) worst case as your operating assumption versus a measured floor for your own harness.

15. The practical rule

Gate trajectory SFT behind a diagnosis; don’t skip to it because it’s the intuitive first move.

1. Stage 0 — diagnose, per row and per challenge subtype, before any trajectory enters the corpus. Run the K/R/P probe battery (Diagnosing which gap) — SliCK familiarity sampling, the in-context oracle probe, teacher-forced-vs-free-running divergence. A row whose critical turn sits in the Weakly-Known/Unknown bucket is not safe to SFT as a confident, un-hedged target (§6).
2. Curate declarative knowledge/prior FIRST. If Stage 0 finds a genuine K-gap, close it with QA-format knowledge injection (Knowledge curation) — many independently-phrased rephrasings, weighted toward numeric/parameter-value facts since those retain worse under fine-tuning (Zhao, Awasthi & Haghtalab, NeurIPS 2025 poster, arXiv:2503.05919) — gated by a held-out re-probe, not inferred from downstream trajectory success alone, since fine-tuned facts don’t automatically transfer into correct multi-step use of that fact.
3. Trajectory SFT LAST, light, capped, and preferentially near-on-policy. Generate trajectories after Stage 2 closes the K-gap, so demonstrations draw on secured knowledge rather than a demonstrator’s own guess. Prefer self-generated/rejection-sampled rollouts over a distant teacher’s raw transcript — per §11’s PEAR finding, an off-policy-trained SFT checkpoint that looks better on its own metric can underperform post-RL. Cap training by tracking entropy and held-out generalization loss during training, not lowest SFT training loss (§4, §9, §11).
4. Stage 3 — re-verify, and re-verify again after RL. Run the interface-perturbation probe (§13) before calling trajectory SFT done. Then run it again after any subsequent RL stage — the non-decoupling theorem (§11) means erosion is bidirectional and ongoing, not a one-time Stage-1 mistake fixed once and forgotten.

Designed to fix exactly the failure the user named: “everyone jumps to the trajectory [SFT] safety.” The jump itself is the risk — not trajectory SFT as a method (it remains the right eventual tool, per FireAct/AgentTuning), but skipping the diagnosis that tells you whether a given row’s critical turns are safe to train on confidently. The fix isn’t “never do trajectory SFT” — it’s “diagnose, curate knowledge/prior first, then trajectory SFT last, then keep re-checking.”

Cross-links

• The three gaps — overview — the K/R/P taxonomy this chapter’s verdict routes into; the three-sentence teaser this chapter fully substantiates.
• Diagnosing which gap — the Stage-0 probe battery (SliCK, in-context oracle, teacher- forced-vs-free-running divergence, interface-perturbation) this chapter’s practical rule depends on.
• What knowledge data looks like — the Stage-2 K-gap fix: QA-format rephrasing, the relabel-to-hedge mitigation from §6, gated by a held-out re-probe.
• Matching the fix to the gap — the full staged curriculum (K→inject, R→re-prior, P→re-rank) this chapter’s §15 rule is the trajectory-SFT slice of.
• The kinds of SFT §4 — the sibling mechanism at the single-row level (synthetic authoring → confabulation); this chapter is what happens even when authoring is clean but the underlying knowledge isn’t secured.
• Data mixing & forgetting — the mirror failure: off-policy trajectory data erasing an existing behavior (CoT-emission) rather than installing a bad new one; §1.5’s “format is cheap to (un)learn” finding is reused directly in §8 above.
• The one axis that predicts everything — the DAgger/off-policy-blindness theorem this chapter’s §7.1 is the trajectory-horizon instance of.
• Diagnosing the gap — a scientific framework — the complementary knowledge/execution/exploration lens; §2.4’s compositional-segmentation test is the same “fork density” concern §7.2 raises from the SFT-data side.
• Contested edges & landmines §1 — the fuller treatment of the “Scalpel vs. Hammer” contested citation used in §14.

Bibliography

Standing rule (matches the overview and every chapter in this section): no load-bearing claim above rests on an academic cybersecurity-LLM training/benchmark paper — every citation is general ML/RL theory or frontier-lab evidence, verified live against arxiv.org/abs/<id> in the underlying survey pass (2026-07-02). The amass/recon-flag running example is the motivation for reading this chain, not the evidentiary basis for it.

Matching the fix to the gap — K→inject, R→re-prior, P→re-rank

The question this chapter answers: once you’ve diagnosed which gap you have, which technique do you actually run, and what does the training row look like? BLUF: K (knowledge) needs off-policy injection — CPT + knowledge-SFT, never trajectory imitation. R (the default/prior) needs a small, off-policy, format-only cold-start SFT that re-shapes what the policy samples first, not what it’s capable of. P (ranking) needs an on-policy preference pass — DPO/KTO or a short GRPO run — localized at the exact state where the model already produces the right action but doesn’t prefer it. These are not interchangeable: pointing a K-shaped fix (raw documents) at an R-gap teaches facts nobody asked for; pointing an R-shaped fix (cold-start SFT) at a P-gap wastes a training run reinforcing a ranking the model can already produce on its own, off-policy, at the wrong dose. Diagnosing the gap — a scientific framework gives you the instruments (pass@k, the elicitation ladder, Pass@(k,T)) that tell you which gap you have; the fuller K/R/P diagnostic battery — pass-at-k sweep, in-context oracle probe, teacher-forced logprob spans, the reversal check, the purpose-built bottleneck task — lives in ./diagnosis.md, deferred from this chapter on purpose. This chapter starts after that diagnosis lands and answers only “now what do I train, on what data shape.”

Standing scope note, same as the sibling chapters this one sits beside: every claim below is general RL/ML theory or frontier-lab practice (DeepSeek-R1, Qwen3, AlphaGo/AlphaGo Zero, Ng-Harada-Russell reward-shaping theory). Academic cybersecurity-LLM papers are not cited anywhere in this chapter — none exist as evidence here, per the project’s standing rule.

0. The running example, stated once

One failure — “the agent doesn’t use amass for subdomain enumeration, defaults to a shallow nmap sweep instead, and misses the flag hidden behind a forgotten staging subdomain” — cashes out as three structurally different problems depending on what’s actually broken, and each needs a different training row:

Threading this through the chapter keeps the point concrete: the same observed symptom (“doesn’t use amass”) routes to three incompatible fixes, and running the wrong one either wastes a training run or actively teaches the wrong thing (raw-trajectory SFT for a K-gap; more format-shaping SFT for a P-gap the model already has the capability for). One diagnostic worth stating up front, cross-linked from ./diagnosis.md: if the same tool-avoidance pattern shows up across several unrelated frontier base models, that’s evidence against a per-model K-gap (a shared pretraining-distribution prior — generic recon over structured OSINT tooling — is a much more parsimonious explanation than every base model independently lacking the same fact) and for an R- or P-gap. Verify per model before trusting this shortcut; it’s a prior, not a proof.

1. Genealogy — three lineages converge, then fragment

Three independent research lineages solved three different problems — “how do you improve a policy from its own samples,” “how do you optimize against comparisons instead of a scalar reward,” and “how do you stop a reward function from being gamed” — and by 2025 all three had to be stitched together to ship a single flagship reasoning model. Understanding why they converge, and where the post-convergence literature has since fractured, is what tells you whether a given fix is settled practice or a live bet.

flowchart TD
  subgraph A["Lineage A — self-improvement / sharpening"]
    direction TB
    A0["Ng, Harada & Russell (ICML 1999, no arXiv id)<br/>potential-based reward shaping is<br/>necessary + sufficient for policy invariance"] --> A1
    A1["AlphaGo (Nature 529, 2016, no arXiv id)<br/>supervised-init THEN RL"] --> A2
    A2["AlphaGo Zero (Nature 550, 2017, no arXiv id)<br/>the supervised step is not always necessary"] --> A3
    A3["STaR (2203.14465)<br/>sample rationale, keep if correct, SFT, repeat"] --> A4
    A4["RAFT (2304.06767) / ReST (2308.08998) /<br/>ReST-EM (2312.06585)<br/>generalize the filter-then-SFT loop"] --> A5
    A5["GRPO (2402.03300)<br/>turns it online, critic-free"] --> A6
    A6["Sharpening Mechanism (2412.01951)<br/>PROVES: can only redistribute mass<br/>already in base support, never create it"]
  end

  subgraph B["Lineage B — preference"]
    direction TB
    B0["DPO (2305.18290)<br/>closed-form reweighting of reference-policy mass"] --> B1
    B1["KTO (2402.01306)<br/>generalizes to unpaired good/bad labels"] --> B2
    B2["Self-Rewarding LMs (2401.10020)<br/>closes the loop online"] --> B3
    B3["Online-vs-offline (2405.08448)<br/>offline DPO helps classification,<br/>not generation — go online"]
  end

  subgraph C["Lineage C — reward design"]
    direction TB
    C0["Concrete Problems (1606.06565)<br/>names reward hacking"] --> C1
    C1["Skalse et al. (2209.13085)<br/>PROVES: unhackability needs a<br/>restricted policy class"] --> C2
    C2["Goal misgeneralization<br/>(2105.14111, 2210.01790)<br/>correct reward ≠ sufficient"] --> C3
    C3["UED / PAIRED (2012.02096)<br/>+ minimax-regret fix (2507.03068)<br/>the curriculum answer"]
  end

  A6 --> R1
  B3 --> R1
  C3 --> R1

  R1["DeepSeek-R1 (2501.12948)<br/>cold-start SFT (R) → RLVR (P) →<br/>rejection-sample 800K rollouts (re-diversify) →<br/>final alignment RL (sharpen).<br/>R1-Zero ablation proves stage-1's marginal value."]

  R1 --> Q3["Qwen3 (2505.09388)<br/>confirms this is now default frontier practice"]

  R1 --> Frag["POST-R1 FRAGMENTATION (2025–2026)"]

  Frag --> F1["Echo Chamber (2504.07912)<br/>RL amplifies pretraining-precursor<br/>behaviors, doesn't invent new ones"]
  Frag --> F2["pass@k boundary debate:<br/>shrink (2504.13837) vs<br/>expand (2505.24864 ProRL) vs<br/>reconcile (2510.04028)"]
  Frag --> F3["Scalpel vs Hammer (2507.10616)<br/>SFT=broad-replace, GRPO=narrow-amplify<br/>[contested, single low-cite preprint]"]
  Frag --> F4["RL's Razor (2509.04259)<br/>on-policy RL is KL-minimal from base<br/>[behavior agreed, mechanism contested]"]
  Frag --> F5["Base Model Barrier (2603.06957)<br/>escaping zero-likelihood costs<br/>exponential reward queries"]
  Frag --> F6["PaST (2601.11258)<br/>SFT/RL weight-space deltas<br/>are near-orthogonal"]

  classDef lin fill:#132b22,stroke:#34d399,color:#eafaf3;
  classDef conv fill:#3a2a10,stroke:#f5b942,color:#fff3d6;
  classDef frag fill:#2b1313,stroke:#f87171,color:#fde8e8;
  class A0,A1,A2,A3,A4,A5,A6,B0,B1,B2,B3,C0,C1,C2,C3 lin;
  class R1,Q3 conv;
  class Frag,F1,F2,F3,F4,F5,F6 frag;

1.1 Lineage A — self-improvement / sharpening

The oldest thread, and the one that names the ceiling this whole chapter has to work around. Ng, Harada & Russell (ICML 1999, no arXiv id) prove that reward shaping via a potential function F(s,a,s') = γΦ(s') − Φ(s) is necessary and sufficient for policy invariance — add any other shaping term and you risk creating a new optimum that isn’t the one you wanted. This theorem is the reason §3 below exists: every later reward-hacking fix in Lineage C is downstream of knowing exactly what form of shaping is safe. AlphaGo (Nature 529, 2016, no arXiv id) establishes supervised-init-then-RL as the working pattern — train a policy on expert demonstrations first, refine with self-play RL second. AlphaGo Zero (Nature 550, 2017, no arXiv id) shows the supervised step isn’t strictly required — pure self-play RL from random init reaches superhuman play — which is the first empirical hint that a sharpening loop can bootstrap capability without an off-policy teacher. STaR (arXiv:2203.14465) ports this to text: sample a rationale, keep it only if the final answer verifies, fine-tune on the survivors, repeat — the minimum-viable self-improvement loop, no reward model, no RL infra. RAFT (arXiv:2304.06767), ReST (arXiv:2308.08998), and ReST-EM (arXiv:2312.06585) generalize STaR’s filter into a formal growing-batch offline-RL loop. GRPO (arXiv:2402.03300) turns the whole thing online and critic-free — full mechanics in Reinforcement — PPO · GRPO · RLVR, not re-derived here.

The lineage’s ceiling gets a formal proof at the end: the Sharpening Mechanism (arXiv:2412.01951) shows this whole family — STaR through GRPO — can only redistribute probability mass the base policy already assigns nonzero weight to. It cannot manufacture support where none existed. This is the theoretical anchor for why a K-gap (zero support, at any N) is categorically un-fixable by anything in this lineage, and it’s the reason §2.1 below routes K-gaps somewhere else entirely.

1.2 Lineage B — preference

DPO (arXiv:2305.18290) collapses the RLHF reward-model-plus-PPO pipeline into a single closed-form classification loss, provably a re-weighting of the reference policy’s own mass toward the chosen side of each pair — full mechanics in Preference — RLHF · DPO · KTO. KTO (arXiv:2402.01306) generalizes the same idea to unpaired desirable/undesirable labels — the shape you get for free out of a rollout pool with a pass/fail verifier and no matched pairs. Self-Rewarding Language Models (arXiv:2401.10020) closes the loop online — the model generates its own new preference pairs each round instead of training on one frozen batch. Understanding the performance gap between online and offline alignment (arXiv:2405.08448) is the mechanistic reason iteration matters here: offline DPO measurably helps the model’s classification accuracy (can it tell chosen from rejected after training) without a matching gain in generation quality (does it actually produce the chosen behavior more often) — going online, resampling from the current policy each round, is what closes that gap. This is the direct citation behind §2.3’s “iterate 2–3 rounds” instruction below, and it’s the same distributional-gap argument Preference and Is the recipe a loop? already make about DPO’s off-policy-by-default weakness — not re-derived here.

1.3 Lineage C — reward design

Concrete Problems in AI Safety (arXiv:1606.06565) names reward hacking as a first-class failure mode a full decade before RLVR made it a daily operational concern. Skalse et al. (arXiv:2209.13085) formalize why it’s so hard to design around: a proxy reward is provably unhackable only under a constant-reward condition or a restricted policy class — for an unrestricted, expressive policy (any modern LLM), essentially every reward you can write down that isn’t the true objective itself has some hackable slack. Goal misgeneralization — Langosco et al. (arXiv:2105.14111) and Shah et al. (arXiv:2210.01790) — is the sharper, and for this chapter the load-bearing, half of that finding: even a correct reward specification is not sufficient, because the training data leaves the target goal underdetermined between several policies that all score equally well on the data you actually trained on. Shah et al.’s own framing is exactly the fix §3 uses below: disambiguate the target by making it structurally necessary — construct training/environment data such that the intended goal is the only remaining explanation for high reward. UED/PAIRED (arXiv:2012.02096) and its minimax-regret fix (arXiv:2507.03068) give the curriculum-design answer to the same disambiguation problem — generate environments that are maximally informative about which goal the policy has actually learned, rather than hand-picking a fixed curriculum that happens to be uninformative.

1.4 Convergence at DeepSeek-R1

DeepSeek-R1 (arXiv:2501.12948) is where all three lineages stop being separate research programs and become one shipped pipeline: cold-start SFT (an R-shaped fix — small, format/behavior-only, off-policy) → reasoning-focused RLVR (a P-shaped fix — GRPO reranking what the cold-started policy already samples, per Lineage A/C’s combined machinery) → rejection-sample ~800K rollouts off the RL-converged checkpoint and retrain from clean base (this re-diversifies/re-broadens the training pool before the next pass — restocking coverage that the RL stage’s entropy collapse had narrowed, not a K-fix in the strict sense but a K-adjacent move in the same “put breadth back before sharpening further” spirit) → a final all-scenario alignment RL pass. The paper’s own R1-Zero ablation (pure RL, zero cold-start SFT) is the direct, published proof that stage one has real marginal value — R1-Zero reaches comparable raw reasoning capability but ships with “poor readability, language mixing,” the exact failure mode an R-gap fix (format/behavior shaping) exists to close. Qwen3 (arXiv:2505.09388) runs the identical stage skeleton and states outright that it deliberately minimizes the cold-start stage’s size — confirming this is now default frontier practice, not a DeepSeek-specific quirk. (Full stage-by-stage mechanics of this convergence are in The recipe is a sequence, not a pick and Is the recipe a loop? — this chapter only needs the convergence point, not the whole sequence.)

1.5 Post-R1 fragmentation

Once the “K/R/P all get separately addressed, in this order” pattern became legible, 2025–2026 work immediately started arguing about the boundaries of what stage 3 (the sharpening/RL stage) can actually do:

• Echo Chamber (arXiv:2504.07912) — RL post-training amplifies behaviors that were already precursors in pretraining; it doesn’t invent behavior with no pretraining trace.
• The pass@k boundary debate. Yue et al. (arXiv:2504.13837) — the base model overtakes RLVR at large k; the reasoning boundary shrinks with training. ProRL (arXiv:2505.24864) — under prolonged, KL-controlled training with reference-policy resets, the boundary genuinely expands past what the base model ever reaches. The Two-Stage Dynamic View (arXiv:2510.04028) reconciles both as two phases of the same training run rather than a contradiction — full treatment of this debate, its stakes for this project, and the CoT-Pass@K/Cover@τ metric caveats already live in Reinforcement’s exploration section and Contested edges §1, §7; not re-derived here.
• Scalpel vs. Hammer (arXiv:2507.10616) — “GRPO amplifies existing capabilities, SFT replaces them,” at the weight level. Flag this honestly every time it’s cited: single, 0-citation preprint, its own authors call it a “preliminary indication” — supporting color for the K/R/P split’s intuition, not the load-bearing evidence for it (that’s the Sharpening Mechanism proof above, plus SFT Memorizes/RL Generalizes, arXiv:2501.17161, cited fully in Contested edges §1).
• RL’s Razor (arXiv:2509.04259) — on-policy RL forgets less than SFT because it’s implicitly KL-minimal from whatever policy preceded it: an on-policy update only has to move probability mass among things the current policy already samples, so it stays close to its own starting point in a way off-policy SFT — which pulls toward an externally-fixed target distribution — does not. The behavioral finding (RL forgets less) is well-replicated; the mechanistic explanation (KL-minimality specifically, vs. some other circuit-level preservation property) is actively being revisited. This is the citation behind §5’s staged-curriculum ordering below, used with that mechanism caveat carried forward.
• Base Model Barrier (arXiv:2603.06957, very recent) — formalizes exactly what Lineage A’s Sharpening Mechanism proved qualitatively: escaping a region of genuinely zero base-model likelihood costs exponentially many reward queries under policy-gradient methods. This is the sharpest, most recent statement of “RL cannot cheaply create what CPT/SFT never put there” — direct grounding for §2.1’s K-gap routing.
• PaST (arXiv:2601.11258, very recent) — measures the actual weight-space deltas SFT and RL produce and finds them near-orthogonal: the two stages aren’t fighting over the same parameter directions, which is a load-bearing empirical fact for §5’s “reconstruct subset-sums of stages without full retraining” ablation move.

None of these six papers overturns the K/R/P routing this chapter uses — they sharpen where the line sits between “elicit” and “expand,” which matters for how aggressively you can lean on a P-gap fix (§2.3) before it starts behaving like a K-gap fix in disguise. Treat the fragmentation cluster as live, contested terrain to watch, not a reason to distrust the routing itself.

1.6 What the genealogy buys you for routing

Three things carry forward from this history into §2’s mechanics, and it’s worth naming them once so the rest of the chapter can lean on them without re-arguing the point each time:

1. Lineage A’s proof (Sharpening Mechanism, arXiv:2412.01951) is why K-gaps get routed away from every technique in §2.2–§2.4. Cold-start SFT, DPO/KTO, and GRPO are all, at bottom, members of the sharpening/self-improvement or preference lineages — they redistribute mass or reweight comparisons over what already has some support. None of them can be the fix for a gap defined as “zero support at any N.”
2. Lineage B’s online-vs-offline finding (arXiv:2405.08448) is why §2.3 insists on iteration, not a single pass. A P-gap fix that stops after one offline round has only improved the model’s ability to tell apart chosen from rejected — not necessarily its tendency to generate the chosen behavior more often. That gap between classification and generation is exactly what makes a one-shot DPO run look like it “didn’t work” on eval even though the loss curve looked fine.
3. Lineage C’s theorem (Ng-Harada-Russell, ICML 1999) is why §3 exists as a separate section rather than a footnote on §2.3. Once you’re tempted to add a reward bonus for the correct action instead of (or alongside) a preference pair, you’ve left the safely-understood territory of “reweight an existing comparison” and entered reward-design territory, where the policy-invariance guarantee only holds for one specific mathematical form of shaping.

2. The matched intervention, gap by gap

flowchart LR
  Diag["Diagnosed gap<br/>(see ../diagnosis.md)"] --> K{"K — never fires<br/>at any N"}
  Diag --> R{"R — fires at moderate N,<br/>but not the DEFAULT"}
  Diag --> P{"P — fires with real mass,<br/>just mis-ranked"}

  K --> K1["CPT: raw/paraphrased continuation rows<br/>(unsupervised, no chat structure)"]
  K --> K2["knowledge-SFT: forward+reverse QA rows<br/>on the fact, NOT full trajectories"]

  R --> R1["cold-start SFT rows:<br/>(prompt, reasoning-block, tool_call)<br/>thousands, format/behavior only"]

  P --> P1["DPO/KTO pair at the exact<br/>decision-divergence state:<br/>(chosen tool_call, rejected tool_call)"]
  P --> P2["or short GRPO pass:<br/>reward = 1 iff correct action used,<br/>prompt = the divergence state only"]

  classDef kfix fill:#132b22,stroke:#34d399,color:#eafaf3;
  classDef rfix fill:#3a2a10,stroke:#f5b942,color:#fff3d6;
  classDef pfix fill:#2b1313,stroke:#f87171,color:#fde8e8;
  class K,K1,K2 kfix;
  class R,R1 rfix;
  class P,P1,P2 pfix;

Reading the diagram left to right: the diagnosis (deferred to ./diagnosis.md) hands you exactly one of three verdicts, and each verdict has exactly one row-shape family attached to it — there is no “run all three, see what sticks” branch, because §1.6 above is precisely the argument for why that wastes budget on two-thirds of a portfolio it shouldn’t touch. The three subsections below walk each branch in the order K → R → P, matching increasing on-policy-ness and decreasing training-row volume — K needs the most data and the least of the model’s own behavior; P needs the least data and the most of the model’s own behavior.

2.1 K → inject: CPT + knowledge-SFT, never raw trajectory-SFT

If the correct action never appears at any N, on any checkpoint, per the diagnostic battery in ./diagnosis.md, there is nothing on-policy to reinforce — the Sharpening Mechanism (arXiv:2412.01951) and the Base Model Barrier (arXiv:2603.06957) both say the same thing from different angles: you cannot cheaply RL your way to mass that isn’t there. The fix is off-policy injection, and it comes in exactly two row shapes, run in this order:

Run this: two-stage K-gap fix.

1. Continued pretraining (CPT) — full-FT, low LR, raw or lightly-paraphrased unsupervised continuation text. Not a chat turn.

   amass enum -passive -d target.com performs passive-only subdomain
   enumeration via OSINT sources (crt.sh, VirusTotal, DNS aggregators)
   without ever sending a packet to the target's own infrastructure —
   this avoids the IDS/WAF triggers that amass enum -active risks...
   

2. Knowledge-SFT — QA-format rows, both directions (dodges the reversal curse, arXiv:2309.12288: a model trained only on “A is B” does not reliably answer “what is A” from “B”).

   {"messages": [
     {"role": "user", "content": "What does the -passive flag do on amass enum?"},
     {"role": "assistant", "content": "Restricts amass to OSINT-only sources (crt.sh, VirusTotal, DNS aggregators) — it never touches the target directly, so it won't trip an IDS/WAF."}
   ]}
   {"messages": [
     {"role": "user", "content": "I need subdomain enumeration that never sends a packet to the target. Which tool/flag?"},
     {"role": "assistant", "content": "amass enum -passive."}
   ]}
   

What this is not: a full multi-turn recon-to-exploit trajectory that happens to include one amass call. That’s raw trajectory-SFT, and it’s the one thing this section explicitly rules out for a K-gap — training on a trajectory teaches “this is the shape a solve looks like,” which is a much weaker, noisier signal for “here is the standalone fact about this tool” than a direct QA row, and it compounds with a separate, serious risk (new-fact SFT rows measurably increase hallucination on unrelated facts once learned) that this chapter defers in full to ./trajectory-amplification.md rather than re-deriving. The full data-curation playbook for this stage — CPT-vs-SFT dosing, WRAP-style paraphrase counts, EntiGraph for narrow corpora, self-play Self-QA, KnownPatch interleaving — is deferred to ./knowledge-curation.md; this section only needs to establish the row shape and the ordering.

Common mistake — mistaking “can recite it” for “genuinely absent.” The K-gap probes in ./diagnosis.md distinguish recognition (the model can describe amass -passive when directly asked) from generation-in-context (it never reaches for the flag at the actual decision point). If a model passes the recognition probe but still fails the pass-at-k sweep during a live rollout, the fact is present — this is an R-gap wearing a K-gap’s costume, and routing it into CPT wastes compute on a fact the weights already encode while leaving the actual default-prior problem untouched. Run the recognition-vs-generation split before committing to §2.1’s two-stage fix, not after it fails to move the eval.

Hyperparameters/gotchas that matter here, carried forward from the genealogy: CPT is full-FT at a low learning rate, not LoRA-first — the Base Model Barrier’s exponential-cost argument (arXiv:2603.06957) is exactly the failure mode a rank-constrained update is worst-positioned to escape, since it further restricts how much of parameter space a single update can move through. Knowledge-SFT rows, by contrast, are cheap and low-rank-friendly — the fact itself is small; only the CPT stage needs the larger capacity budget.

2.2 R → re-prior: cold-start SFT, thousands of format/behavior rows

If the correct action fires at moderate N but the model’s default, low-temperature sampling never reaches for it, the fact is present but the prior isn’t — this is not a knowledge problem, it’s a distribution-shaping problem, and it’s solved the same way DeepSeek-R1’s own stage 1 solves it: a small, deliberately narrow, off-policy cold-start SFT pass whose entire job is to move where the policy’s probability mass starts before any RL or preference stage touches it.

The row is a full (prompt, reasoning-block, tool_call) unit — not a QA pair, because the target here is behavior at a decision point, not a fact:

{"messages": [
  {"role": "system", "content": "You are a security agent. Tools: run_command, submit_flag."},
  {"role": "user", "content": "Challenge: enumerate target.com for attack surface."},
  {
    "role": "assistant",
    "content": "<think>Before touching the host directly, passive subdomain enumeration (amass -passive) is lower-risk and often surfaces attack surface a direct nmap sweep misses — staging subdomains, forgotten hosts. Run that first, then port-scan whatever it finds.</think>",
    "tool_calls": [{"id": "call_1", "type": "function",
      "function": {"name": "run_command", "arguments": "{\"cmd\": \"amass enum -passive -d target.com\"}"}}]
  }
]}

Thousands of rows, not hundreds of thousands — Qwen3’s own explicit design intent is to minimize this stage’s size (arXiv:2505.09388), and DeepSeek-R1’s cold-start set is “thousands,” deliberately tiny (arXiv:2501.12948). R1’s own four sourcing methods, directly reusable here:

1. Few-shot elicitation — prompt a strong model with 2–3 hand-written examples of the desired recon-first behavior, let it generate more in the same shape.
2. Zero-shot elicitation with an explicit verify/reflect instruction — ask the model to solve the challenge and self-check before committing, then keep the outputs that pass.
3. Self-distillation of the policy’s own rollouts — sample the current model, keep the (rare) rollouts where it did reach for amass first, use those as the seed.
4. Reward-filtered rejection sampling — sample broadly, verify against the real environment, keep only trajectories where the correct-first-move pattern led to a genuine solve.

Cast every sourced row into the same reasoning-block template above, then, per PEAR (arXiv:2602.01058), do not pick the checkpoint that scores highest on the SFT loss/accuracy itself — pick the one at peak sample diversity (entropy or self-BLEU over K≥8 sampled completions per prompt). PEAR’s own finding is exactly the trap an R-gap fix can fall into: the SFT checkpoint that looks best by its own metric can be the worst initialization for whatever comes next (a P-gap fix, §2.3, or a sharpening pass, §4) because it has already collapsed onto one narrow phrasing of the target behavior — which is the opposite of what a re-prior stage should hand off. Measure diversity at every candidate checkpoint along the SFT run, not just at the final step.

Common mistake — letting cold-start grow into a capability-SFT pass. The row shape above is narrow on purpose: one decision point, one reasoning block, one tool call. It’s tempting to pad the set with full downstream trajectories once you’re already generating rows — resist it. A cold-start stage that grows past “thousands of narrow rows” starts behaving like the heavy SFT/DPO stage The recipe is a sequence already warns caps a later RL stage’s exploration room; the R-fix’s whole value is that it’s small enough to nudge the prior without over-constraining what §2.3 or §4 need to explore afterward. If the set feels too small to be “real training,” that’s closer to correct than a set that feels comfortably complete.

Hyperparameters/gotchas that matter here: dedupe aggressively across the four sourcing routes — few-shot elicitation and self-distillation of the model’s own rollouts can easily converge on near-identical phrasings of the same recon-first pattern, which quietly re-introduces the low-diversity failure PEAR’s checkpoint-selection rule is designed to catch. Cap volume per archetype the same way The kinds of SFT §6 recommends for any SFT corpus, not just this one.

2.3 P → re-rank: on-policy DPO/KTO at the divergence point, or a short GRPO pass

This is the amass example’s third variant, and the most commonly misdiagnosed of the three: the model already produces the correct action, with real, nontrivial probability mass, sometimes in the very same rollout batch as the wrong one — it’s outranked, not absent. Feeding this into an R-shaped cold-start SFT pass wastes a training run reinforcing a capability that’s already there off-policy; feeding it into a K-shaped CPT/knowledge-SFT pass teaches a fact the model already knows. What it needs is a ranking intervention, run on-policy, localized to the exact state where the divergence happens — not the whole trajectory.

DiaTool-DPO (arXiv:2504.02882) is the direct template: build the preference pair at the specific decision-state of a multi-turn tool-dialogue MDP, not by contrasting two entire episodes. For the amass case, that state is “just received the recon prompt, about to choose the first tool call” — not turn 40 of a 100-turn trajectory.

// DPO pair — localized to the divergence state
{
  "prompt": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "Challenge: enumerate target.com for attack surface."}
  ],
  "chosen":   {"tool_calls": [{"function": {"name": "run_command",
                "arguments": "{\"cmd\": \"amass enum -passive -d target.com\"}"}}]},
  "rejected": {"tool_calls": [{"function": {"name": "run_command",
                "arguments": "{\"cmd\": \"nmap -sV target.com\"}"}}]}
}

// KTO pair — same divergence state, unpaired labels (fits a mined pool of
// verified-good / verified-bad rollouts where matched pairs don't exist)
{"prompt": [...], "completion": {"tool_calls": [{"function": {"name": "run_command",
  "arguments": "{\"cmd\": \"amass enum -passive -d target.com\"}"}}]}, "label": "desirable"}
{"prompt": [...], "completion": {"tool_calls": [{"function": {"name": "run_command",
  "arguments": "{\"cmd\": \"nmap -sV target.com\"}"}}]}, "label": "undesirable"}

Where the pairs come from: mine them from the policy’s own rollout failures, not a hand-written contrast set — Boosting Tool Use (arXiv:2501.09766) mines deficiency pairs via tree search directly over the model’s own generated branches, comparing what it did produce against what it could have produced at the same state. This keeps the pair on-policy by construction, which matters more than which loss head you pick: on-policy-ness dominates loss-head choice (arXiv:2406.09279) — preference-data quality and how close it sits to the current policy’s own distribution explains more of the outcome variance than whether you run DPO vs. KTO vs. some other variant. Choose DPO if matched pairs exist naturally at the divergence point, KTO if you only have single-sided verified-good/verified-bad labels from a trace-verification pass.

Two guardrails, both load-bearing:

• Guard against likelihood displacement. Near-duplicate chosen/rejected pairs — two amass invocations differing only in a flag — can cause Unintentional Unalignment (arXiv:2410.08847): DPO’s gradient pushes down the rejected sequence’s likelihood so hard it drags down neighboring, correct sequences in embedding space along with it. Discard near-duplicate pairs at data-construction time, not after observing the regression.
• Iterate, don’t run once. Per §1.2’s online-vs-offline finding (arXiv:2405.08448), a single offline DPO/KTO pass improves the model’s ability to classify chosen-vs-rejected without a matched gain in how often it actually generates the chosen behavior at inference. Redeploy the updated checkpoint, resample fresh pairs from its current rollouts (not the original policy’s), retrain — 2–3 rounds, closing the classification-vs-generation gap each round. This is the same restart-vs-continue asymmetry Is the recipe a loop? already documents for preference-stage revisits — continue-from-current-checkpoint, regenerate fresh on-policy pairs each round, don’t reuse round-1 pairs in round 3.

Or, when the divergence is best expressed as a scalar reward rather than a pairwise contrast: a short GRPO pass, prompts fixed to the divergence state, reward = 1 iff the correct action was used, 0 otherwise. This is the P-gap’s RL-shaped alternative to DPO/KTO — same target (re-rank what’s already reachable), different mechanism (on-policy sampling + group-relative advantage instead of a closed-form pairwise loss). It should be short: this is a ranking fix on a narrow decision point, not a capability-expansion run, and §4 below is the caution against letting it run long enough to start behaving like one.

Common mistake — contrasting whole episodes instead of the divergence state. The most common way a P-gap fix silently degrades into wasted compute is building the (chosen, rejected) pair from two entire trajectories that happen to end differently, rather than the single state where the paths actually split. Two full episodes differ in dozens of ways beyond the one decision that matters — DPO’s gradient has no way to know which of those differences caused the reward difference, and the update spreads thin across all of them instead of sharpening the one branch you diagnosed. DiaTool-DPO’s whole contribution (arXiv:2504.02882) is refusing to do this — build the pair at the MDP state where the tool choice diverges, nowhere else.

Hyperparameters that matter: β (DPO’s KL-strength term — Preference has the full mechanics) should stay conservative here, since the goal is a targeted re-rank at one decision point, not a broad behavioral shift; a too-high β risks the likelihood-displacement failure above spreading further than the intended state. For the GRPO alternative, keep the prompt distribution narrow (the divergence state and close variants of it, not the full challenge portfolio) and the group size N modest — this is a polish pass, not a from-scratch RLVR run, and Reinforcement’s 30–60% baseline-band requirement still applies to whatever prompt set you construct.

3. Making the target instrumentally necessary — env/reward design as a P-gap force multiplier

A DPO/KTO pass or a short GRPO reward can re-rank the target action if the model already reliably reaches states where using it matters. If amass-style enumeration is merely helpful but optional — the challenge is also solvable, just less elegantly, by brute-forcing the visible subdomain — then any reward bonus for using it is fighting an uphill battle against Goal Misgeneralization (arXiv:2210.01790): the training data underdetermines why the bonus fires, and the policy can just as easily learn “sometimes get a bonus for a specific string in my tool call” as “OSINT-first recon is the right general strategy.” Shah et al.’s own fix, restated for this setting: make the target action structurally, instrumentally necessary — design the environment so the correct behavior is the only remaining path to reward, not one of several equally-scoring paths.

Concretely: withhold the flag-bearing staging subdomain from any DNS record reachable by a naive nmap/brute-force sweep, so it is recoverable only through OSINT-style passive enumeration. Run large-k rollouts against that bottleneck; if the sub-skill never fires even when it’s the sole path to reward, that’s the strongest possible confirmation of a genuine K-gap (not R or P) — this is Probe 4 of the diagnostic battery, deferred in full to ./diagnosis.md, but the environment-design move that produces the bottleneck is this section’s contribution, not that one’s.

Four guardrails on doing this without reopening a reward-hacking failure mode:

• Keep any shaping strictly potential-based. Ng, Harada & Russell’s theorem (ICML 1999, no arXiv id) is the reason “just add a bonus for calling amass” is dangerous on its own — an arbitrary shaping term can change which policy is optimal, not merely which one is found faster. A potential-based term (F(s,a,s') = γΦ(s') − Φ(s), for some state-potential Φ) is provably policy-invariant: it can only change how fast you find the right answer, never which answer is right. Any dense intermediate signal layered onto the terminal flag-verified reward — matching the security-agent/GRPO reward-shaping discipline already established in Agentic & multi-turn RL — must take this form or it risks silently relocating the optimum.
• Verify the trajectory, not just the outcome. A coarse, outcome-only reward under-specifies which tool use actually mattered — ToolRL (arXiv:2504.13958) is the direct citation for why a flag-only reward is insufficient once you’re trying to reinforce a specific mid-trajectory action: score whether the structurally-necessary tool call actually appears in the winning trajectory, not merely whether the episode ended in success (which, per the bottleneck design above, should now be equivalent — but verify it directly rather than assuming the bottleneck is airtight).
• Don’t route through a hackable PRM if you can avoid it. If dense credit does need to flow through a learned scorer rather than a hard-coded environment bottleneck, know that state-of-the-art process reward models are systematically exploitable under RL pressure — arXiv:2603.06621, very recent — the policy learns to satisfy the PRM’s proxy for “good reasoning” without the reasoning itself improving. If a PRM-scored path is unavoidable, use min-form credit assignment — PURE (arXiv:2504.15275) — which removes the single-step reward-farming exploit that summed/averaged PRM scores are prone to.
• When multi-step reward hacking is the live risk (not just single-step), reach for myopic-plus-approval instead of a fully-learned critic. MONA (arXiv:2501.13011) trades a cheap per-step human/oracle approval signal against optimizing multi-step reward directly, specifically to prevent the kind of multi-turn reward hacking a hard-coded bottleneck can’t fully close off (e.g., an agent that finds an unintended second path to the flag once the intended one gets bottlenecked). MONA and PURE represent a genuine, unresolved trade-off — cheap step-level oracle approval (MONA) vs. trusting a learned PRM that’s itself hackable (PURE) — no consensus exists on which is preferable; pick based on whether a cheap-enough step-level oracle is available for your environment.

Worked resolution — the amass bottleneck, end to end

Putting §2.3 and §3 together for the running example: suppose diagnosis confirms a P-gap — amass calls appear in maybe 15% of sampled rollouts at the recon decision point, nmap-only appears in the other 85%, and both lead to a genuine solve when the target’s staging subdomain happens to also be brute-forceable. A DPO pass built on this environment is fighting Goal Misgeneralization (arXiv:2210.01790) the whole way: the training data can’t tell the model why amass should be preferred, because right now it isn’t actually necessary — it’s merely one of two equally-scoring paths, and the “preference” the pair encodes might just as easily be learned as “sometimes use this specific string” as “OSINT-first is the general strategy.”

The environment-design move: retarget the challenge so the flag-bearing subdomain is DNS-registered but never brute-force-guessable within the harness’s turn budget — reachable only through certificate-transparency-style passive discovery. Now:

Φ(state) = 1 if OSINT-derived subdomain list is non-empty, else 0
F(s, a, s') = γ·Φ(s') − Φ(s)     # potential-based bonus, Ng-Harada-Russell-safe

This bonus rewards reaching the OSINT-derived-list state faster, without ever changing which final policy is optimal — the terminal, ground-truth flag check remains the only large-magnitude reward, exactly the discipline Agentic & multi-turn RL’s reward-shaping section already establishes for this project. Re-run the P-gap DPO pair or short GRPO pass from §2.3 against this environment, and the pair now teaches something that generalizes — “OSINT-first recon is instrumentally necessary here,” not “this specific string sometimes scores higher.”

4. Sharpen (rejection-sampling/RFT) is last-stage-only

Rejection-sampling FT and short RFT/GRPO polish passes belong strictly after K, R, and P are separately addressed — they are a sharpening operation in the Lineage-A sense (redistribute mass within existing support, per the Sharpening Mechanism, arXiv:2412.01951), not a substitute for injecting missing knowledge or re-shaping a default prior. Run it inside the 0 < p < 1 band — a challenge subtype the current checkpoint sometimes solves, never a 0% or already-saturated one — and instrument three things simultaneously, not pass@1 alone:

• Exploration — unique-correct-completions per prompt, tracked the way B-STaR does it (arXiv:2412.17256): are you still discovering new correct paths, or resampling the same one?
• Exploitation — reward-spread collapse across the sampled group; a shrinking spread is the early warning that the policy is narrowing before the exploration signal says so.
• pass@large-k alongside pass@1.

Run this: the tripwire. If pass@1 rises while pass@256 (or your largest affordable k) falls, that is not a stable win — it is the elicit-not-expand signature (Yue et al., arXiv:2504.13837): the sharpening pass is narrowing the reasoning boundary in exchange for a higher hit rate on the easy part of it. The fix is not more training steps on the same data — it’s more rollouts per prompt. BroRL (arXiv:2510.01180) shows scaling the rollout count per prompt directly counteracts the step-count plateau that drives this collapse; add breadth before you add duration.

This is the same tripwire Reinforcement’s graduation trigger instruments via entropy — read that section for the mechanism (R = -a·exp(H)+b) this section’s pass@1-vs-pass@256 check is the outcome-level symptom of.

5. Staged curriculum, and the ablate-a-stage protocol

Given all three gap fixes plus a sharpening pass might apply to different subsets of the same failing portfolio, the ordering isn’t arbitrary. Mirror DeepSeek-R1’s own stage order (§1.4): highest-distribution-shift stage first, lowest-drift stage last.

• K first — CPT/knowledge-SFT is the largest single distributional shift you’ll apply (new facts, potentially a new token distribution entirely), and it’s cheapest to detect and repair forgetting from early, while you still have full budget to catch a regression before compounding it with three more stages on top.
• R second — cold-start SFT is a smaller, narrower shift (format/behavior only, thousands of rows), and needs the K-stage’s facts already in place to have anything correct to shape a prior toward.
• P third — on-policy DPO/KTO or a short GRPO reranking pass is smaller still, and per RL’s Razor (arXiv:2509.04259) an on-policy update is implicitly KL-minimal from whatever precedes it — it moves probability mass only among things the current policy already samples, so it drifts least from the checkpoint it starts on. (Behavioral finding well-replicated; the KL-minimality mechanism specifically is being revisited by newer mechanistic work questioning whether KL-proximity or circuit-level preservation is the real causal variable — carry that caveat forward, don’t state the mechanism as settled.)
• Sharpen last — §4’s rejection-sampling/RFT polish, by construction, only redistributes mass that K/R/P have already put in the right place; running it earlier just sharpens a still-wrong distribution faster.

The ablate-a-stage-on-a-frozen-eval protocol, directly reusable here: build a checkpoint at each of base → K-fix → R-fix → P-fix → sharpen, and score every one of them on a single frozen shared eval suite — never a stage’s own training metric. This is the same discipline Quagmires in SFT-RL Post-Training (arXiv:2510.01624) establishes for the SFT→RL boundary specifically (high SFT-stage accuracy on its own metric can predict worse downstream RL) and PEAR (arXiv:2602.01058, §2.2 above) establishes for picking a checkpoint by diversity instead of accuracy — generalized here across all four stages, not just the SFT/RL pair. If a stage’s marginal delta on the frozen suite is negative, stop and re-diagnose rather than pushing forward to the next stage — a negative delta at, say, the R-fix checkpoint means either the K-fix underneath it was incomplete, or what looked like an R-gap was actually something else.

A genuine shortcut this protocol buys you, if it holds on your own data: PaST (arXiv:2601.11258) found SFT and RL weight-space deltas are near-orthogonal — if that holds for your K/R/P-fix deltas too, you can reconstruct subset-sums of stages (K+P without R, say) by adding the relevant weight deltas together, without a full retrain from base for every combination you want to test. Verify this on your own checkpoints before relying on it; it’s a very recent, single-source finding, not yet independently replicated.

When to skip the staging entirely. The four-stage order above is the answer when a portfolio-wide diagnosis turns up a mix of K/R/P failures across different challenge subtypes — which is the common case per Diagnosing the gap’s own bottom line (“expect a split verdict, not one number”). If your diagnosis instead comes back clean — every failing challenge in the current batch is the same gap type — skip straight to that single fix rather than running all four stages on principle; §5’s ordering exists to sequence multiple, different fixes safely, not to mandate a fixed four-stage pipeline regardless of what was actually diagnosed.

Exit criteria per stage, adapted from Is the recipe a loop?’s general loop-exit rule to this chapter’s four named stages: stop advancing to the next stage, and re-diagnose instead, when any two hold — (a) the frozen-suite delta for the current stage is within noise of the previous checkpoint, (b) pass@64 on the current checkpoint is flat vs. the checkpoint before it, (c) a non-targeted behavior’s own rate has measurably dropped since the previous stage. This is the same exit test Is the recipe a loop? §4 uses for the broader SFT↔RL loop, applied here one level more granularly — per K/R/P/sharpen stage, not per pipeline round.

Cross-links

• Diagnosing the gap — a scientific framework — the pass@k / Pass@(k,T) / Cover@τ / elicitation-ladder instrumentation that tells you which gap (K/R/P) a given failure is, before this chapter’s routing applies.
• ./diagnosis.md — the fuller K/R/P-specific diagnostic battery this chapter assumes as input: the pass-at-k sweep, the in-context oracle probe, teacher-forced logprob spans, the reversal check, and the purpose-built bottleneck task from §3 above.
• ./knowledge-curation.md — the full K-gap data-curation playbook (CPT dosing, WRAP-style paraphrasing, EntiGraph for narrow corpora, forward/reverse QA construction, self-play Self-QA, KnownPatch interleaving) that §2.1 only sketches.
• ./data-centric-methods.md — generic data-selection machinery (quality/complexity/diversity filtering, IFD/LESS/DEITA-style scoring) that applies across all three gap-fix row shapes, not repeated here.
• ./trajectory-amplification.md — the full warning behind §2.1’s “never raw trajectory-SFT for a K-gap”: why new-fact SFT rows compound into hallucination on unrelated facts, and why the highest-scoring SFT checkpoint can be the worst RL init.
• Imitation — SFT · distillation · rejection sampling and The kinds of SFT — the on/off-policy presets and row-shape taxonomy §2.1/§2.2’s CPT/knowledge-SFT/cold-start rows are instances of.
• Preference — RLHF · DPO · KTO and Reinforcement — PPO · GRPO · RLVR — full DPO/KTO/GRPO mechanics behind §2.3 and §4, not re-derived here.
• The recipe is a sequence, not a pick and Is the recipe a loop? — the stage-ordering and continue-vs-restart mechanics §1.4 and §5’s curriculum lean on directly.
• The decision — the one-line K/R/P-adjacent routing tree (knowledge / execution / ranking gap) this chapter’s three-way split is the fuller, matched-intervention version of.
• Contested edges & landmines §1, §7, §9 — the full “does RL elicit or expand” debate behind §1.5’s fragmentation cluster and §4’s tripwire.
• Agentic & multi-turn RL — the reward-masking and shaping discipline (mask tool-output tokens, keep the terminal signal ground-truth) §3’s guardrails extend into the multi-turn setting.

Bibliography

Confidence calibration, stated once: every id above was carried forward from a prior verification pass against arxiv.org/abs/<id> (artifacts/three-gap-survey/ledger-E.md), not recalled from training-data memory. “MED — very recent” ids (2601.11258, 2602.01058, 2603.06957, 2603.06621, 2510.04028) are single-source, low-citation-count-at-verification-time findings — treat as promising, directionally load-bearing for this chapter’s routing, not yet independently replicated. arXiv:2507.10616 is flagged LOW everywhere it’s cited in this book, this chapter included — its own abstract calls the finding preliminary.

The data-curation toolkit — selection, negatives, coverage

The question this chapter answers: given a pile of run logs and a diagnosed gap, what are the actual mechanics — which row to keep, how to mine a negative that teaches the right ranking, how to synthesize what’s missing, how to prove your data has a hole before you find out the expensive way — that turn “we have logs” into “we have a training set”? BLUF: there is no single “clean the data” step. Selection, negative-mining, active acquisition, decontamination, curriculum, synthesis, and coverage-auditing are seven separate instruments, each answering a different sub-question, and the field’s post-2022 LLM literature is almost entirely three much older ideas — curriculum/self-paced learning, active learning, and hard-negative mining for contrastive retrieval — re-derived for autoregressive transformers. Knowing the lineage tells you which tool each new-sounding 2025/2026 paper is actually handing you.

This chapter is the cross-cutting toolkit, not the per-gap playbook. It answers how you select, mine, and audit data; it deliberately does not re-decide which fix (SFT vs DPO vs GRPO) a given failure needs — that routing lives in Matching the fix to the gap and The three gaps, defined, whose K/R/P vocabulary (Knowledge gap / prior-ranking “R” gap / policy-vs-judgment “P” gap) is exactly the taxonomy this chapter’s methods feed: §1–2 and §7–8 are mostly load-bearing for K- and R-gaps (what to inject, how to know it’s missing), §3–4 are mostly load-bearing for R- and P-gaps (how to mine the negative that teaches the ranking). (K/R/P maps onto framework.md’s knowledge/execution/exploration lens as execution = R ∪ P, one continuous axis at two granularities — see taxonomy.md §0 for the full reconciliation, canonical.) The kinds of SFT §6 already covers LIMA/AlpaGasus/IFD/LESS/DEITA and n-gram decontamination at the SFT-row level — this chapter extends that into the full toolkit (production-scale gotchas, hard-negative mining for preference pairs, active/curriculum acquisition, synthesis mechanics, coverage-gap detection) without repeating its worked examples. Per-gap-specific curation (what a confirmed K-gap corpus recipe looks like end to end) is Knowledge-gap curation; which fix to route to per gap is Intervention per gap; preference-method mechanics (the DPO/KTO loss itself) is Preference.

Running example, threaded through every section: your CTF-solving agent’s run logs, where you need to decide (a) which of your run-log rows survive into a training set, (b) how to construct (shallow-tool ≺ expert-tool) and (default-args ≺ expert-args) preference pairs — the agent reaches for nmap/curl by reflex and never reaches for amass (passive subdomain/recon enumeration) even when it’s the objectively better tool for the job — and (c) how you’d prove, before you spend a training run finding out, that your corpus has a coverage hole where amass/passive-recon should be.

Status: every arXiv id below is verified live against arxiv.org/abs/<id> per the source ledger (artifacts/three-gap-survey/{section-F,ledger-F}.md, verification pass 2026-07-02); non-arXiv sources (OpenReview, ACL Anthology, pre-arXiv classics) are cited by venue, not linked to an invented id. Confidence tags follow the ledger: unmarked = CONFIRMED and load-bearing; flagged inline where a claim is single-paper, contested, or too recent to be independently replicated.

0. The genealogy in three lines

Three 2009–2010-era ideas, developed independently for entirely different problems, are what every 2023–2026 LLM-data-curation paper in this chapter re-derives:

Why the lineage matters practically: every time a 2025/2026 paper’s headline sounds novel — “online curriculum for RL,” “prompt-perturbation uncertainty,” “policy’s-own-failures hard-negative mining” — check which of these three roots it’s re-deriving. It tells you the failure mode it doesn’t fix (self-paced curricula still need re-estimation as the policy moves; uncertainty sampling still confounds “unsure of the fact” with “unsure of the phrasing”; hard-negative mining still needs a floor against too-easy or too-hard negatives) before you trust a paper’s improvement as free.

1. Selection — which rows to keep

“Selection” answers one question: given a pool of candidate SFT rows, which subset do you actually train on? The seminal finding, repeatedly confirmed at small-to-medium scale then complicated at production scale, is that more data is not more signal — a smaller, better-chosen set beats a larger, unfiltered one.

What “quality” operationally means, collapsed to one sentence per axis: correctness (AlpaGasus) is “is this response actually right,” informativeness (IFD) is “did the model need this instruction to produce this response,” targeted influence (LESS/BIDS) is “does this row’s gradient point at the specific capability I diagnosed as missing” — three different, all legitimate, all not interchangeable meanings of “good data.”

The production-scale correction

The single most important corrective in this literature: at production scale, most of this collapses toward random. Ivison et al. (AI2, up to 5.8M-row pools) arXiv:2503.01807 found IFD/DEITA/complexity-style selection falls below random selection once you count their own compute cost — only a cheap, gradient-free, judge-free representation-similarity method (RDS+: weighted mean-pooled hidden states of a pretrained-not-instruction-tuned LM, similarity/kNN against a small target set) stays Pareto-optimal at every scale. This is independently corroborated by the Qwen/Alibaba team at million-scale, arXiv:2410.09335: self-scoring methods that beat random at 10K–100K scale become statistically indistinguishable from random at million-scale, on two independent million-scale pools.

This is a genuine, unresolved contested edge, not settled: the seminal small-scale papers (10K–100K pools) all say sophisticated selection wins; the two production-scale studies above say it collapses to or below random once compute is counted. The field has not published a principled crossover pool-size threshold. Regime-gate your method choice (§9) rather than trusting either side unconditionally.

A related complication for your teacher choice, not selection per se: arXiv:2411.07133 found that a stronger model’s own benchmark performance does not monotonically predict how good its responses are as SFT training targets for a given student — don’t assume “biggest teacher available” is automatically the best distillation source without checking student-side downstream fit.

2. The quality / diversity / complexity axes

DEITA arXiv:2312.15685 is the paper that names this precisely, and its most load-bearing finding for your case isn’t the score itself — it’s the failure mode of naive scoring: ranking candidates by quality/complexity alone and taking the top-k reproduces whatever pattern already dominates the pool. If 80% of your run logs are nmap-then-curl trajectories, a pure quality-score top-k selection over that pool gives you a curated set that is still 80% nmap-then-curl — high-scoring rows cluster around the harness’s existing skew unless an explicit diversity/coverage constraint forces otherwise. This is the exact mechanism behind “the model reaches for one salient tool, uses default args”: your existing SFT/rollout pool has that skew, and score-only selection amplifies rather than corrects it.

InsTag arXiv:2308.07074 operationalizes diversity and complexity as two independently measurable axes via open-set LLM tagging: diversity = number of unique tags present, complexity = tags-per-example, both correlating with SFT quality holding dataset size fixed. This is the tagging machinery §7’s synthesis-conditioning and §8’s coverage audit both reuse.

The concrete fix for the “one salient tool” failure, combining §1’s methods with the diversity constraint:

1. Score the pool by quality/informativeness (AlpaGasus or IFD).
2. Walk the score-ranked list; keep a candidate only if its (tool_name, arg_pattern)-distance (or embedding distance, if you don’t have a structured tool signature) to every already-kept row exceeds a threshold τ.
3. This — DEITA’s diversity-dedup applied as a hard constraint on top of, not instead of, quality ranking — is the mechanism that specifically prevents top-k selection from reproducing the prior-collapse it’s supposed to fix.

Note what this does and doesn’t do: it makes your existing good amass rows survive selection instead of being crowded out by a thousand near-duplicate nmap rows. It does not create amass rows that don’t exist in the pool at all — that’s a synthesis problem (§7) or a coverage-gap problem (§8), not a selection problem. Selection can only re-weight what’s already there.

3. Hard-negative mining for preference pairs

This is the P-gap machinery: constructing (chosen, rejected) pairs so that DPO/KTO actually teaches the right ranking, rather than a pair so trivial the gradient vanishes or so extreme it teaches a shortcut.

The lineage, applied

ANCE’s founding result arXiv:2007.00808 — random or in-batch negatives give a vanishing gradient once a model is even moderately good, because the model already ranks them far below the positive; a hard negative has to come from the model’s own current top-ranked wrong candidates, mined globally, not sampled at random. RSO arXiv:2309.06657 ports this to preference optimization directly: both DPO’s static pairs and SLiC’s SFT-policy-only pairs diverge from the maximum-likelihood target, and rejection-sampling toward the estimated optimal policy before pairing closes that gap. DPO itself arXiv:2305.18290 establishes the (prompt, chosen, rejected) triplet objective — but the load-bearing finding for curation, not algorithm choice, is Ivison et al.’s systematic ablation arXiv:2406.09279: across {preference data, algorithm, reward model, training prompts}, preference-data quality dominates outcome variance more than DPO-vs-PPO-vs-KTO algorithm choice. Put engineering budget in the mining pipeline below, not in loss-function tuning.

The worked pipeline, for (shallow-tool ≺ expert-tool)

flowchart TD
  A["Roll out K=4-8 samples per task,<br/>current checkpoint, T≈0.7-1.0"] --> B["Decompose at the tool-call<br/>DECISION POINT, not whole trajectory<br/>(ANCE/RSO principle)"]
  B --> C["Verify each candidate:<br/>tool-exists · arg-schema-valid ·<br/>arg-semantically-correct<br/>(structured rubric, not judge alone)"]
  C --> D["Mine the hard negative =<br/>highest-rubric-scoring WRONG<br/>action at that decision point"]
  D --> E["Reject near-zero-probability<br/>candidates — vanishing gradient<br/>just like random negatives"]
  E --> F["Construct the MINIMAL PAIR:<br/>anchor at the tool-call boundary,<br/>continue BOTH branches with the<br/>POLICY'S OWN rollout"]
  F --> G{"DIAGNOSTIC GATE:<br/>does the policy complete the<br/>'chosen' branch at a non-trivial<br/>rate (mini pass@k)?"}
  G -->|No| H["K-GAP masquerading as a<br/>ranking problem —<br/>route to §7/knowledge-curation,<br/>NOT DPO"]
  G -->|Yes| I["CHES similarity check<br/>(chosen vs rejected embeddings)"]
  I -->|Too similar| J["Discard/diversify —<br/>likelihood-displacement risk"]
  I -->|OK| K["Train; monitor chosen<br/>log-prob directly, not<br/>just the margin"]

Concretely, at the amass-vs-nmap decision point (recon stage, target has a large unenumerated subdomain surface):

// Anchor: same prefix, same decision point. Two branches diverge here.
{
  "prompt_prefix": [
    {"role": "system", "content": "Tools: nmap, amass, curl, submit_flag."},
    {"role": "user", "content": "Target: corp-app.example.com. Find the flag."}
  ],
  "chosen": {
    "tool_calls": [{"function": {"name": "amass", "arguments": "{\"domain\": \"example.com\", \"mode\": \"passive\"}"}}],
    "rationale_for_chosen": "Broad passive enumeration before touching any single host — surfaces staging/internal subdomains nmap alone won't find."
  },
  "rejected": {
    "tool_calls": [{"function": {"name": "nmap", "arguments": "{\"target\": \"corp-app.example.com\", \"ports\": \"1-1000\"}"}}],
    "why_this_is_the_hard_negative": "Not a strawman — nmap is a HIGH-rubric-scoring, plausible, frequently-correct action; it's wrong here specifically because it skips subdomain enumeration on a target with a large unenumerated surface. That's what makes it a hard negative, not a random one."
  }
}

The diagnostic gate (Verified Critical Step Optimization arXiv:2602.03412) is the step people skip and shouldn’t: force the current policy to continue executing from the chosen branch and check it succeeds at a non-trivial rate. If the policy can’t complete the amass-first path even when handed it, that’s not a ranking problem — the model doesn’t know how to use amass effectively (a K-gap), and training DPO on this pair risks amplifying an unfixed knowledge gap rather than fixing a preference ranking. Route to synthesis/knowledge-injection (§7, Knowledge-gap curation) instead.

The ceiling: likelihood displacement

arXiv:2410.08847 (Razin, Malladi, Bhaskar, Chen, Arora, Hanin — Princeton, ICLR 2025) is the hard ceiling on “how minimal can a mined pair be.” A chosen/rejected pair that is too embedding-similar can catastrophically push probability mass onto a third, unintended, semantically-distant completion, rather than genuinely re-ranking chosen above rejected. Diagnosed via a CHES (centered-hidden-embedding-similarity) check between the pair before training; monitored during training via the chosen-response log-probability directly, not just the chosen-minus-rejected margin — a shrinking margin can hide a chosen log-prob that’s also silently falling.

Contested, and worth A/B-testing before you build on-policy mining infra: is on-policy negative sampling (the ANCE/RSO prescription above) always better than static negatives? A 2026 ICLR poster (OpenReview tz9mJmgrdM) found it ranges from 3× better to 0.4× worse depending on model/task, with the flip-sign mechanism uncharacterized. Don’t assume on-policy is automatically correct for your setup — verify on your own model before investing in the rollout infrastructure the pipeline above needs.

4. Active-learning / uncertainty-guided acquisition

Selection (§1) filters a pool you already have. Active learning decides which new example to generate/label next, targeting the diagnosed gap directly instead of hoping a fixed pool happens to contain it. Classic roots: core-set arXiv:1708.00489 (diversity over per-example uncertainty in batch settings), BALD arXiv:1703.02910 (MC-dropout mutual information as an uncertainty proxy).

The crucial LLM-era fix — semantic, not token, entropy. Raw token-level entropy is dominated by paraphrase/lexical variation, not by genuine uncertainty about the underlying fact or decision. Semantic entropy arXiv:2302.09664 (Kuhn, Gal & Farquhar) fixes this: sample k completions, cluster them by bidirectional NLI entailment, compute entropy over the cluster distribution rather than the raw token distribution. This is your direct K-vs-R distinguishing instrument.

Active Instruction Tuning arXiv:2311.00288 operationalizes query-by-committee via prompt-paraphrase disagreement instead of a model ensemble — no second model needed. Runnable today, directly on your tool-choice problem:

1. Take the same scenario (target with a large unenumerated subdomain surface), generate 3–5 paraphrases varying phrasing, not the underlying task.
2. Run the current checkpoint on each paraphrase at T≈0.7–1.0.
3. Measure the tool/argument-choice flip-rate across paraphrases.

RLVR dynamic sampling — active acquisition inside the training loop

On the RL side, the single most-replicated 2025–2026 finding — PCL arXiv:2510.01135, AdaRFT arXiv:2504.05520, SEC (Self-Evolving Curriculum) arXiv:2505.14970, DAPO’s Dynamic Sampling arXiv:2503.14476, and VCRL (OpenReview FBhWTuMTYA) — is that training signal is maximized at intermediate difficulty: pass-rate ≈ 50%, equivalently maximal reward-variance across a rollout group. This is free: the k rollouts GRPO already generates give you pass_rate and reward_std with zero extra inference.

# free — reuses rollouts you already generate
pass_rate = mean(reward for _ in rollouts)          # per prompt, across k samples
reward_std = std(reward for _ in rollouts)
if reward_std ≈ 0:                                   # all-correct or all-wrong: zero advantage signal
    drop_and_resample(prompt)                        # DAPO's Dynamic Sampling, arXiv:2503.14476
oversample(prompts_with(pass_rate in [0.2, 0.6]))    # this is the literature-side justification
                                                       # for keeping a GRPO baseline in the ~30-60% band

Because difficulty is policy-dependent and goes stale as the policy improves, PCL/SEC recompute it online via a learned value model or bandit rather than caching a once-computed label — a static difficulty score is wrong by definition after enough training steps.

Contested, two different decisions, not a contradiction: arXiv:2508.14094 finds that under a fixed data-acquisition budget (new problems to annotate/verify), prioritizing the hardest available examples gives the largest GRPO gains (up to 47%) — the opposite of the intermediate-difficulty rule above. The two rules answer different questions: what to collect (acquisition budget → go hard) vs. what to train on per training step from an already-large pool (batch composition → stay intermediate). Don’t apply one where the other belongs.

5. Decontamination for eval integrity

This is a load-bearing confound on every method above: a curation or synthesis pipeline that accidentally leaks your held-out eval into the training pool doesn’t fail loudly — it inflates the training-set score without inflating real capability, and every diagnostic in this book that reads “the fix worked” is only as trustworthy as this gate.

Stage 1 — cheap, mandatory baseline. N-gram overlap: a token counts as contaminated if it sits in a shared run of more than ~10 tokens between an eval sample and the training set — this is the methodology the Llama 2 report uses (§A.6) and is a standard, reproducible first pass. Pair with exact-substring/suffix-array and MinHash near-duplicate detection across the full corpus, not just the eval set — deduplication of the training corpus itself measurably improves downstream models independent of any eval-leakage concern arXiv:2107.06499.

Stage 2 — mandatory if any rephrase/persona/backtranslate step touched source material (§7 does, by construction). N-gram-only decontamination is trivially bypassed by paraphrase arXiv:2311.04850 (the LLM Decontaminator paper) — exactly the kind of rephrasing your synthesis pipeline does on purpose. An embedding-similarity flag (cosine >~0.80–0.85) plus an LLM-judge confirmation pass (“is this a paraphrase/restatement such that the answer is directly inferable?”) is required on top of, not instead of, n-gram/MinHash dedup.

Stage 3 — RL-specific, and the reason this section isn’t just an SFT concern. A single very recent controlled study arXiv:2601.06103 (Jan 2026, small models 0.5B–4B, not yet independently replicated) found GRPO-style RL on clean data generalizes leaked pretraining contamination to uncontaminated same-family items, whereas SFT only re-inflates the literal contaminated rows. A clean RL environment sitting on a contaminated base model can silently look like genuine capability gain. Before trusting a GRPO/RLVR pass@k jump as real, decontaminate the reward/eval set against the base model’s own pretraining corpus, not just your own synthetic pipeline’s outputs.

Contested — does contamination matter at frontier scale at all? Bordt et al. arXiv:2410.03249 found moderate repeated contamination is effectively “forgotten” via weight decay once pretraining exceeds ~5× Chinchilla-optimal tokens. This describes large, diluted, public benchmarks — it does not license skipping decontamination on a small, high-value held-out set (a handful of CTF flags, a curated eval trace), which is exactly the regime where memorization risk is highest, not lowest.

6. Curriculum / difficulty-ordering

Curriculum learning’s classic root — Bengio et al., ICML 2009 (no arXiv id) — orders/weights training examples by a computable “easiness.” Self-paced learning, Kumar/Packer/Koller, NeurIPS 2010 (no arXiv id), sharpens this: the model itself sets its own pace rather than an external teacher fixing an order in advance. Nearly every RL-curriculum paper in §4’s “RLVR dynamic sampling” subsection (DAPO arXiv:2503.14476, PCL arXiv:2510.01135, AdaRFT arXiv:2504.05520, SEC arXiv:2505.14970, VCRL) is this same self-paced-learning idea, re-derived: the pass-rate/reward-variance signal is the model’s own, computed online, exactly because a static curriculum staled the moment the policy moved past it.

For offline SFT curricula specifically (as opposed to online RL batch composition), order training rows by IFD (§1) or by difficulty-as-judged, low-to-high, on the theory that early-training gradient steps on rows the model already half-knows are more stable than starting on the hardest rows cold. This is weaker evidence than the RL-side finding and is genuinely contested for generative tasks specifically:

Contested — difficulty vs. coverage, for generative fine-tuning. Dataset Cartography’s original framing arXiv:2009.10795 (§8) treats “ambiguous/hard-and-persistently-wrong” as the interesting region for classification-era tasks. A 2025–2026 workshop paper (OpenReview g1DiK2Yi4j) argues that for generative SFT/agent-trajectory tasks specifically, difficulty-based selection narrows output-distribution coverage and underperforms random selection — not yet reconciled with cartography’s classification-era prescription. If your curated set is agentic-trajectory-shaped (it is), don’t import a pure difficulty-ordering rule from classification-era curriculum literature without checking it against a coverage metric (§8) first.

Net practical rule: online RL batch composition → intermediate difficulty, recomputed continuously (§4). Offline SFT-row ordering → don’t lean on difficulty alone; check coverage isn’t collapsing (§2, §8) at the same time.

7. Synthetic-data generation mechanics

Selection (§1–2) and negative-mining (§3) work on data you already have. This section is how you produce rows to fill a gap those methods can’t fill by re-weighting alone — the mechanics behind the corpus you’d curate, not a replacement for curation.

The lineage, in order:

• Self-Instruct arXiv:2212.10560 — bootstrap instruction/I-O triples from ~175 seeds via in-context generation, ROUGE-L<0.7 dedup as the core diversity control.
• Evol-Instruct / WizardLM arXiv:2304.12244 — LLM-driven in-depth/in-breadth rewrite operators escalate instruction complexity beyond what’s cheap to hand-author.
• InsTag arXiv:2308.07074 — the tagging machinery from §2, reused here to condition generation toward under-tagged regions, not just to measure a finished pool.
• Instruction Backtranslation arXiv:2308.06259 — predict the instruction a piece of real web/document text answers (text→instruction direction, not the reverse), then self-curate. Grounds every synthesized row in genuine text — the same “grounded in reality, not imagined” principle §1’s LIMA and this whole chapter’s K-gap concerns rest on.
• WRAP arXiv:2401.16380 — rephrase noisy source docs into a small, fixed set of styles (3–5: Wikipedia-like, Q&A, ELI5), not open-ended paraphrase volume. A small fixed style-set, empirically, beats unbounded paraphrase generation.
• Persona Hub arXiv:2406.20094 — 1B mined personas; injecting a persona into a generation prompt biases sampling toward that persona’s knowledge slice, explicitly demonstrated for synthesizing tools/functions at scale in the source paper.
• Magpie arXiv:2406.08464 — truncate the chat template right before the user turn, let the aligned model autoregress its own query. Cheapest possible query generator, zero seeds needed — but it faithfully reproduces whatever skew already exists in the policy, so it diagnoses prior skew (it’s a good coverage-audit probe, §8) but cannot fix it by construction.

The R-gap fix, concretely: force coverage of amass without fabricating a tool result

1. Do not self-sample (Magpie-style) for this — by definition it reproduces the existing nmap-first skew.
2. Build/reuse a tool taxonomy independent of your current data (InsTag-style tagging over your tool documentation, not over what’s already in your run logs — critical, see §8).
3. Condition synthesis on personas/scenarios that structurally require the underused tool (Persona Hub): “a security researcher who always starts with passive OSINT before touching a single host” → generates a scenario where the winning first move is amass, not nmap.
4. This synthesizes the scenario/query, not the tool result. Cross-reference kinds-of-sft.md §4: fabricating what amass’s output looks like teaches confabulation exactly like any other ungrounded tool result. Run the synthesized scenario for real, against a real target, and keep the executed transcript.
5. Verify with the same three-stage contract ToolACE arXiv:2409.00920 and APIGen-MT arXiv:2504.03601 both independently converge on: format-valid → executes against a real/sandboxed API → LLM-judge/committee semantic-match.
6. Apply DEITA’s diversity-dedup (§2) on the resulting set so it doesn’t just become a second, smaller monoculture.

A very recent inversion worth knowing about, not yet broadly validated: Firefly arXiv:2605.17558 starts from real API outputs and works backward to the query — the opposite direction from ToolACE/APIGen-MT — specifically to fix synthetic-schema drift (where a model-generated tool call is plausible-looking but doesn’t match the real tool’s actual schema/output shape). Same grounding principle as Instruction Backtranslation, applied to tool calls instead of documents.

8. Coverage-gap detection — proving the hole exists

This is the closest thing in the literature to a direct methodological answer to “how do I know what’s missing” — and it’s the step that turns “I suspect we never train on amass” into a defensible, falsifiable claim.

Dataset Cartography arXiv:2009.10795 (Swayamdipta et al.) plots every training example on (confidence, variability-across-epochs) from a single training run — no held-out taxonomy needed to run it — exposing easy / ambiguous / hard-and-persistently-wrong regions. On its own this tells you about examples you have; it says nothing about examples you don’t.

The step that actually finds the hole: cross-reference the hard-and-never-improving bucket against an independent skill taxonomy — built first, from your tool documentation or --help output, via InsTag-style LLM self-tagging arXiv:2308.07074 or metacognitive labeling arXiv:2405.12205 — not derived from what’s already in your run logs, or you’ll define your skill taxonomy in a way that can’t see the hole it’s supposed to find.

Worked example — detecting the amass/passive-recon hole:

1. Build the taxonomy from your tool surface, independent of usage: {nmap, amass, curl, sqlmap, http_get, ...} × {active-scan, passive-recon, exploit, exfil, ...}.
2. Count raw mentions of each tag combination across your run-log corpus — this is the Kandpal-style mention-count check arXiv:2211.08411, which established a causal relationship between corpus mention count and answerability for facts; the same logic applies directly to tool-usage rows.
3. amass × passive-recon appears 0 times across the corpus. That’s not “the model happens to underperform on this” — it’s a directly measurable, confirmed coverage hole, distinguishable from a merely-rare-but-present pattern.
4. Cross-check against capacity, not just presence: Allen-Zhu & Li’s knowledge-capacity scaling laws arXiv:2404.05405 (~2 bits of knowledge per parameter, in controlled synthetic setups, degraded by junk-data dilution) tell you that even after you inject amass coverage, if it’s a vanishingly small token-fraction of a much larger/noisier mix, the effective capacity a given model size allocates to it shrinks — track domain-token-fraction, not just raw row count, once you’ve closed the zero-coverage gap.

Confirming the hole is real, not an artifact of your training pipeline (two independent per-example diagnostics):

• Gekhman et al. arXiv:2405.05904 (Technion/Google): SFT rows containing facts/skills genuinely unknown to the base model, flagged by a pre-SFT in-context-oracle probe, are fit more slowly and, once fit, linearly increase hallucination on unrelated held-out inputs. If you push amass rows into SFT and unrelated tool-choice accuracy gets worse, that’s this mechanism confirming the pre-injection gap was real and structural, not incidental.
• The Incomplete Learning Phenomenon arXiv:2604.10079 (UNSW/Tencent, April 2026, single paper, not yet broadly replicated) shows even converged SFT models fail to reproduce a persistent subset of their own training data, with “missing prerequisite knowledge in the base model” as one of five named, empirically-distinguishable causes (alongside SFT-data inconsistency, sequential-finetuning forgetting, and under-optimized rare patterns) — a per-example diagnostic, not just an aggregate score, for confirming why a specific amass row didn’t stick even after training on it.

Closing the loop end-to-end: STAT arXiv:2510.10023 (Princeton) operationalizes exactly this cycle — tag training data by skill, train, probe the trained model per-skill-tag to build a Missing-Skill-Profile from its own post-training failures, then reweight or synthesize specifically for the failing tags. Reported as complementary to GRPO/RL, not a substitute for it — run STAT’s probe after a training round, not instead of one.

9. Apply these in what order

None of §1–8 is a default pipeline you run start to finish every time — pick the subset the diagnosis calls for. But when you are building a curated/mined set, apply them in this order:

1. Coverage-gap detection first (§8). Before selecting or mining anything, confirm what’s actually missing (tag corpus against an independent taxonomy, check mention counts). Selection and negative-mining can only re-weight what already exists — running them on a pool with a zero-coverage hole just curates around the hole more efficiently.
2. If the gap is confirmed missing (not just rare): synthesize (§7), grounded — condition on personas/taxonomy gaps for scenario diversity, execute for real, verify with the three-stage contract. Do not self-sample (Magpie-style) to fill a confirmed hole; it reproduces the hole by construction.
3. Verifier-filter, always, before anything else touches the pool — this alone removes the majority of harmful noise (per kinds-of-sft §6).
4. Scale-gate your selection method (§1). Target set ≤~50–100K rows (the realistic regime for a targeted gap-patch): run the full quality→difficulty→diversity→targeted-gradient stack (AlpaGasus → IFD → DEITA → LESS/BIDS). Target set 100K+ rows: skip straight to representation-similarity (RDS+-style) or well-stratified random, and validate anything fancier against a compute-matched random baseline before trusting an uplift.
5. Apply the diversity constraint (§2) as a hard filter, not an afterthought, regardless of which selection method above you used — this is what prevents re-collapsing onto the existing skew.
6. If the diagnosed gap is a ranking problem (R/P-gap), mine hard negatives (§3) from the selected, verified pool — on-policy, decision-point-anchored, gated by the K-vs-P diagnostic, checked for likelihood displacement before training.
7. If acquiring genuinely new examples under a fixed budget, go hard (§4’s acquisition rule). If composing an already-large pool into RL batches, stay intermediate-difficulty, recomputed online, not cached.
8. Decontaminate against your eval set before training, every time (§5) — n-gram baseline always, embedding+judge if any synthesis/rephrase step touched source material, base-model-pretraining-corpus check if the downstream training is RL, not just SFT.

flowchart TD
  Start["Diagnosed gap<br/>(K / R / P — see decision.md)"] --> Cov["§8 Coverage-gap detection:<br/>is it actually MISSING,<br/>or just under-weighted?"]
  Cov -->|"Missing (0 mentions)"| Syn["§7 Synthesize, grounded —<br/>persona/taxonomy-conditioned<br/>scenario + REAL execution"]
  Cov -->|"Present but rare/skewed"| Sel["§1-2 Selection + diversity constraint<br/>on the existing pool"]
  Syn --> Ver["Verifier-filter (mandatory)"]
  Sel --> Ver
  Ver --> Gate{"Gap type?"}
  Gate -->|"K — inject off-policy"| Done1["SFT on the synthesized/selected set"]
  Gate -->|"R/P — mis-ranked"| Neg["§3 Hard-negative mining,<br/>on-policy, decision-point-anchored"]
  Neg --> LD["Likelihood-displacement check<br/>before training (CHES + chosen log-prob)"]
  LD --> Done2["DPO/KTO on the mined pairs"]
  Done1 --> Decon["§5 Decontaminate against eval —<br/>always, before trusting any resulting number"]
  Done2 --> Decon

  classDef your fill:#132b22,stroke:#34d399,color:#eafaf3;
  class Cov,Neg,LD your;

Cross-links

• The kinds of SFT §6 — the SFT-row-level version of §1’s selection stack (LIMA/AlpaGasus/IFD/LESS/DEITA), plus the forgetting/replay side this chapter doesn’t repeat.
• Method → Data — which training method (SFT/DPO/KTO/GRPO) consumes which data object; this chapter is the mechanics behind producing/filtering that object, not the method-selection question itself.
• Post-training dataset registry — concrete, proven-by-usage downloadable datasets if you need a general-capability ingredient (tool-calling format anchors, preference mixes) rather than building your own from run logs.
• Diagnosing the gap — the instrumentation (pass@k, Pass@(k,T), Cover@τ, elicitation ladder) that tells you which gap you’re curating for before you pick a method from this chapter.
• The decision — the K/R/P routing tree this chapter’s methods feed into.
• Knowledge-gap curation — the confirmed-K-gap corpus recipe end to end (deferred from §7/§8 here).
• Intervention per gap — which fix routes to which gap type, in full.
• Preference — RLHF · DPO · KTO — the loss mechanics §3’s mined pairs feed into.

Bibliography

Confidence calibration: every arXiv id above is verified live per the source ledger (artifacts/three-gap-survey/ledger-F.md, pass 2026-07-02), not recalled from training memory. High confidence on the genealogy claim (curriculum/active-learning/hard-negative-mining as the three roots) and on the core mechanics of each named method — these are widely-cited, independently-corroborated findings. Explicitly flagged single-paper or very-recent results (Verified Critical Step Optimization, the RL-contamination-generalization study, the Incomplete Learning Phenomenon, Firefly) are marked “promising, not yet validated” throughout and should not be treated as settled. Three genuine contested edges are called out inline rather than resolved: selection-vs-random at production scale, on-policy-vs-static negatives for DPO, and hard-vs-intermediate difficulty for acquisition-vs-batch-composition — the field has not published a reconciliation for any of the three, and this chapter deliberately does not manufacture one.

The one axis that predicts everything

You’re leaving the Problem section behind — the diagnostic chapters told you the agent solves a minority of your challenge set and left the “why.” Learnings starts the general theory from scratch, unattached to your specific case, beginning with the one axis every method in this book is built on.

If you internalize one thing, make it this: on-policy vs off-policy. It predicts which methods can fix which failures, and it’s the reason a colleague’s SFT run can move the eval by two points and stall.

Definitions (precise)

• Off-policy: training targets are sampled from a distribution other than the model’s current policy π_θ — a human, a teacher model, a frozen dataset. The model raises the likelihood of sequences it did not generate.
• On-policy: the training data is sampled from π_θ itself (the current weights), then scored/labeled. The model learns from its own rollouts.

This is standard RL vocabulary, not a framing I invented — see any policy-gradient treatment; the LLM-specific consequences are laid out formally in the imitation-learning reduction below.

Why off-policy imitation is structurally blind to execution gaps

The mechanism, as a flow:

graph LR
  A["Train on the teacher's states"] --> B["Deploy: π_θ drives,<br/>visits π_θ's OWN states"]
  B --> C["One slip → a state the<br/>teacher never visited"]
  C --> D["No training signal there<br/>→ error compounds"]
  D --> B

This isn’t hand-waving — it’s a theorem. Ross, Gordon & Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning” (AISTATS 2011, arXiv:1011.0686), Thm 2.1: a behavior-cloned policy with per-step error ε incurs total cost bounded by J(π̂) ≤ J(π*) + ε·T² — quadratic in horizon T, because a single deviation moves you to states off the expert’s distribution where you have no supervision, and errors accrue at up to unit cost for the rest of the episode. Their DAgger correction — aggregate data from the learner’s own induced state distribution — restores near-linear O(ε·T) regret.

Translate to your setting: an execution failure happens, by definition, in a state π_θ reaches on its own. Off-policy data is — structurally — data about states π_θ doesn’t reach. So off-policy SFT optimizes correctness in the wrong region. On a long agentic CTF trajectory (large T), the T² vs T gap is the whole story.

The fix, stated once

Make the data on-policy: sample from π_θ, then score those samples. Every method that “fixes execution” — rejection-sampling FT, GRPO/RLVR, on-policy distillation — is a different way of doing exactly that. The on-policy distillation line makes the connection explicit: it exists specifically to kill the train/inference mismatch of fixed-dataset KD by sampling from the student during training (GKD, Agarwal et al., arXiv:2306.13649).

The corollary you’ll use constantly

• Failure lives in π_θ’s own distribution (it solves sometimes, fails often) → on-policy method.
• The correct behavior is absent from π_θ entirely (never appears at high N) → nothing on-policy to reinforce → you must inject off-policy (SFT / teacher data / a tool).
• The behavior exists but is mis-ranked → preference method.

That routing is the spine of The decision.

What this axis is not

This is one axis, not a set of independently-toggleable knobs. Crossing on/off-policy against the imitation/preference/reward paradigm split does not yield a free combinatorial grid of methods — signal and policy-source largely determine each other, so each named method (SFT, rejection-sampling FT, DPO, GRPO/RLVR, on-policy distillation) is a fixed preset on this axis, not a pick-two-knobs product. An earlier pass at this material packaged it as an interactive matrix implying free combination, which produced nonsense for some products — that framing is retired; see Contested edges §6 for the correction and Methods overview for the preset list.

A caution on evidence quality while you’re building intuition here: a low-citation 2025 preprint (arXiv:2507.10616, “Scalpel vs. Hammer”) frames the amplify/replace split in exactly this vocabulary (“GRPO amplifies, SFT replaces”) but is a single, contested, 0-citation source — its own authors call it “preliminary.” Don’t anchor on it. The load-bearing evidence for “on-policy reinforces what already fires, off-policy is required to inject what doesn’t” is this chapter’s own DAgger argument above, plus “SFT Memorizes, RL Generalizes” (Chu, Zhai et al., arXiv:2501.17161, ICML 2025, 694 citations) and the elicit-not-expand result (Yue et al., arXiv:2504.13837, NeurIPS 2025 Best Paper Runner-Up) — both verified live, both far better cited than 2507.10616. See Contested edges §1 for the full, honest state of that debate (it’s genuinely contested at the margins — prolonged, KL-controlled RL can expand the boundary; standard-recipe RL elicits).

Where this axis goes next

This single axis is the foundation the rest of the book’s structure is built on:

• The fixed presets above don’t stand alone — frontier recipes run them as an ordered stage sequence, not a menu (The recipe is a sequence, not a pick).
• That sequence isn’t strictly one-shot either — it loops, and whether a re-applied stage continues from checkpoint or restarts from base changes what “on-policy” even means the second time around (Is the recipe a loop?).
• The ordering constraints on which stage-type can safely follow which are a direct consequence of the εT² argument above (Ordering rules: interleaving stages & fixing N problems).
• Off-policy injection has its own failure mode worth knowing before you reach for it — naive off-policy SFT can cause format/behavioral collapse independent of weight magnitude (Data mixing, ratios & not forgetting how to think).
• For “what do I actually run first,” the axis becomes a ranked, proven-first decision, not just a diagnostic (Start here: a proven-first ranking of the methods).

What “data” actually means for an agent

You asked the right question earlier: “demonstration = the answer — are you talking about trajectories?” Yes. Being concrete about the data object dissolves most of the confusion, because each method eats a different-shaped object, and for an agent those shapes are not what the chatbot literature implies.

This chapter stays deliberately narrow: it’s about the shape of the training object, not about which method to pick. That routing question belongs to The one axis that predicts everything — on-policy vs off-policy is the real axis, and each method below is a fixed preset on it, not one of several independent knobs you mix and match freely (see Contested edges on why a “combinatorial grid” framing over-reaches).

A “demonstration” is a trajectory, not an answer

• Chatbot SFT example = (prompt → ideal response text).

• Agent SFT example = the whole trajectory:

  system prompt
  → [assistant: reasoning] → [tool_call: shell "nmap …"] → [tool_result: <output>]
  → [assistant: reasoning] → [tool_call: http_request …] → [tool_result: <output>]
  → …
  → [assistant: submit_flag("FLAG{…}")]
  

The training target is the path, not the flag. That is what SFT and rejection-sampling FT imitate token-by-token.

Loss-masking: don’t train the model to predict the world

Tool outputs / observations are part of the sequence but are not the policy’s own tokens — they come from the environment. Standard practice is to mask the loss on prompt and observation spans and only compute loss on the model-generated reasoning + tool_call + submit tokens. Otherwise you teach the model to hallucinate stdout. (This is the same principle as instruction-tuning masking the prompt; for agent traces it’s applied to every observation span.) The project’s own trajectory-synthesis note treats observation-masking as a first-class filter step — see lessons/post-training/verified-trajectory-synthesis-recipe.md in the shared memory pool. (Caveat: that recipe’s specific numbers derive from an unverified third-party report — trust the procedure, not the figures.)

The data shapes, per method preset (forward reference)

These rows are not independent toggles you compose freely — each is a named method with a fixed data object, and which preset applies is determined by where the failure sits on the on/off-policy axis (Foundations, The decision):

This table is the hinge of the whole book. It’s expanded in Method → Data, which is the chapter that actually addresses your stated bottleneck (“we don’t know what data we want”). The short version: you don’t choose data and then a method — you choose the method, and the method dictates the data object.

A caution worth stating before you take that table as license to freely order/mix methods: the object shape here answers “what does one round of this method eat,” not “in what order do rounds run” or “what happens when you feed a later stage’s data mixture badly.” Those are separate, larger questions this book answers elsewhere — see The recipe is a sequence, not a pick, Is the recipe a loop?, Ordering rules: interleaving stages & fixing N problems, and Data mixing, ratios & not forgetting how to think (the last one is the direct continuation of this chapter’s “trajectory as training object” framing — it’s what happens when an off-policy trajectory mixture is wrong even though the shape per row above is right). When you’re ready to rank candidate methods/recipes by how proven they are rather than how novel, that’s Proven-first ranking.

The family map

You now have the axis (on-policy vs off-policy, from the previous chapter) and the shape of the data object (a trajectory, not an answer). This chapter zooms out one level and puts every named method in this book on those two coordinates — the map every family chapter that follows hangs off of.

Two canonical axes, not a grid you build methods from: what signal you learn from (imitation / preference / reinforcement — three families) and on/off-policy (Foundations — whose rollouts the training data comes from), plus one orthogonal axis, PEFT, that is a delivery mechanism, not a learning signal. Every named method below is a fixed preset on those two axes, not a free combination of them — signal + policy-source largely determine what a method does, so “pick a family, then pick a policy-source, then invent the method” produces non-methods, not real ones. This page used to teach the two axes as “three independent knobs you toggle,” a scaffold that over-reached; see Contested edges §6 for why that framing was retired. Learn the family, then the preset within it — that’s the whole map.

graph TD
  ROOT["Post-training<br/>push mass toward good behavior"]
  ROOT --> IM["IMITATION<br/>signal = demonstrations"]
  ROOT --> PR["PREFERENCE<br/>signal = comparisons A≻B"]
  ROOT --> RL["REINFORCEMENT<br/>signal = reward / verifier"]

  IM --> SFT["SFT"]
  IM --> DIST["Distillation<br/>off-policy / on-policy"]
  IM --> RS["Rejection-sampling FT<br/>= 'RL without RL'"]

  PR --> RLHF["RLHF (RM + PPO)"]
  PR --> DPO["DPO · KTO · IPO · ORPO · SimPO"]

  RL --> PPO["PPO"]
  RL --> GRPO["GRPO → GSPO / DAPO"]
  RL --> RLVR["RLVR (verifiable reward)"]
  RL --> AG["Agentic / multi-turn RL"]

  PEFT["PEFT: LoRA · QLoRA · DoRA<br/>a HOW, applied to any of the above"]
  ROOT -.delivery.-> PEFT

Picking a family here only answers which signal. Four questions this map deliberately leaves open, each with its own chapter, once you know which preset you’re reaching for: does the recipe run once or loop back over earlier stages? In what order can stage-types safely interleave, and how do you batch N fixes into one round? How do you mix data so a later stage doesn’t erase what an earlier one taught? And — for this project specifically — which preset should you actually start with, ranked by proven adoption rather than novelty? The status table below is what’s real in 2026; those four chapters are sequence, iteration, mixing, and rank.

What’s actually load-bearing in 2026 (verified)

The status column below is from a fresh Exa pass over model tech reports + lab blogs (2025–2026), not recalled — sources cited per row throughout the method chapters.

Read the family chapters for the mechanism + “what data it eats” + when to reach for each.

Cross-links: this page answers which family; The recipe is a sequence, not a pick answers in what order the families compose into an actual pipeline; Is the recipe a loop? and Ordering rules: interleaving stages & batching cover whether and how you revisit a family once you’ve left it; Data mixing, ratios & not forgetting how to think covers what happens to an earlier stage’s behavior when a later one trains on foreign data; The one axis that predicts everything is the on/off-policy half of this page’s two-axis claim, spelled out in full; and Start here: a proven-first ranking turns this family-level status table into one ranked starting sequence for this project specifically.

Imitation — SFT · distillation · rejection sampling

Signal = demonstrations. Objective = cross-entropy on target tokens. Imitation is one of three fixed paradigm presets on the canonical on/off-policy axis (the other two are preference and reinforcement) — see Foundations for the axis itself and Contested edges §6 for why “three independent knobs you toggle” is a retired teaching scaffold: the axes aren’t independent, so what differs across SFT / distillation / rejection-sampling is whose trajectories you imitate, not a free combination. This chapter is also Sequence B’s stage 0 (cold-start) and stage 3 (execution-gap bridge) in the recipe is a sequence, and the two T1 proven-first picks in proven-first ranking — read those for where in the pipeline and why start here, this chapter is the what/mechanism.

Per-method template: what · data it eats · on/off-policy · when · gotcha · cite.

SFT (Supervised Fine-Tuning)

• What: MLE on (prompt → target); for agents, target = a full trajectory (What “data” means). The original instruction-tuning result is InstructGPT (arXiv:2203.02155).
• Eats: curated/human/teacher demonstrations.
• Policy: off-policy (targets aren’t π_θ’s samples).
• When: inject a capability or format the model lacks; establish a cold-start before RL. Stage 0 of the ordered skeleton — never the whole recipe (the recipe is a sequence §3, stage ordering).
• Gotcha: off-policy ⇒ blind to execution gaps (the εT² compounding, Foundations) and it tends to memorize rather than generalize — the load-bearing evidence is Chu, Zhai et al., “SFT Memorizes, RL Generalizes,” arXiv:2501.17161 (ICML 2025, 694 citations): outcome-reward RL transfers to unseen rule/visual variants where SFT overfits the training distribution, plus the elicit-not-expand genealogy in Contested edges §1. A smaller, 0-citation preprint (“Scalpel vs. Hammer,” arXiv:2507.10616) frames the same split at the weight level as “GRPO amplifies, SFT replaces” — its own authors call this only a “preliminary indication” and their follow-up ablation “inconclusive,” so treat it as a rhyming, supporting citation, not the basis for the claim (see Contested edges §6 and the fuller writeup in RL long-horizon/exploration §4). Also worse on small models (less capacity to absorb without forgetting) — see data mixing & forgetting for the LoRA-format-collapse case study. And don’t over-invest here before RL: Llama 4’s recipe deliberately keeps SFT lightweight because heavy SFT/DPO restricts downstream RL exploration (Contested edges §4).

Distillation (a kind of SFT — the teacher supplies the demonstrations)

Knowledge distillation originates with Hinton et al., arXiv:1503.02531. Two variants, and the split is the on/off-policy axis:

• Off-policy distillation = SFT on the teacher’s completions. Mainstream: DeepSeek transfers R1’s reasoning into V3/V3.2 as a named post-training stage (DeepSeek-V3, arXiv:2412.19437; V3.2, arXiv:2512.02556); “distilled from GPT-4/R1” datasets are how most small OSS models get capability.
• On-policy distillation = student samples its own rollouts, teacher grades them densely (reverse-KL per token). Fixes the fixed-dataset train/inference mismatch (GKD, arXiv:2306.13649: +90% relative on GSM8K vs supervised-KD). Thinking Machines’ 2025 write-up reports Qwen3-8B ← Qwen3-32B reaching ~70% AIME’24 in 150–200 steps at ~9–30× less compute than RL-from-scratch (thinkingmachines.ai, 2025-10-27).
  • Honest status: niche / promising, not lab-confirmed. As of a 2026 pass, no frontier lab (OpenAI/Anthropic/Google/DeepSeek/Qwen) has stated on-policy distillation as its production recipe — evidence is GKD + one lab blog. Treat the efficiency numbers as directional, not settled. (This corrects an earlier over-strong “sleeper” framing.)
• Eats: teacher completions (off) / teacher-graded student rollouts (on). Requires a teacher genuinely better at your task.

Rejection-sampling FT (“RL without RL”)

Terminology landmine: “RFT” is overloaded. This chapter’s rejection-sampling FT (STaR/RAFT/ReST family, cheap, positives-only SFT) is a different thing from Reinforcement Fine-Tuning (the OpenAI/Fireworks product term for actual online RL/GRPO against a grader, expensive). Say “rejection-sampling SFT” to avoid accidentally speccing a GRPO run — see Contested edges §2.

• What: sample N completions from π_θ, keep verifier-accepted winners, SFT on them; iterate. The lineage: STaR (arXiv:2203.14465), ReST (arXiv:2308.08998), RAFT (arXiv:2304.06767), RFT (arXiv:2308.01825).
• Eats: your own verifier-passed trajectories — which for a CTF harness with a flag check you already generate. Reward must be ground-truth-verified against real tool/server output, never format-matched, or SFT on the resulting set trains confabulated flags (Contested edges §5).
• Policy: on-policy data, SFT update. It’s the first-order special case of policy gradient (reward∈{0,1}, upweight winners).
• When: an execution gap, and you have a verifier but no stronger teacher. Cheapest on-policy move; reuses your SFT pipeline. “Iterate” here is the micro version of the pipeline-level loop question — see is the recipe a loop? for whether you restart from base or continue from checkpoint each round.
• Gotcha (measurable graduation trigger): positives-only ⇒ policy-entropy collapse — fast early gains then plateau. GRPO’s real edge over it is not group-normalization (ablated → negligible) but discarding all-same-reward groups (implicit filtering). See “A Minimalist Approach to LLM Reasoning: From Rejection Sampling to Reinforce” (arXiv:2504.11343). Watch entropy; when it collapses, graduate to GRPO/RLVR.
• Production proof: explicit named stage in Llama 3 (arXiv:2407.21783) and DeepSeek-R1 (~800K rejection-sampled examples between its two RL stages, arXiv:2501.12948). Both are also the T1 proven-first picks ranked in proven-first ranking §4.

The kinds of SFT — it is the data, not the algorithm

Imitation named the family and its three methods — SFT, distillation, rejection sampling — as fixed presets on the on/off-policy axis. What it left open is the question that actually decides whether an SFT run helps or quietly teaches confabulation: not which algorithm (there’s only one, cross-entropy on target tokens), but which of several very different-shaped sources you point that algorithm at. This chapter names that hidden axis.

You’ve correctly sensed something the field doesn’t name clearly enough: SFT is one algorithm. Cross-entropy loss on (input → target tokens). That’s it — there’s no “SFT-v2” or “agentic SFT algorithm” that’s mathematically different from “instruction-tuning SFT.” What actually varies, and what actually determines whether your model gets better or gets confidently wrong, is a question the loss function itself is silent about: where did the target sequence come from?

This chapter names that hidden dimension precisely — on two orthogonal axes plus a third gate — maps your intuited “types of SFT” onto it (you’re missing two), and then answers the practical follow-on: given raw challenges + run logs, which shape of training row do you actually build, in what format, and how do you keep a ~27B dense model’s existing capability intact while you do it.

Framing: general research reference, common practice across the field — not pinned to any one project’s benchmark or numbers. Status: every arXiv id below was verified live (title, authors, abstract pulled from arxiv.org/OpenReview, not recalled from training memory); confidence is stated per section, and the taxonomy packaging itself (the 3-axis table) is a synthesis for legibility, not a quoted framework from any single paper — flagged where that matters.

1. “Cold-start SFT” names WHEN, not WHAT

If you’ve read Imitation or the recipe is a sequence, you’ve seen SFT called “the cold-start stage” — the thing that runs before RL, stage 0 of the pipeline. That’s a true statement about timing. It tells you nothing about what’s inside the training file.

This is exactly the gap you noticed. “Cold-start SFT,” “instruction-tuning SFT,” “agentic SFT,” “distillation SFT,” “rejection-sampling SFT” all compile to the identical loss:

loss = -log P(target_tokens | input_tokens)     # cross-entropy, every single time

The word before “SFT” in each of those phrases isn’t describing a different algorithm — it’s silently describing a different data-generating process that produced target_tokens. Two SFT runs with identical hyperparameters, identical base model, identical loss curve on paper can produce a model that generalizes cleanly or a model that’s confidently making things up, purely because of where the targets came from. That’s the variable this chapter names.

2. The two axes (plus a verifier gate)

Axis A — WHO produced the trajectory

Axis B — EXECUTED, or SYNTHETICALLY AUTHORED

This is the axis nearly everyone skips, and it matters more than Axis A for agentic data specifically.

• Executed / grounded — the trajectory came from an actual run against the real environment: real tool calls, real stdout/stderr, real pass/fail. The text you train on is literally what happened.
• Synthetically authored / ungrounded — an LLM (or a human) wrote what a solve “would look like” without ever running it. Tool outputs, error text, intermediate observations — all invented, and only resembling the real thing.

Why this beats Axis A in importance for your case: a synthetically-authored trajectory teaches “this is the shape a plausible tool output takes” — not “this is what actually comes back when I run this exact command.” Train on enough of that and the model learns to confabulate tool results instead of reacting to real ones. That risk is additive on top of, and independent from, who wrote it — a stronger-teacher-authored-but-never-run trajectory carries it exactly as much as a human-authored one.

Axis C — VERIFIER-FILTERED, or not

Orthogonal to both axes above: was there a check (unit test, flag match, exact-match grader, human review) that discarded failed attempts before they hit the training set?

• Filtered — only successes (or high-scoring attempts) survive. This is what turns “the model generated something” into “the model generated something correct, and only correct gets reinforced.”
• Unfiltered — everything generated goes in, right or wrong. Rare on purpose: it teaches wrong patterns exactly as strongly as right ones unless something else downweights them.

graph TD
  A["A training row's target sequence"] --> B{"Who produced it?<br/>human / teacher / self / other-model"}
  A --> C{"Executed against the<br/>real environment, or authored?"}
  A --> D{"Verifier-filtered<br/>before it entered the set?"}
  B --> E["Determines style/vocabulary<br/>match to your model"]
  C --> F["Determines grounding —<br/>does it teach real tool physics<br/>or a plausible-looking shape?"]
  D --> G["Determines noise floor —<br/>does the gradient point at<br/>correct behavior only?"]

3. The taxonomy — your four intuited types, plus the two you missed

Confidence: High on every arXiv id/abstract (verified live). High on the axis-based framing matching each cited paper’s actual mechanics. Medium-high on the specific “3-axis” packaging — it’s a synthesis for engineer-legibility, not a quoted taxonomy from one source. No claim above is contested in the literature: grounded-vs-synthetic for agentic data, and quality-over-volume for SFT curation, are both widely-replicated findings.

Cross-reference: Imitation already names SFT / distillation / rejection-sampling as three on/off-policy presets. This table is the same territory sliced one level finer — Axis A+C here map onto that chapter’s “who supplies the demonstrations” split; Axis B (executed vs. authored) is the addition this chapter contributes, and it’s specifically what makes agentic data different from plain instruction data.

4. The agentic crux: synthetically-authored trajectories teach confabulation

This is the highest-confidence, first-principles part of this chapter, and it’s the reason Axis B gets its own section. (Related but distinct: Contested §5 covers the reward-verification side of confabulation — a loose reward matcher firing on fabricated FLAG{…}-shaped output at RL/eval time; this section is about the training-data side — how ungrounded SFT rows teach the model to produce that fabricated shape in the first place.)

If you (or an LLM) write a transcript like:

> run_exploit(target)
[tool result] Shell obtained. flag{a1b2c3...}

without ever actually running run_exploit against a real target, the text after [tool result] is fiction that merely looks like a tool result. Train on enough of this and the model learns the shape of a successful transcript — not the conditional relationship “IF I emit this exact call, THEN this category of output comes back.” At inference time, when the real tool returns something it’s never seen worded that way, the model either ignores it or fabricates the next step to match the shape it memorized. This is a training-induced version of what people call tool-output hallucination.

This is exactly why FireAct (arXiv:2310.05915) and AgentTuning (arXiv:2310.12823) — and the whole rejection-sampling family (RAFT/STaR/ReST/ReST-EM) — insist on trajectories from real execution. It’s also why DeepSeek-R1 (arXiv:2501.12948) — verified live: its own abstract describes cold-start SFT → RL → rejection sampling on the RL checkpoint to build ~600K new SFT rows, keeping only correct completions → a second SFT/RL pass — is the closest published description of “run your own agent, keep the verified wins, that’s your next SFT set.” A run-log corpus like yours, filtered to flag-verified wins, is already sitting in the gold-standard cell of the table above: executed and verified.

Concretely, side by side — same challenge, two data sources:

// GROUNDED — real tool call, real stdout, keep it (Axis B: executed; Axis C: verified)
{"role": "tool", "tool_call_id": "call_2",
 "content": "{\"user\":\"admin\",\"email\":\"admin@corp.local\",\"notes\":\"flag{1dor_pwn3d_9f3a}\"}"}

// SYNTHETICALLY AUTHORED — an LLM imagined this JSON shape; it never ran. Looks identical. Is fiction.
{"role": "tool", "tool_call_id": "call_2",
 "content": "{\"user\":\"admin\",\"email\":\"admin@corp.local\",\"notes\":\"flag{1dor_pwn3d_9f3a}\"}"}

You cannot tell these apart by reading the row. That’s precisely the danger — the failure mode is invisible at data-authoring time and only shows up as unreliable, confabulating behavior after training, on challenges the fabricated shape didn’t happen to match.

5. What shape of training row you actually build

Two raw ingredients: challenge definitions (prompt, tools, grading/flag check) and run logs (transcripts of an attempt — yours or a stronger model’s — ending in pass/fail). From these you can derive up to four different shapes of training row. They are not interchangeable — each teaches a different capability, and picking the wrong one is why “I added more data and the score didn’t move.” (The fuller per-method — not per-shape — data-object mapping, including preference pairs and RL prompts, lives in Method → Data; this section is one level more granular: which row-shape you extract from the same raw material for SFT specifically.)

5.1 Agentic trajectory — the main event

Teaches: the actual skill being measured — multi-step tool use under uncertainty, recovering from a bad tool result, deciding when to stop and answer.

Derive it from: a run log that ended in a verified flag/success. Slice it into the literal sequence (system, user) → (assistant: think + tool_call) → (tool: tool_result) → … → (assistant: final answer). Do not re-author it — use the exact text sent to/from the model and the tool, including the exact JSON schema your inference stack uses.

Worked example — one training row (structure is standard across current open-weight tool-calling formats; the exact field names must come from your model’s chat template, never assumed):

{
  "messages": [
    {"role": "system", "content": "You are a security agent. Tools: run_command, read_file, submit_flag. Find the flag."},
    {"role": "user", "content": "Challenge: web app at 10.0.0.5:8080 has an IDOR. Find the flag."},
    {
      "role": "assistant",
      "content": "The endpoint /api/user/{id} likely doesn't check ownership. I'll try incrementing the id.",
      "tool_calls": [
        {"id": "call_1", "type": "function",
         "function": {"name": "run_command", "arguments": "{\"cmd\": \"curl http://10.0.0.5:8080/api/user/2\"}"}}
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "{\"user\":\"admin\",\"email\":\"admin@corp.local\",\"notes\":\"flag{1dor_pwn3d_9f3a}\"}"
    },
    {
      "role": "assistant",
      "content": "Found it in the notes field.",
      "tool_calls": [
        {"id": "call_2", "type": "function",
         "function": {"name": "submit_flag", "arguments": "{\"flag\": \"flag{1dor_pwn3d_9f3a}\"}"}}
      ]
    },
    {"role": "tool", "tool_call_id": "call_2", "content": "{\"correct\": true}"}
  ]
}

Loss mask: compute cross-entropy on assistant turns only — mask system/user/tool tokens. You want the model to produce the call and the final answer, not to predict the environment’s output (that’s the environment’s job, and training on it teaches the confabulation failure mode from §4 by a second route). This is a silent, common bug — check your framework’s loss_mask / train_on_inputs=False setting explicitly.

Where this comes from: FireAct §3 (arXiv:2310.05915) and AgentTuning (arXiv:2310.12823) are doing exactly this — turning ReAct-style (thought, action, observation) transcripts into SFT rows.

5.2 Knowledge / Q&A

Teaches: static facts (“what does this CVE affect,” “what’s the flag format”). Does not teach tool use or multi-step behavior.

Derive it from: challenge writeups/solutions, not transcripts — extract the underlying fact and turn it into a direct question-answer pair (Self-Instruct-style, arXiv:2212.10560).

{"messages": [
  {"role": "user", "content": "I found an endpoint at /api/v1/debug/config on a web service. What's the security implication?"},
  {"role": "assistant", "content": "Debug/config endpoints left enabled outside dev commonly leak env vars (API keys, DB creds), internal service URLs, and stack traces. Next step: GET it directly and grep the response for 'key', 'secret', 'token', 'flag'."}
]}

When to pick this: only if error analysis shows the model doesn’t know a fact — not because it can’t act on tools. For a model already around a third on an agentic benchmark, this is rarely the real bottleneck; most failures at that stage are behavioral (bad sequencing, giving up early, not verifying before submitting). Prefer a retrievable tool/RAG lookup over baking facts into weights when the “knowledge” really is just a fact — reserve weight-training for facts that are actually reasoning patterns in disguise (e.g. “why is a debug endpoint dangerous” needs to generalize to endpoints it hasn’t seen named; a CVE number does not).

5.3 Tool-usage / function-calling pairs

Teaches: the mechanical skill of correctly calling a tool given its schema — right name, right argument types, right JSON — decoupled from multi-step planning.

Derive it from: isolated (tool_call → tool_result) slices out of run logs, or synthesize directly from your tool schemas — this is the one row-shape that’s safe to author ungrounded, because there’s no tool result being faked, only the call.

{"messages": [
  {"role": "system", "content": "Tools: run_nmap(target, ports), http_get(url), sqlmap_scan(url, param)."},
  {"role": "user", "content": "Check if the 'id' parameter on http://10.10.1.4/product?id=1 is SQL-injectable."},
  {"role": "assistant", "content": null,
   "tool_calls": [{"id": "call_1", "type": "function",
     "function": {"name": "sqlmap_scan", "arguments": "{\"url\": \"http://10.10.1.4/product?id=1\", \"param\": \"id\"}"}}]}
]}

When to pick this: run logs show the model getting the idea right but the call malformed — wrong argument names, hallucinated parameters, wrong JSON. A narrow, cheap, low-risk fix because it’s format-only.

5.4 CoT reasoning — usually inline, not a separate pipeline

Teaches: deliberation before commitment — reading an ambiguous observation and reasoning about which of several plausible next actions is right, before calling the tool.

Derive it from: the same run logs as §5.1. If a log doesn’t contain live reasoning text (just tool calls with no thinking-out-loud), you have two options: leave it as-is (output-only agentic SFT is what most of the field does), or use a stronger model to retroactively annotate a rationale for an already-executed, already-correct trajectory — STaR’s “rationalization” mode (arXiv:2203.14465): generate the reasoning after seeing the correct action. This keeps the action grounded while making the reasoning explicit.

{"role": "assistant",
 "content": "<think>Port 8080 is non-standard and often hosts admin/debug panels rather than the main app on 80 — check it first.</think>",
 "tool_calls": [{"id": "call_2", "type": "function",
   "function": {"name": "http_get", "arguments": "{\"url\": \"http://10.10.1.4:8080/\"}"}}]}

Don’t over-engineer a fourth pipeline for this — in a well-formed §5.1 row, the reasoning is already inline in the content field preceding tool_calls.

5.5 Pick by diagnosed gap

In practice, for moving an agentic benchmark score meaningfully, the overwhelming majority of your budget should be §5.1 — full, executed, verifier-filtered trajectories. That’s what FireAct and AgentTuning both do, and it’s the shape the entire rejection-sampling literature is built around: your own model’s verified wins are simultaneously on-policy, executed, and filtered — the single highest-signal, lowest-risk data you can generate, and it’s a byproduct of running the benchmark at all.

6. Data selection — “only accept what it needs”

A dense ~27B model already has broad capability. A naive “dump every transcript in” full fine-tune risks overwriting general capability to fit a narrow domain — every gradient step nudges every weight, not just the “agentic” ones, and a 27B model has far less spare parameter capacity to absorb noise without collateral damage than a much larger dense or MoE model. This section is the short version; Data mixing, ratios & not forgetting how to think is the long version (replay ratios, LoRA-vs-full-FT forgetting mechanics, reasoning-trace collapse).

Evidence base (verified):

• LIMA (arXiv:2305.11206) — 1,000 carefully curated human-quality pairs matched/beat models trained on orders of magnitude more data. Their own ablation: doubling the set size did not improve quality; quality-filtering the same size did. “Superficial Alignment Hypothesis”: most capability comes from pretraining; SFT mainly teaches which subdistribution of its own behavior to surface.
• AlpaGasus (arXiv:2307.08701) — LLM-judge-filtering Alpaca’s 52K examples down to ~9K high-quality ones produced a better model, 5.7× faster to train. Bad examples actively hurt, not just waste compute.
• IFD / Cherry selection (arXiv:2308.12032) — a cheap, no-external-judge score (the model’s own loss on the response given the instruction, vs. without it) that ranks training value per row; kept only 10% of a dataset and beat training on the full set.
• LESS (arXiv:2402.04333) — gradient-similarity selection of ~5% of a dataset, targeted at the specific downstream capability you want, outperformed the full dataset for that capability. The most directly relevant paper to “which subset of my logs actually moves the gap I diagnosed.”
• DEITA (arXiv:2312.15685) — formalizes selection along complexity/quality/diversity axes; a 6K-example selected set matched/beat 10× more unfiltered data.
• LoRA Learns Less and Forgets Less (arXiv:2405.09673) — full fine-tuning learns more of the target domain but measurably forgets more of everything else; LoRA learns less but preserves out-of-domain behavior — a real, measured trade-off. Curation is the data-side lever for the same problem LoRA solves on the optimizer side; they’re complementary, not redundant.
• Scaling Laws for Forgetting (arXiv:2502.06042) — fine-tuning on a narrow domain measurably increases pretraining-distribution loss as training proceeds, scaling with fine-tune volume and model size; injecting as little as ~1% general/pretraining-like data into the mix meaningfully arrests it. A commonly-cited practical default in adjacent replay literature (arXiv:2605.15220) puts the standard mitigation baseline nearer 10% replay.
• Decontamination (arXiv:2307.09288 §A.6, the Llama 2 report) — token n-gram overlap (>10 tokens shared between an eval sample and the training set) is a simple, reproducible check to confirm your SFT rows don’t secretly contain your benchmark’s own held-out challenges. This is a live, underdiagnosed risk field-wide (arXiv:2406.04244, arXiv:2404.00699).

Concrete curation moves, in order:

1. Verifier-filter first, always — only train on flag-verified/passed runs. Removes the majority of harmful noise in one step (§2–4 above).
2. Prefer your own on-policy verified rows over teacher/off-policy rows when both exist — smaller, more targeted weight movement (see Foundations: the one axis that predicts everything).
3. Score what’s left with IFD, then judge-filter the survivors — cheap first pass, quality second pass.
4. Deduplicate and cap per-archetype volume — if 40% of your logs are the same challenge pattern, training on all of them teaches a shortcut, not a skill.
5. Prefer fewer, harder, correctly-solved trajectories over many easy ones — LIMA’s quality-over- count finding, applied.
6. Replay ~1–10% general-instruction data alongside the agentic corpus if forgetting is a live concern.
7. Decontaminate against your eval set before training, not after a suspiciously good score.
8. Consider LoRA/QLoRA if “don’t overwrite what the model already knows” is a harder constraint than “maximize domain score” — arXiv:2405.09673 gives the actual trade-off curve, not an assumption.

7. Decision aid — which SFT data source for my case

graph TD
  A["Diagnosed failure mode<br/>(see diagnosis/framework.md)"] --> B{"Does the correct behavior<br/>ever appear in the model's<br/>own output, even rarely?"}

  B -->|"Never — knowledge/skill genuinely absent"| C{"Do you have a<br/>stronger teacher<br/>with real tool access?"}
  C -->|Yes| D["Teacher solves for real,<br/>you distill the EXECUTED transcript<br/>(FireAct / AgentTuning style)"]
  C -->|No| E["Human demos, small + curated<br/>(LIMA-style, few hundred rows)"]

  B -->|"Sometimes — partial-solve regime"| F{"Do you have a<br/>verifier / pass-fail check?"}
  F -->|Yes| G["Rejection-sampling SFT:<br/>sample your OWN model,<br/>keep verifier-passed wins<br/>(RAFT / STaR / ReST family)"]
  F -->|"Yes + want student to<br/>stay on its own distribution"| H["On-policy distillation (GKD):<br/>student samples, teacher grades"]

  B -->|"Mis-ranked, not absent"| I["Not an SFT problem —<br/>see preference methods<br/>(DPO/KTO)"]

  G --> J{"Is the row a<br/>full multi-step trajectory?"}
  J -->|Yes| K["Full trajectory (5.1) —<br/>the bulk of your budget"]
  D --> J

  style E fill:#fdd
  style D fill:#dfd
  style G fill:#9f9

Reading it: the green path (your own verified rollouts, or a teacher’s real-execution transcripts) is the safe default. The red path (human demos) is fine but expensive and caps out fast — reach for it only when neither a teacher nor your own model can produce the behavior at any sampling temperature. Nowhere on this tree is “synthetically author a plausible-looking trajectory” a legitimate node for tool-call data — that’s the failure mode §4 names, not a fifth branch.

Cross-links

• Foundations: the one axis that predicts everything — the on/off-policy distinction this chapter’s Axis A is built on, and why off-policy imitation is structurally blind to execution gaps.
• Imitation — SFT · distillation · rejection sampling — the per-method view of the same territory (what/eats/policy/when/gotcha/cite); this chapter is the per-data-source view one level under it.
• The recipe is a sequence, not a pick — “cold-start SFT” as a pipeline stage; §5 there (“the synthetic-trajectory bootstrap”) is the fuller treatment of the confabulation risk from §4 of this chapter.
• Data mixing, ratios & not forgetting how to think — the long version of §6: replay ratios, why LoRA doesn’t fully solve forgetting, reasoning-trace collapse mechanics.
• Method → Data (your real bottleneck) — the per-training-method data-object table (SFT, DPO, KTO, GRPO, agentic RL) this chapter’s §5 sits one level under, for SFT specifically.

Verified citation registry

Confidence calibration: High on every id/abstract above (verified live on 2026-07-02, not recalled from training memory). High on the grounded-vs-synthetic distinction for agentic data and on quality-over-volume for curation — both widely replicated, not contested. Medium-high on the 3-axis taxonomy packaging specifically (§2–3) — it correctly describes each cited paper’s mechanics but is this chapter’s synthesis, not a quoted framework from one source; trust the primary papers’ own mechanics over the packaging if they ever appear to diverge. Explicitly not grounded on any academic cybersecurity-LLM training paper — every citation above is general frontier/ML data-construction or data-selection literature, per this chapter’s brief.

Hint-guided bootstrapping — put the walkthrough in the prompt, then train it away

The kinds-of-sft taxonomy just gave you the axes a training row is built from — who produced the trajectory, executed vs. synthetically authored, verifier-filtered or not. This chapter takes one specific point on that grid — on-policy, executed, hint-seeded — and follows it end to end: from a plausible cold-start idea, to the four-decade-old literature it belongs to, to its measured failure modes, to a concrete recipe.

You’ve independently arrived at an idea for cold-starting a sparse-reward agentic task: a fresh policy solves ~0% of the hard cases, so there’s no gradient, nothing to learn from. Your fix — put the solution/walkthrough for that exact task in the system prompt, let the model execute the task for real against the actual environment (real tool calls, real stdout, a real pass/fail check), then mask the walkthrough out of the recorded trajectory and fine-tune the model, supervised, to produce that same executed trajectory when given only the bare task. Your own framing: it’s an exploration task; hand the agent the map, harvest its own walk to the goal, then take the map away.

This chapter’s first job is to tell you: this is not novel, and that’s good news. You’ve reinvented a family the literature has been building out under at least four different names since 2009, each naming a different facet of the same operation. The second job is the harder one — the family has real, measured failure modes, and at least one of them lands directly on the specific design choice (“full walkthrough,” not an abstract cue) you proposed. This chapter names the family, shows the mechanism precisely, tables the evidence, and gives you the honest risk register before you spend compute on it.

Framing: general research reference — the technique family, not any one project’s pipeline. Status: every arXiv id below was verified live against arxiv.org abstract pages (title, authors, date cross-checked against body text, not a bare crawler title) during the research pass this chapter draws on; confidence is stated per section, and — per NuRL’s own ablation (§4) — the chapter does not conclude “this works, go build it.” It concludes “this is a real family with one specific, well-evidenced caveat that lands on your exact design,” which is a more useful and more honest answer.

1. The idea, and its real names

Four bodies of work converge on the same four-step move, each catching a different piece of it:

• STaR-rationalization (Zelikman, Wu, Mu, Goodman, arXiv:2203.14465) — the generate-under-hint-then-mask mechanism at the level of a single scalar hint: append the correct final answer to the prompt, let the model generate a rationale that reaches it, train on the (question → rationale) pair with the hint stripped out.
• Context / prompt distillation (Askell et al., arXiv:2112.00861; Snell, Klein, Zhong, arXiv:2209.15189) — the same mechanism scaled up to a rich hint: a whole instruction block, scratchpad, or system prompt, internalized into the weights by training the model to reproduce hint-conditioned outputs without the hint present.
• Hint-guided exploration for sparse reward (NuRL, arXiv:2509.25666; HiLL, arXiv:2604.00698; ReGFT, arXiv:2603.01223) — the RL-training-loop framing: privileged guidance is what recovers a non-degenerate reward/advantage distribution when every rollout in a group currently fails, and the field has explicit vocabulary and metrics for whether that guidance was genuinely necessary and whether it transfers once removed.
• Learning Using Privileged Information / asymmetric actor-critic (Vapnik & Vashist, Neural Networks 2009, DOI:10.1016/j.neunet.2009.06.042; Pinto, Andrychowicz, Welinder, Zaremba, Abbeel, arXiv:1710.06542) — the formal name for “the teacher sees information at train time the student will never see at deployment,” which is the abstraction all three techniques above are instances of.

None of these is “solved” in the sense of a settled best-practice recipe. All four have real evidence they work, and all four have documented conditions under which the same recipe silently produces a model that looks trained but hasn’t actually learned the thing you think it has. That’s the substance of this chapter.

2. The mechanism: exploration compiled into supervision

What the hint is doing. An untrained (or weak) policy facing a sparse/binary reward has to find the solution by search — an exponentially large space of tool calls and token sequences, almost none of which lead to a verified success. A hint (an answer, a scratchpad, a full walkthrough) collapses that search: it narrows the space down to a specific, concrete, correct path at generation time. Once that path exists as a real (input → output) pair, imitating it is a supervised problem — dramatically cheaper and more stable than making the model rediscover the same path from scratch via RL exploration in a sparse-reward regime. This reframing — “the hint pays the exploration cost once, then supervision harvests it” — is the single sentence that unifies STaR-rationalization, context distillation, and hint-guided RL.

Grounded-and-executed beats confabulated-and-narrated. There are two very different ways to produce the (task, trajectory) pair the hint enables. One: ask a model to write a plausible-looking solve — no environment involved, nothing actually ran. Two: give the model the hint and let it act against the real environment — real tool calls, real stdout/stderr, a real pass/fail check. Your proposal insists on the second. This is a genuine, load-bearing design choice, not a detail: a narrated rationale can confabulate a result that never happened (this is the exact failure mode the field calls unfaithful chain-of-thought — biasing a prompt toward an answer produces plausible-but-non-load-bearing reasoning text, dropping accuracy up to 36% on BIG-Bench Hard when the bias points at a wrong answer, Turpin, Michael, Perez, Bowman, arXiv:2305.04388) — while an executed, verifier-checked trajectory structurally cannot silently hallucinate its own success. That’s real protection against one failure mode. It is not, on its own, protection against the other one (§4) — a real, verified trajectory can still have been shaped by the hint in ways that don’t transfer.

Mask-the-hint is the training-time move, formalized twice.

Snell et al. give the exact loss for the general case — teacher (with hint) supervises student (without hint), same underlying model, hint stripped by an extractor f before the target is built:

L(θ_student) = E_x~D [ E_y~P_θteacher(·|T_teacher(x)) [ log P_θstudent( f(y) | T_student(x) ) ] ]

T_teacher(x) = [hint/instructions/walkthrough] + x        # what generates y
T_student(x) = x                                            # what the trained model sees
f(y)         = y with the hint/scratchpad stripped out      # what the trained model must produce
θ_teacher stays frozen — it's the same model, hint present, at generation time only

STaR/TRICE give the same move a probabilistic reading, one level more mechanistic. Treat the rationale (or trajectory) z as a latent variable; the real objective is the marginal likelihood of the correct outcome y given the task x, marginalizing over all possible z: log p_θ(y|x) = log Σ_z p_θ(y|x,z) p_θ(z|x). You can’t compute that marginal directly — you need a sample from the posterior p_θ(z|x,y), “the trajectory distribution given that we already know it succeeds.” The hint-conditioned generation step is exactly an approximation of drawing from that posterior. TRICE (Phan et al., arXiv:2312.02179) formalizes STaR-rationalization as one-sample, uncorrected MCMC against that posterior, and proves it is a biased stochastic-EM estimator: the true gradient of the marginal log-likelihood is p_θ(y|x) ∇_θ log p_θ(y|x) — weighted by how likely the unhinted model already was to succeed. That weighting means the gradient signal shrinks toward zero exactly on the hardest examples — the ones where the hint is doing the most work relative to the model’s own competence. This is the formal version of “did I really teach it something, or did the hint just do the work” — and it’s the mathematical spine under NuRL’s elicit-vs-expand framing (§4).

LUPI names the shape of the whole thing. Vapnik’s framing: training triplets (x_i, x*_i, y_i) where x* (privileged info) is available only at train time; the learned function f: X → Y must use only x at deployment. Your writeup is x*; the executed-and-masked trajectory is the (x, y) the model is trained on. The theoretical payoff, when privileged info is genuinely informative, is a faster generalization-error convergence rate (O(1/n) vs O(1/√n), the “Oracle SVM” analysis) — but the empirical follow-up is candidly mixed: randomly-generated “privileged” features can perform similarly to genuinely meaningful ones in some SVM+ settings (Serra-Toro, Traver, Pla, Pattern Recognition Letters 2014) — privileged-at-train-time is not automatically privileged-and-useful. The asymmetric actor-critic lineage (Pinto et al., arXiv:1710.06542) is the deep-RL sim-to-real instance of the same idea done right: the critic sees full simulator state, the actor only ever sees what it will have at deployment — and it’s now the standard recipe precisely because the privileged channel there (ground-truth physics/pose) is verifiably informative, not noise.

3. Named techniques — what each contributes, and where each breaks

4. Failure modes — named, measured, and the one that lands on your exact design

Three distinct risk classes, not one — conflating them leads to the wrong mitigation.

(a) Shortcut learning / Clever-Hans / hint-copying

The general phenomenon: a decision rule that performs well on the training distribution by exploiting a cue that isn’t the actual task (Geirhos et al., “Shortcut Learning in Deep Neural Networks,” arXiv:2004.07780 — 1650 citations, named for Clever Hans, the horse that appeared to do arithmetic by reading its handler’s involuntary cues). In NLP specifically: BERT-class NLI models adopting fallible syntactic heuristics that collapse on a controlled challenge set (HANS, McCoy, Pavlick, Linzen, arXiv:1902.01007); the same effect on COPA, with the sharp finding that the same recipe can produce a Clever-Hans learner or a genuine one depending on architecture (Kavumba et al., arXiv:1911.00225); a detection technique — strip words from the input while confidence stays unchanged, meaning the model wasn’t reading the content at all (RAWR, Feng et al., arXiv:1804.07781).

The version specific to this proposal is sharper still. Peng et al., “Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation,” arXiv:2602.14469 studies exactly “synthesize a trace from a known (query, answer) pair” — precisely “write the trajectory knowing the destination” — and formalizes “answer as cognitive anchor” across lexical/entropic/probabilistic axes. Their most important negative result: the naive mitigation, “tell the model to ignore the answer while generating,” backfires — it reduces surface lexical overlap with the hint but increases entropic/probabilistic anchoring, meaning the model leans on the hint just as much internally, only hides it better. Direct implication: masking the hint text out of the trajectory is necessary but not sufficient. The model’s tool-call sequence itself — target selection, the order it tries things — can still be suspiciously well-aimed relative to what an unhinted agent would ever discover, even with every trace of the writeup’s literal text gone from the training row.

(b) Distribution mismatch / off-policy exposure bias

Named explicitly by OPCD: the classical Askell/Snell recipe trains on a frozen, once-sampled teacher-conditioned completion — the student never generated it — which produces exposure bias (trained on teacher-shaped sequences, must autoregress its own at test time) and mode-covering forward-KL (spreads mass across all teacher behaviors instead of committing to a coherent policy). ReGFT names the same failure from the SFT side: raw expert/reference text sits outside the model’s own reasoning distribution, and training on it directly teaches imitation of an alien voice, not the underlying skill — their explicit fix is to have the model itself generate the continuation, guided-but-not-copying the reference. Your proposal’s insistence on executing the trajectory rather than distilling the writeup’s text directly is already the correct mitigation here, by construction — keep it.

(c) Elicit-not-expand

The sharpest open question, and the one with a formal answer (§2’s TRICE result) and an empirical instrument (NuRL’s pass@1024 metric): does hint-guided training expand what the policy can do, or does it just make an already-latent, rarely-sampled solution more probable? NuRL’s headline finding is that done right (self-generated, abstract hints, gated on genuine 0%-pass-rate difficulty, applied late — after unhinted training has already converged) it does raise the model’s true ceiling (pass@1024), which vanilla GRPO cannot move. But their own ablation is the load-bearing caveat for you specifically: revealing the gold answer/full solution directly hurt performance relative to an abstract hint — a full CTF walkthrough sits at exactly the dangerous end of that spectrum, closer to “here’s the answer” than to “here’s the abstract cue.” This is not a reason to abandon the idea; it’s the single most concrete, most actionable finding this literature offers against the specific form (“full walkthrough in system prompt,” not “abstract hint”) you proposed.

Detection — ablate the hint, don’t trust the narration

• HiLL’s hint-reliance check, operationalized cheaply: re-roll the same task from the same checkpoint without the writeup, N times; check whether the hinted trajectory’s key decision points (target selection, tool choice, the specific exploit path) show up in any of the unhinted rollouts. Low overlap = high reliance = the hint did the work, not the model.
• NuRL’s pass@1024-style check: hold out tasks that never had a writeup generated for them at all (not just masked at inference — never authored, so the eval set itself can’t leak task-specific info from writeup-authoring). Compare the SFT’d model’s pass rate on those to the pre-SFT base model’s. A genuine capability gain shows up there; a hint-shortcut gain does not.
• Do not trust the trajectory’s own narrated reasoning as evidence of non-reliance. RL optimization pressure is documented to increase reliance on prompt-present hints without proportionally increasing disclosure of that reliance in the model’s own chain-of-thought — the model can use the hint and simply not say so. The ablation checks above are load-bearing precisely because introspection is not sufficient evidence.

Mitigate

1. Mask thoroughly, then audit for residual fingerprints — not just “is the writeup’s literal text gone,” but whether the tool-call order and target selection look anomalously well-aimed compared to the model’s other, unhinted successful trajectories on structurally similar tasks (Peng et al.’s finding that masking-only backfires against internal anchoring).
2. Fade the hint, don’t cut it binary. Multiple independent lineages converge on the same fix: graded hint depths rather than one full walkthrough (Zhang et al., “Multi-level Stepwise Hints,” arXiv:2507.02841); scheduling easy-to-hard curricula to prevent overfitting to the assistance signal (E2H Reasoner, arXiv:2506.06632; AdaRFT, arXiv:2504.05520); and explicit annealing of an auxiliary demonstration loss as the policy’s own competence overtakes the demonstrator (Nair, McGrew, Andrychowicz, Zaremba, Abbeel, arXiv:1709.10089) — the clearest classical precedent for “hint fading” as a design pattern, not a novelty.
3. Verify unaided, on tasks that never had a hint at all, per the detection section above — the single check that actually distinguishes expansion from elicitation.

5. Verdict, and the concrete recipe

Honest verdict. The mechanism is real and well-precedented across four independent lineages spanning 2009–2026; “hint-conditioned execution, then strip-and-train” reliably converts intractable exploration into tractable supervision, and grounding it in a real, verifier-checked environment (rather than a narrated rationale) is a genuine, non-trivial improvement over the weakest member of this family (vanilla STaR-rationalization against a string-match check). But this is contested at the specific design point you proposed, not in general: the one paper in this family that directly ablates “full answer/solution vs. abstract hint” (NuRL) found the full-solution end of that spectrum hurts relative to an abstract cue, and no paper surveyed here tests the exact setting — a full external walkthrough, executed in a live tool-using sandbox, verified against a real target — end to end. Treat the mechanism as validated and the specific recipe (full walkthrough, not graded/abstract hints) as the open empirical risk to test, not to assume away.

The concrete recipe, incorporating the mitigations above rather than the naive one-shot version:

graph LR
  A["Writeup / solution<br/>in SYSTEM PROMPT<br/>(privileged hint)"] --> B["Agent EXECUTES<br/>in the real environment<br/>(real tool calls, real output)"]
  B --> C{"Verifier check<br/>(e.g. flag_verified)"}
  C -->|"fails"| D["Discard, or keep as a<br/>hinted-failure negative<br/>(V-STaR-style contrast signal)"]
  C -->|"passes"| E["MASK the writeup out of<br/>the recorded trajectory"]
  E --> F["Hint-reliance filter:<br/>re-roll unhinted N times,<br/>drop trajectories with<br/>near-zero unhinted overlap"]
  F --> G["SFT: task-without-hint<br/>-> executed, masked trajectory"]
  G --> H["FADE hint strength across<br/>a curriculum: full walkthrough<br/>-> abstract cue -> none"]
  H --> I["VERIFY UNAIDED on held-out<br/>tasks that never had a<br/>writeup authored at all"]
  I -->|"gap holds vs. pre-SFT base"| J["Genuine capability expansion"]
  I -->|"gap closes / vanishes"| K["Hint-shortcut — degrade the<br/>hint toward abstract, re-run"]

Reading it end to end: the loop is not “run this once and trust the flag check.” The flag check (step C) rules out confabulation, which is real but is only one of the two failure modes this family documents. Step F and step I are what rule out the other one — hint-copying and elicit-not-expand — and neither is optional if you want to trust the resulting model’s behavior on a task it’s never seen a writeup for, which is presumably the entire point of doing this at all.

Cross-links

• The kinds of SFT — it is the data, not the algorithm — this chapter’s masked, executed trajectory is a specific instance of that chapter’s Axis A/B/C (self-generated, executed, verifier-filtered) — the highest-signal cell in that taxonomy, with one extra wrinkle (the trajectory was generated under privileged information) that chapter doesn’t cover and this one does.
• Method → Data (your real bottleneck) — hint-guided bootstrapping produces the same “full trajectory” data object that chapter’s SFT row already names; the difference is entirely in the generation process (hint-conditioned, then masked), not in the shape of the resulting row.
• RL that creates value — long-horizon · exploration · reasoning · novelty — the [E]/[N] exploration and novelty-tagged techniques there are the RL-native cousins of the same problem this chapter solves via a one-shot SFT harvest; read that chapter for the online-RL version of “recover a gradient when a group is all-fail.”
• Is the recipe a loop? — the fade-the-hint curriculum in §5 (full walkthrough → abstract cue → none) is a specific instance of that chapter’s question: does a stage run once or get revisited, and from which checkpoint does the next hint-level start.
• Contested edges & landmines — the elicit-not-expand tension named here in §4(c) is the same fault line that chapter’s §1 covers for vanilla RLVR (pass@1 vs. pass@k, base-model boundary); this chapter’s TRICE result and NuRL’s pass@1024 check are the SFT-side and hint-injection-side versions of that same unresolved question.

Verified citation registry

Confidence calibration: High on every id/abstract above (verified live, not recalled from training memory) and high on the mechanism itself (four independent lineages, 2009–2026, converge on the same generate-under-privilege → strip → supervise structure). Contested, explicitly: whether a full walkthrough specifically (as opposed to an abstract hint or partial reference prefix) is the right hint granularity — NuRL’s own ablation says it isn’t, and no paper surveyed tests the exact executed-in-a-live-sandbox setting end to end, so treat that as the open empirical question, not a settled negative. Several cited ids are very recent (2602.x–2606.x, weeks to months old at verification time) — their mechanisms and diagnostics (hint reliance, teacher flip rate, pass@1024 checks) are more load-bearing for a design decision than their specific benchmark numbers, which have not yet accumulated independent replication. Not grounded on any academic cybersecurity-LLM paper — every citation is general ML/RL data-construction, exploration, or shortcut-learning literature, per this chapter’s brief.

Teaching a tool, teaching recon — the data behind a skill

You want to teach a model two different things: (a) a specific tool with a fixed signature — grep, the Vercel CLI, a REST endpoint — and (b) a search methodology — “explore many things to find some things,” a grounded decision tree that fans out and prunes. Both are “tool use” in the loose sense. They are not the same training problem, and — this is the crux — for neither one does the answer route through Q&A. This chapter is about the DATA OBJECT: what shape of row you write to disk (or generate online), how it gets made, whether the tool was actually executed to produce it, and which technique it pairs with.

Framing: general research reference, drawn from seven sources (Toolformer, ToolLLM/ToolBench, APIGen/xLAM, Gorilla/DocPrompting, terminal-CLI-agent literature, search/recon-RL literature, and a procedural-vs-declarative synthesis). Every arXiv id below was cross-checked live 2026-07-02 against the source notes; only ids present in those notes appear here.

1. The axes, and the up-front verdict

Three bodies of vocabulary are in play, and disambiguating them is the first move:

• Tool learning / tool-use training / function-calling / tool-integrated reasoning (TIR). The literature for teaching a model to call a fixed-signature tool correctly — right name, right arguments, right moment, right reaction to the result. This is the “teach grep” problem.
• Search / information-seeking RL. The literature for teaching a model to explore — issue a query, read what comes back, decide whether to search again or answer, with no single correct path. This is the “teach recon” problem. It is tool-use RL’s cousin, not tool-use RL itself: the object being learned is a branching strategy, not a call signature.
• Procedural vs. declarative knowledge. The cognitive-science distinction that explains why the first two need different data than a knowledge base does. Declarative knowledge is “facts you can state” (what a flag does); procedural knowledge is “a skill you can execute” (when to reach for that flag, how to recover when it returns nothing). Training data that only states facts caps out at declarative recall no matter how much of it you have.

The up-front verdict, unanimous across all seven sources: Q&A (Q: "What does grep do?" A: "...") teaches facts about a tool. It does not teach skill with a tool — sequencing, argument selection under a real task, reading an observation to decide the next action, recovering from a failed call. Every source that runs the comparison (ToolBench, APIGen/xLAM, Gorilla/DocPrompting, procedural-vs-declarative) lands on the same conclusion: trajectories — ideally executed — are necessary for procedural transfer; Q&A is a legitimate thin seed for call-syntax, never the primary signal. Toolformer is the instructive edge case: it isn’t Q&A and isn’t a full multi-step trajectory either — it’s a self-supervised single (call, result) insertion into raw text, and it lands on the executed-trace end of the spectrum because the keep/discard filter is grounded in the real API output’s effect on next-token loss, not in a hand-written answer.

This is the same principle this book states elsewhere as “execution is the engine” — see kinds-of-sft.md for the general argument that whether a trajectory was executed against a real environment (rather than synthetically authored/narrated) is the single biggest lever on whether SFT teaches a transferable skill or teaches confabulation. This chapter is that argument applied specifically to tool-calling and search.

2. Teaching A TOOL (grep, the Vercel CLI, any fixed-signature callable)

2.1 The data-object ladder

Quality increases strictly down this ladder — each rung is necessary, none is sufficient on its own:

Rung 1 — doc-grounding / retrieval. Gorilla (arXiv:2305.15334) trains SFT with the oracle API doc concatenated straight into the prompt, so the model learns to read a doc rather than memorize one — this is what drops its hallucination rate to 6.98% vs. GPT-4’s 36.55% on TorchHub. DocPrompting (arXiv:2207.05987) goes one step further and trains a retriever + FiD generator jointly over (NL_intent, top-k_doc_paragraphs, reference_code). Neither executes the tool during data creation — the doc is trusted as ground truth. DocPrompting’s own ablation is the sharpest argument for why this rung exists at all: retrieving docs (versus retrieving worked examples) roughly doubles unseen-function recall (9.03% → 18.30%), because a doc generalizes to argument combinations an example never showed, but an example doesn’t. This rung is necessary — it’s where hallucinated signatures die — but it’s still one-shot code/call generation, not a decision loop.

Rung 2 — tool-call trajectories. ToolLLM/ToolBench’s canonical training row (arXiv:2307.16789):

{
  "instruction": "<user task>",
  "relevant_apis": ["API_1", "API_2"],
  "solution_path": [
    {
      "step": 1,
      "thought": "<reasoning about next action>",
      "api_name": "SearchByTitle",
      "parameters": {"title": "Batman", "country": "us"},
      "api_response": { "...real JSON response from actual API execution..." }
    },
    {
      "step": 2,
      "thought": "<reasoning based on step 1 response>",
      "api_name": "GetStreamingAvailability",
      "parameters": {"movie_id": "12345"},
      "api_response": { "...real response..." }
    }
  ],
  "final_answer": "<final output to user>"
}

Every API call in the 126,486-instance dataset is annotated by DFSDT (depth-first search with decision tree) — the annotator model is allowed to branch, backtrack on a failed call, and keep only working paths — which is what makes this trajectory-shaped rather than a single-shot call. The API documentation (name, params, example response) is attached at every step, so the model learns API semantics alongside call syntax, not call syntax alone. All 469,585 API calls behind ToolBench were made against real RapidAPI endpoints — not simulated REST mocks, which is exactly the gap ToolBench closes relative to prior single-tool, no-execution datasets like APIBench and ToolAlpaca.

Rung 3 — executed + verified traces. This is where every source converges as the non-negotiable floor for CLI-grade reliability. APIGen’s row (arXiv:2406.18518), using grep directly as the worked example:

{
  "query": "Check if the word 'error' appears in the log file at /var/logs/app.log",
  "function_calls": [
    { "name": "grep", "arguments": { "pattern": "error", "file": "/var/logs/app.log", "case_insensitive": false } }
  ],
  "observations": [
    { "type": "tool_output", "content": "[Line 42] ERROR: Connection timeout\n[Line 88] ERROR: Auth failed", "execution_time_ms": 14 }
  ],
  "expected_action": "The tool executed successfully; report the found errors.",
  "parallel_calls_supported": true
}

And xLAM/APIGen-MT’s multi-turn extension (arXiv:2409.03215, arXiv:2504.03601) — the worked example is literally the Vercel CLI:

{
  "conversation_id": "conv_001",
  "turns": [
    { "user_query": "Deploy my Vercel site",
      "function_calls": [ { "name": "vercel_list_projects", "arguments": {} } ],
      "tool_output": { "projects": [{"name": "my-app", "id": "proj_123"}] } },
    { "user_query": "Now deploy it to production",
      "function_calls": [ { "name": "vercel_deploy", "arguments": { "project_id": "proj_123", "environment": "production" } } ],
      "tool_output": { "deployment_url": "https://my-app.vercel.app", "status": "ready" } }
  ],
  "blueprint": {
    "task": "Deploy a web app to production via Vercel",
    "ground_truth_actions": ["list_projects", "deploy"],
    "dependencies": ["project_id must come from list_projects output"]
  }
}

Note the state-threading in the blueprint: project_id in turn 2 must literally be the value that came back from turn 1’s real execution — this is only checkable because the tool was actually run. APIGen’s gate that produces rows like this is a 3-stage verifier: (1) format check — does the call parse and match the schema (does the grep call have pattern/file, is case_insensitive a bool); (2) execution — actually run it in a sandbox (real grep, real Vercel SDK against a test project, real HTTP request) and capture stdout/exit code/latency; (3) semantic check — an LLM judge asks “does this (query, call, observation) triple actually make sense,” which is what catches a call that executed successfully but whose logged observation was hallucinated rather than the tool’s real return. From 3,673 executable APIs this produced 60,000 verified samples. Toolformer’s loss-reduction filter (arXiv:2302.04761) is the self-supervised sibling of the same idea — keep an inserted <API> call → result </API> block only if the real API result measurably lowers next-token loss (L_i^- − L_i^+ ≥ τ_f) — grounded in execution, gated by a different (automatic, LM-internal) signal rather than an LLM judge. Models trained on rung-3 data (APIGen/xLAM, 1B–70B) top BFCL and beat GPT-4o/Claude-3.5 on τ-bench — the empirical payoff for paying the execution cost.

2.2 How each rung gets MADE

2.3 The pipeline, end to end

flowchart LR
    A["Docs / --help / man pages\n(declarative floor)"] --> B["LLM proposes candidate\ninstruction + call"]
    B --> C["EXECUTE the tool for real\n(sandbox / test project / live CLI)"]
    C --> D{"Verify\nformat -> execution -> semantic\n(or: loss-reduction filter)"}
    D -- fail --> B
    D -- pass --> E["SFT on (instruction, trajectory,\nreal_observation) rows"]
    E --> F{"Plateau on\nheld-out tasks?"}
    F -- no, good enough --> G["Ship"]
    F -- yes --> H["Optional RL refinement\n(decomposed reward, on-policy\nexecution each rollout)"]

2.4 Worked recipe: teaching grep / the Vercel CLI

Step 1 — Docs-in-context (declarative floor)
  Data: {tool_name, signature, flags[], description, example_invocations} from `man`/`--help`/API docs.
  Purpose: kill hallucinated flags. Gorilla-style: concatenate the retrieved doc into the training
  prompt so the model learns to READ docs, not memorize them.

Step 2 — Executed CLI trajectories (procedural, grounded from the start — do not skip execution)
  Data object: {instruction, [{thought, command, real_stdout, real_stderr, exit_code}]_1..n, final_answer}
  How made: propose a command for a realistic task ("find all TODOs under src/ modified this week");
  RUN IT in a real/sandboxed filesystem or a test Vercel project; capture the ACTUAL output — never
  simulate. This is APIGen's execution stage and ToolBench's "real responses, not simulated"
  requirement, applied to a CLI instead of a REST API.
  Diversity axes: vary flags, vary phrasing, include multi-command pipes (`grep | sort | uniq`),
  include distractor tools (`find` vs `locate` vs `grep`), include failure cases (wrong flag, 0
  matches) so the model sees real error text, not invented text.

Step 3 — Verifier-filtered SFT
  Filter: APIGen's 3-gate check (format-valid -> executes without error -> semantically answers the
  instruction, LLM-judge or exact-match) OR Toolformer's loss-reduction filter (keep only if the
  executed result measurably helps predict the continuation).
  Train: standard next-token CE loss on (system, user, assistant-with-tool-call, tool_result) tuples.
  500-1000 verified trajectories is APIGen/xLAM's working scale for a single tool; ToolBench's ratio
  (~4 reasoning turns/instruction) is a reasonable trajectory-depth target.

Step 4 — Optional RL (only if Step 3 plateaus, or exit-code/efficiency matters)
  Reward: dense, decomposed (ToolRL) — R_format (valid syntax) + R_execution (ran without error) +
  R_answer (task solved) — NOT one sparse outcome reward, which collapses past ~4 steps for
  compositional CLI tasks. GRPO on-policy, executing the sampled command for real each rollout.
  Structurally this is imitation -> RL refinement; rejection sampling is the simplest version —
  canonical home: methods/imitation.md. Reach for full RL only when SFT plateaus.

Q&A’s only legitimate role in this recipe: a small (~50-100), format-only seed (Q: find TODOs in python files -> A: grep -r TODO --include='*.py') to bootstrap valid call syntax before Step 2 — never the primary signal.

3. Teaching RECON — search / information-seeking RL

Recon is not a longer tool-call trajectory. It’s a search tree with an outcome-only reward, and the winning recipe is structurally different from §2:

methodology-seed (thin SFT, format-only)  ->  RL discovers the branches
(cold-start: teaches the CALL FORMAT,          (Search-R1 / R1-Searcher / DeepResearcher / ToolRL:
 NOT the decision tree)                         outcome reward on REAL execution, retrieved-token
                                                 masking, GRPO — the tree structure emerges)

3.1 The data object

(query, trajectory=[(action, real_observation)]_1..n, final_answer, outcome_reward) — crucially, the intermediate steps are not supervised; only the terminal outcome is rewarded. This is what forces the model to discover which branches are worth exploring rather than imitate one hand-authored path. Search-R1’s worked row (arXiv:2503.09516):

{
  "question": "Who won the 2024 US presidential election?",
  "trajectory": [
    {
      "step": 1,
      "reasoning": "I need to search for the 2024 US election results",
      "search_query": "2024 US presidential election winner",
      "observation": "<retrieved_passage_text>",
      "answer": "Harris won the election"
    }
  ],
  "final_answer": "Kamala Harris",
  "reward": 1.0
}

Note the intermediate answer field is allowed to be wrong — it’s not supervised, only the terminal one is. This row is generated online, during training, not pre-computed: for each task, sample N rollouts against a live/sandboxed retriever (Search-R1 used online BM25 over 2018 Wikipedia), each a real cold-start → search → reason → answer sequence, reward = exact-match on the final answer.

3.2 Retrieved-token masking — the mechanism that makes the strategy transfer

During the GRPO backward pass, loss is computed only on model-generated tokens (the query text, the reasoning) — never on the retrieved/observed passage tokens. This stops the model from overfitting to one retrieval corpus’s exact phrasing and is exactly why the learned strategy — when to search again, when to stop — transfers to a new corpus or a new tool, whereas a model trained by imitating fixed retrieval snippets would not. The same masking rule generalizes past web search: shell stdout, an HTTP response body, a scan’s raw output — anything that came back from the environment rather than from the model gets masked the same way. Full mechanics of this masking rule inside the broader tool-integrated-reasoning RL literature (ReTool, ToRL, Search-R1) live in methods/rl-long-horizon-exploration.md §1.8 — not re-derived here.

3.3 Why SFT-first-then-RL is contested, not settled

Nemotron-Research-Tool-N1 (arXiv:2505.00024) found pure RL from a cold start outperformed SFT-then-RL on their benchmark — a thick SFT seed on distilled reasoning traces overfit to that reasoning style and hurt downstream RL. R1-Searcher (arXiv:2503.05592) runs no cold-start SFT at all, using a two-stage reward instead: stage 1 rewards invoking retrieval in the right format, regardless of correctness (teaches the call shape); stage 2 rewards getting the final answer right (teaches using the retrieved info correctly). WebGPT (arXiv:2112.09332), by contrast, used human demonstrations (behavior cloning) followed by reward-model-guided rejection sampling — and that worked well precisely because its action space (arbitrary web clicks) is far larger and less constrained than a small fixed API surface, closer to the terminal-agent action spaces below.

Working rule for this book: a thin, format-only SFT seed — teaching valid call syntax, not a decision tree — is safe and typically improves sample efficiency; a thick SFT seed on full expert reasoning traces risks exactly the Nemotron overfitting failure. This is the same trade-off this book covers in depth for the general hint-guided case — see hint-guided-bootstrapping.md — apply that chapter’s risk register before deciding how thick your recon seed should be.

3.4 Let branching emerge; do not author it

DeepResearcher’s outcome-only RL (arXiv:2504.03160) produces self-reflection, cross-validation, and honesty (“I can’t find X”) as emergent behaviors — none were hand-coded into the reward or the seed data. This is the empirical case for seed-then-RL-discovers over hand-authoring a decision tree: a fixed methodology (“always whois, then DNS, then web”) is brittle, because real recon has too many valid orderings. TIER (arXiv:2605.16790) makes the negative case concrete: a trajectory-supervised reward (reward the model for matching one authored path) collapses past depth 4–6, because it penalizes perfectly valid alternative orderings the annotator didn’t happen to take. Execution-grounded, outcome-only reward doesn’t have this failure mode — it rewards any trajectory that reaches the correct terminal state, at whatever depth. If a curriculum is needed at all, order by task difficulty (easy targets first) — R1-Searcher found that specifically including hard questions prevents early convergence and under-exploration.

Full RL mechanics — GRPO, PPO-clip, group-relative advantage, long-horizon credit assignment at depth — are this book’s own territory: methods/rl-long-horizon-exploration.md and methods/agentic-rl.md. Not re-derived here.

3.5 Worked recipe: teaching recon

Step 1 — Methodology seed (thin SFT, NOT a full decision tree)
  Data: a small number (tens, not thousands) of demonstrated (query, action, real_observation, ...,
  outcome) trajectories showing the CALL FORMAT for the recon tools available (search, grep,
  nmap-equivalent, whois, etc.) — teach "here is how you emit a well-formed action," not "here is the
  one correct order to explore in." Over-specifying (full expert reasoning traces) risks the
  Nemotron-Tool-N1 failure: the model memorizes the demonstrated path instead of learning to search.

Step 2 — RL with outcome-only reward + retrieved-token masking
  Data object generated ONLINE during training (not pre-computed): for each task, sample N rollouts,
  each a real sequence of (action, real_observation) pairs against a live/sandboxed target, terminating
  in an answer; reward = binary or graded task success (flag found / answer exact-match / vulnerability
  confirmed) — intermediate steps unsupervised.
  Mask the loss to generated tokens only (actions + reasoning), never retrieved/observed text.
  GRPO update per batch of rollouts.

Step 3 — Let branching emerge; do not author it
  Do not hand-write a fixed scaffold ("first whois, then DNS, then web") — it is brittle. RL with a
  good outcome reward reliably discovers cross-validation, backtracking-on-dead-end, and stopping
  conditions on its own. If curriculum is needed, order by difficulty (easy targets first) to avoid
  early convergence / lack of exploration.

4. Consolidated table: skill-type → data object → how made → executed? → technique

5. Two concrete recipes, side by side

6. Confidence and what’s contested

• High confidence, convergent across all 7 sources: the Q&A-vs-trajectory verdict (§1), the executed > synthetic ordering for tool data (§2.1), and retrieved-token masking as the mechanism that lets a learned search strategy generalize (§3.2). These are not single-paper claims.
• Contested, stated as such by the sources themselves: whether recon needs any SFT seed before RL (Nemotron-Tool-N1’s pure-RL-wins result vs. R1-Searcher’s two-stage-reward-no-SFT vs. WebGPT’s imitation-then-rejection-sampling for a much larger action space) — see §3.3. Don’t present “thin seed then RL” as settled; present it as this book’s working rule given the evidence on both sides.
• Working from the source notes, not a fresh independent search this session — the arXiv ids above were verified live during the research pass the notes themselves record (2026-07-02); this chapter synthesizes rather than re-verifies.

Cross-links (canonical homes — referenced, not re-derived here)

• Why execution is the training engine, generally: kinds-of-sft.md.
• Rejection sampling / STaR-family imitation-refinement (the RL-lite mechanism behind the optional RL step in §2.4, and WebGPT’s rejection-sampling stage): methods/imitation.md.
• Full RL mechanics for search/exploration and long-horizon agentic credit assignment (GRPO, outcome-only reward at depth, retrieved-token masking’s place in the wider RL literature): methods/rl-long-horizon-exploration.md + methods/agentic-rl.md.
• STaR / hint-guided bootstrapping (thin-seed-then-let-the-model-discover, and the risk of an over-thick seed collapsing exploration): hint-guided-bootstrapping.md.
• On-policy vs. off-policy as the organizing axis underneath “executed at rollout time” vs. “pre-computed and replayed”: foundations/on-off-policy.md.
• General method→data framing: method-to-data.md.

Verified arXiv IDs referenced in this chapter

Toolformer 2302.04761 · ToolLLM/ToolBench 2307.16789 · APIGen 2406.18518 · xLAM 2409.03215 · APIGen-MT 2504.03601 · Gorilla 2305.15334 · DocPrompting 2207.05987 · Search-R1 2503.09516 · R1-Searcher 2503.05592 · ToolRL 2504.13958 · Nemotron-Research-Tool-N1 2505.00024 · DeepResearcher 2504.03160 · WebGPT 2112.09332 · TIER 2605.16790. All cross-checked live 2026-07-02 per source notes.

Preference — RLHF · DPO · KTO

Signal = comparisons (A ≻ B). This family exists for the case where you cannot write verify() — “helpful / harmless / on-brand” has no programmatic checker, but a human (or an AI judge) can rank two outputs. Preference is one of three fixed paradigm presets on the canonical on/off-policy axis (the other two are imitation and reinforcement) — see Foundations for the axis itself and Contested edges §6 for why “three independent knobs you toggle / a combinatorial grid” is a retired teaching scaffold: the axes aren’t independent, so what differs across RLHF/DPO/KTO/IPO/ORPO/SimPO is where the pairs come from and how the loss is shaped, not a free combination. Load-bearing property: preference methods reshape ranking over behaviors π_θ can already produce — they inject no new capability (lessons/post-training/dpo-kto-for-agent-tool-selection.md, shared memory). This is grounded in the elicit-not-expand genealogy (RLVR/RL reweights mass already in the base distribution rather than adding new ones — Yue et al., arXiv:2504.13837) and the stronger SFT-memorizes/RL-generalizes anchor (Chu, Zhai et al., arXiv:2501.17161, ICML 2025, 694 citations) — not on the single 0-citation “Scalpel vs. Hammer” preprint (arXiv:2507.10616), whose own authors call their result a “preliminary indication” and which is contested by pushback re-reading the SFT→RL relationship as OOD-forgetting restoration rather than replacement (arXiv:2509.12235); see Contested edges §1.

This chapter is also stage 4/5 of the ordered skeleton (preference opt, after cold-start SFT and rejection-sampling, before/around RLVR) in the recipe is a sequence, one leg of the iterated tail in is the recipe a loop (DPO run N times across rounds, not once), and the T1/T2/T3 tiering in proven-first ranking §3 — read those for where in the pipeline and why start here; this chapter is the what/mechanism.

RLHF (reward model + PPO)

• What: train a reward model on preference pairs (Bradley-Terry), then optimize π_θ against it with PPO + KL-to-reference (PPO mechanics: Reinforcement). The canonical pipeline is InstructGPT (arXiv:2203.02155).
• Eats: preference pairs → a learned scalar reward.
• Still alive in 2026, not dead: Gemini 2.5 runs an explicit Reward-Model + Critic + RL loop (“RLF”, arXiv:2507.06261 §2.4); GPT-5’s sycophancy fix scores conversations and uses that as a training reward (OpenAI GPT-5 system card / model-training page).
• Gotcha: a learned RM has parameters to exploit → reward hacking. Deterministic verifiers (RLVR) avoid this; see the gameability ladder in Contested edges.

DPO and the direct-preference family

• What: skip the RM + RL loop — a closed-form loss directly raises logπ_θ(chosen) − logπ_θ(rejected) against a frozen reference, provably equivalent to the RLHF objective under Bradley-Terry (DPO, arXiv:2305.18290). Key hyperparameter: β (KL strength).
• Eats: (prompt, chosen, rejected) triples.
• Policy: off-policy by default (pairs usually from another model / earlier checkpoint) — its weakness; iterative/online DPO resamples from current π_θ each round to make it on-policy. Off-policy preference optimization “often suffers from a distributional gap between the policy and the preference data” (arXiv:2406.11827); regenerate the preference pairs from the current checkpoint before each new round rather than reusing stale ones — see is the recipe a loop for the full mechanics and failure mode.
• Production proof: Llama 3 chose DPO over PPO for its offline preference stage for stability/scalability at their scale, and runs it iteratively (their “iTeC” = rejection-sampling + SFT/DPO/IPO + online RL, several rounds) (arXiv:2407.21783); Llama 2 runs ~6 rounds of SFT+DPO. Qwen2.5 stages an explicit SFT → offline DPO → online GRPO sequence (arXiv:2412.15115).
• Ordering caveat (contested, not settled): DPO before or after RLVR depends on whether the preference signal and the verifiable-reward signal target the same behavior (fold into one RL stage, à la DeepSeek) or orthogonal behaviors — report quality/style vs. flag-captured — (sequence them, harder-to-specify objective last, à la Tülu 3/Nemotron). Full table: the recipe is a sequence §5. Also: heavy pre-RL DPO is erosive to downstream RL exploration (“SFT and DPO can over-constrain the model, restricting exploration during the online RL stage” — Llama 4 blog); a light, on-policy-anchored DPO pass after RL (Llama 4’s own move) is conditional/safe. See stage ordering & batching and data mixing & forgetting for the dose-dependence.

Variants and their niche

• KTO (arXiv:2402.01306): learns from unpaired good/bad labels (Kahneman-Tversky value model) — no matched pairs needed. This fits mined agent logs exactly (a pile of failed runs + a pile of clean solves).
• IPO (arXiv:2310.12036) stabilizes DPO’s tendency to collapse both logprobs at high β; ORPO (arXiv:2403.07691) folds preference into SFT with no reference model; SimPO (arXiv:2405.14734) drops the reference via length-normalized reward.
• Honest status: DPO is the T1 proven-first pick here (9,399 citations, named in Llama 3’s Herd-of-Models report, Tülu 2/3’s post-bake-off choice, default trainer in TRL/Axolotl/LLaMA-Factory/Unsloth); KTO is T2 — strong conditional fit for unpaired pass/fail logs specifically. IPO/ORPO/SimPO are otherwise niche/T3 — real, used in fine-tuning shops and ablated in Tülu 3 (which explicitly bake-off’d SimPO against DPO-norm and kept DPO-norm), but no Llama/Qwen/DeepSeek/GPT/Claude/Gemini tech report names them as the production choice (survey: arXiv:2503.11701). Plain DPO + iterative DPO are the mainstream ones. Full tier table with citation counts and the “why start with DPO here”: proven-first ranking §3.

RLAIF / Constitutional AI

• What: replace human preference labels with AI feedback against a written constitution (Constitutional AI, arXiv:2212.08073).
• Status: mainstream at Anthropic (it is the core method) and partially adopted at Google (Gemini 2.5 safety is “loosely inspired by Constitutional AI”, arXiv:2507.06261). 2026 refinement: Anthropic now teaches the constitution via synthetic document fine-tuning (SDF) → SFT → RL, because “demonstrating desired behavior is insufficient — the model must learn why” (alignment.anthropic.com, “teaching Claude why”, 2026).

Reinforcement — PPO · GRPO · RLVR

Signal = reward / verification. Fully on-policy, online, uses the whole reward landscape (push winners up and losers down). Most powerful, most expensive/unstable. This is the family driving every 2025–2026 reasoning model. Reinforcement is one of three fixed paradigm presets on the canonical on/off-policy axis (the other two are imitation and preference) — see Foundations for the axis itself and Contested edges §6 for why “three independent knobs you toggle / a combinatorial grid” is a retired teaching scaffold: the axes aren’t independent, so what differs across PPO/GRPO/GSPO/DAPO/RLVR is how the advantage is computed and where the reward comes from, not a free combination. In the staged recipe this is the last stage — the recipe is a sequence places RLVR/GRPO after cold-start SFT, rejection-sampling, and preference-opt; proven-first ranking has the T1 pick inside this family (RLVR-via-GRPO); is the recipe a loop? covers revisiting this stage across rounds.

PPO

• What: clipped-surrogate policy gradient with a value/critic network + KL-to-reference (arXiv:1707.06347).
• Eats: prompts + a reward (learned RM or verifier).
• 2026 status: still used for classic preference-RL at the proprietary labs, but declining share for reasoning-RL (the critic is expensive; GRPO/GSPO replaced it there).

GRPO (Group Relative Policy Optimization)

• What: drop the critic. Sample a group of N completions per prompt; the group mean reward is the baseline; advantage Aᵢ = rᵢ − mean(r) (optionally std-normalized). Introduced in DeepSeekMath (arXiv:2402.03300).
• Eats: prompts + a reward fn; no fixed target dataset.
• 2026 status: the reasoning-RL default — DeepSeek-R1’s core algorithm (arXiv:2501.12948), still the base of V3.2’s mixed RL (arXiv:2512.02556).
• Requirement (project rule): baseline solve rate must sit in 30–60% per prompt-group — all-pass or all-fail groups give zero advantage → zero gradient (llmresearch-handbook.md rule 7; mechanics in handbook.md §10, shared memory).

GRPO successors

• GSPO (Qwen) [R][L] — the problem with vanilla GRPO/PPO at scale: the token-level importance ratio r_t = π_θ(a_t|s_t)/π_old(a_t|s_t) compounds multiplicatively over a long response, and noisy per-token drift is specifically what destabilizes MoE RL (expert routing shifts mid-rollout, under-policy). GSPO clips at the sequence level instead:

  # GRPO/PPO: one ratio PER TOKEN, clipped per token — variance compounds over length L
  r_t = pi_theta(a_t|s_t) / pi_old(a_t|s_t)
  
  # GSPO: one ratio for the WHOLE sequence (length-normalized geometric mean)
  r_seq = (pi_theta(y|x) / pi_old(y|x)) ** (1 / len(y))
  loss  = -mean(min(r_seq * A, clip(r_seq, 1-eps, 1+eps) * A))
  

  This is Qwen3’s actual stated production RL algorithm (arXiv:2507.18071) — the first GRPO-successor with a flagship behind it, and it gets more relevant, not less, as episodes lengthen: a 100-turn agentic trajectory with tool calls interleaved is exactly the long-sequence regime where token-level ratios drift furthest from 1 by the last token. If a future GRPO/RLVR run on the CTF agent shows training instability, sequence-level clipping is the first thing to try — not more KL-coefficient tuning.

• DAPO (ByteDance Seed) [E][R][L] — four concrete engineering fixes, not one new algorithm, each independently adoptable as a verl loss-mode flag (arXiv:2503.14476):

  1. Clip-Higher — decouple the PPO clip bounds (eps_low ≠ eps_high, e.g. 0.20 / 0.28 vs. the symmetric PPO-default 0.20/0.20) so a rare-but-good token can gain probability faster than a bad one loses it. Symmetric clipping caps how fast a rare-correct action can ever be reinforced — a direct driver of entropy collapse (below).
  2. Dynamic Sampling — resample any prompt whose whole group of G rollouts is all-correct or all-incorrect (std(group_rewards) == 0 → zero advantage → zero gradient in GRPO) instead of paying for a wasted rollout batch.
  3. Token-level loss — average the policy-gradient loss over every token in the batch, not per-sample-then-averaged, so long correct/incorrect responses aren’t down-weighted relative to short ones. [L] — a credit-assignment fix that matters more the longer responses get.
  4. Overlong reward shaping — a soft length penalty instead of a hard truncation penalty, so a response cut off by the context window isn’t punished as if it were simply wrong.

  eps_low, eps_high = 0.20, 0.28
  ratio   = exp(logp_new - logp_old)
  clipped = clip(ratio, 1 - eps_low, 1 + eps_high)
  loss_pg = -min(ratio * adv, clipped * adv)          # per-token, mean over ALL tokens in the batch
  
  while std(group_rewards) == 0:                       # dynamic sampling
      prompt = resample_prompt()
      group_rewards = rollout_and_score(prompt, n=G)
  

  Mainstream in OSS RL tooling (verl and open GRPO reproductions default to these four fixes) and independently reproduced as a 50-point AIME 2024 result beating R1-Zero-Qwen-32B with half the training steps — one of the few fully open (algorithm + infra + data) large-scale reasoning-RL reproductions. Project rule 7 (GRPO baseline must hit 30–60%) is DAPO’s dynamic-sampling problem, stated as a portfolio-composition constraint instead of a training-loop fallback — keeping the baseline in-band is how you avoid feeding all-pass/all-fail groups into the update in the first place; DAPO’s dynamic sampling is the fallback for whatever still lands there. At ~100 turns per rollout, resampling a whole-group-zero-reward challenge is expensive — prefer upstream curriculum/difficulty filtering (drop challenges outside the 30–60% band) over paying for resamples on genuinely-unsolved-yet challenges.

  Designed to fix a common failure: no-methodology pivoting after one failed attempt, and committing to a single ungrounded guess → Clip-Higher. Agentic policies commonly abandon a promising line of attack after a single setback and default to whichever guess is left standing when no path clearly dominates. Symmetric clipping caps how much probability mass a rare, correct enumeration branch (or a well-grounded, as opposed to lucky, guess) can ever accumulate — which is precisely what narrows a policy onto one brittle script. See Exploration and entropy below for the mechanism this is patching.

• Dr. GRPO [R][E] — a smaller, easy-to-miss companion fix: vanilla GRPO’s per-sample length- and std-normalization secretly rewards longer wrong answers and shorter right ones (an optimization artifact, not a real preference). Fix is a two-line change — drop the 1/|response| length term and the group-std division, keep only A = r − mean(r) (arXiv:2503.20783). Matches or beats vanilla GRPO’s accuracy at the same compute while removing the length-inflation drift. For a 100-turn agent this bug has a much bigger attack surface than a single-turn math answer: a policy trained on the unfixed objective can learn to “look busy” (extra tool calls, redundant enumeration) after a wrong guess without the enumeration being useful — nearly indistinguishable from legitimate PTES-style enumeration unless you’re specifically checking for it.

RLVR (RL with Verifiable Rewards)

• What: GRPO/PPO where the reward is a deterministic verifier (unit tests, math checker, flag check) rather than a neural RM. No parameters to game.
• Eats: prompts + a verify(state) → {0,1} function. This is your setup — the CTF flag verifier is a textbook verifiable reward.
• 2026 status: arguably the defining technique of the era. Every reasoning model (o1/o3, R1, Gemini-thinking, Qwen3, Kimi) scales RL against verifiable/rule-based rewards as the capability driver; Gemini 2.5 explicitly allocates increased RL compute to “verifiable rewards” (arXiv:2507.06261; OpenAI “Learning to reason with LLMs”; R1, arXiv:2501.12948).

Exploration and entropy: the GRPO graduation trigger

Cybersecurity is exploration — every technique in the PPO/GRPO/GSPO/DAPO family above is [E]-tagged, and entropy collapse is what turns “graduate SFT → GRPO/RLVR” from a vague heuristic into a measurable trigger. The graduation trigger, precisely stated: don’t wait for the reward curve to plateau — watch mean(entropy); once it’s tracking toward the flat part of the fitted collapse curve, more rejection-sampling-SFT epochs on the same policy distribution won’t move the needle (you’re re-sampling an already-narrowing distribution) — that’s the signal to graduate to GRPO/RLVR. Full mechanism (the fitted law, the covariance driver, Clip-Cov/KL-Cov, the clip-asymmetry result, high-entropy minority tokens, ProRL’s boundary-expansion evidence, and the amplify-vs-elicit contested-edges reconciliation): Long-horizon & exploration RL §2.1–2.7.

Everything past this point — pass@k as a training signal, diversity/curiosity/count-based intrinsic rewards, parameter-space noise for temporally-coherent exploration, tool-call-sequence diversity as the project’s own novel opportunity, and the full turn-level/step-level credit-assignment literature for the ~100-turn setting — is covered in depth in the dedicated long-horizon and exploration-sweep chapters, and in Agentic & multi-turn RL for the multi-turn training-loop shape itself. Read this section for the graduation trigger; read those for the harder credit-assignment and boundary-expansion questions once you’re past the initial DAPO-recipe GRPO baseline.

The reward-model question (PRM vs outcome/rubric)

• PRM (process reward, dense step-level) — score each reasoning step (Lightman et al., arXiv:2305.20050). Niche / avoided in production: DeepSeek explicitly rejected PRM for R1 due to step-level reward hacking (arXiv:2501.12948).
• Outcome verifier + rubric/critic grading — the real 2026 answer to “what replaced PRM”: not dense step rewards, but LLM-judge/rubric-based outcome grading. Gemini’s “Critic” (prompted rubric grader, arXiv:2507.06261) and OpenAI’s RFT “model grader” are both this in production. Your deterministic flag verifier is the ungameable end of this spectrum — keep it there (gameability ladder in Contested edges).

When to reach for RL

An execution gap where rejection-sampling FT has plateaued (entropy collapsed), and you want the negative-sample gradient + online updates. Cost: online rollout infra, reward plumbing, KL control, instability, and the train↔inference precision mismatch (its own rabbit hole — see the shared memory note research/fp8-quantization-mechanics-training-serving.md).

This is not a one-shot decision. Is the recipe a loop? covers revisiting GRPO/RLVR across rounds (continue-from-checkpoint vs. restart-from-base, loop-exit criteria); ordering rules covers where a DPO/preference stage sits relative to this one and whether SFT applied afterward erodes the RL gains; data mixing & forgetting covers what happens to reasoning traces when RL-stage output gets mixed back into later SFT rounds; and proven-first ranking is the concrete tier table for picking an optimizer inside this family (GRPO/RLVR T1, PPO/DAPO T2 fallbacks, GiGPO/GTPO T3/T4 watch-list).