Matching the fix to the gap — K→inject, R→re-prior, P→re-rank

The question this chapter answers: once you’ve diagnosed which gap you have, which technique do you actually run, and what does the training row look like? BLUF: K (knowledge) needs off-policy injection — CPT + knowledge-SFT, never trajectory imitation. R (the default/prior) needs a small, off-policy, format-only cold-start SFT that re-shapes what the policy samples first, not what it’s capable of. P (ranking) needs an on-policy preference pass — DPO/KTO or a short GRPO run — localized at the exact state where the model already produces the right action but doesn’t prefer it. These are not interchangeable: pointing a K-shaped fix (raw documents) at an R-gap teaches facts nobody asked for; pointing an R-shaped fix (cold-start SFT) at a P-gap wastes a training run reinforcing a ranking the model can already produce on its own, off-policy, at the wrong dose. Diagnosing the gap — a scientific framework gives you the instruments (pass@k, the elicitation ladder, Pass@(k,T)) that tell you which gap you have; the fuller K/R/P diagnostic battery — pass-at-k sweep, in-context oracle probe, teacher-forced logprob spans, the reversal check, the purpose-built bottleneck task — lives in ./diagnosis.md, deferred from this chapter on purpose. This chapter starts after that diagnosis lands and answers only “now what do I train, on what data shape.”

Standing scope note, same as the sibling chapters this one sits beside: every claim below is general RL/ML theory or frontier-lab practice (DeepSeek-R1, Qwen3, AlphaGo/AlphaGo Zero, Ng-Harada-Russell reward-shaping theory). Academic cybersecurity-LLM papers are not cited anywhere in this chapter — none exist as evidence here, per the project’s standing rule.

0. The running example, stated once

One failure — “the agent doesn’t use amass for subdomain enumeration, defaults to a shallow nmap sweep instead, and misses the flag hidden behind a forgotten staging subdomain” — cashes out as three structurally different problems depending on what’s actually broken, and each needs a different training row:

Threading this through the chapter keeps the point concrete: the same observed symptom (“doesn’t use amass”) routes to three incompatible fixes, and running the wrong one either wastes a training run or actively teaches the wrong thing (raw-trajectory SFT for a K-gap; more format-shaping SFT for a P-gap the model already has the capability for). One diagnostic worth stating up front, cross-linked from ./diagnosis.md: if the same tool-avoidance pattern shows up across several unrelated frontier base models, that’s evidence against a per-model K-gap (a shared pretraining-distribution prior — generic recon over structured OSINT tooling — is a much more parsimonious explanation than every base model independently lacking the same fact) and for an R- or P-gap. Verify per model before trusting this shortcut; it’s a prior, not a proof.

1. Genealogy — three lineages converge, then fragment

Three independent research lineages solved three different problems — “how do you improve a policy from its own samples,” “how do you optimize against comparisons instead of a scalar reward,” and “how do you stop a reward function from being gamed” — and by 2025 all three had to be stitched together to ship a single flagship reasoning model. Understanding why they converge, and where the post-convergence literature has since fractured, is what tells you whether a given fix is settled practice or a live bet.

flowchart TD
  subgraph A["Lineage A — self-improvement / sharpening"]
    direction TB
    A0["Ng, Harada & Russell (ICML 1999, no arXiv id)<br/>potential-based reward shaping is<br/>necessary + sufficient for policy invariance"] --> A1
    A1["AlphaGo (Nature 529, 2016, no arXiv id)<br/>supervised-init THEN RL"] --> A2
    A2["AlphaGo Zero (Nature 550, 2017, no arXiv id)<br/>the supervised step is not always necessary"] --> A3
    A3["STaR (2203.14465)<br/>sample rationale, keep if correct, SFT, repeat"] --> A4
    A4["RAFT (2304.06767) / ReST (2308.08998) /<br/>ReST-EM (2312.06585)<br/>generalize the filter-then-SFT loop"] --> A5
    A5["GRPO (2402.03300)<br/>turns it online, critic-free"] --> A6
    A6["Sharpening Mechanism (2412.01951)<br/>PROVES: can only redistribute mass<br/>already in base support, never create it"]
  end

  subgraph B["Lineage B — preference"]
    direction TB
    B0["DPO (2305.18290)<br/>closed-form reweighting of reference-policy mass"] --> B1
    B1["KTO (2402.01306)<br/>generalizes to unpaired good/bad labels"] --> B2
    B2["Self-Rewarding LMs (2401.10020)<br/>closes the loop online"] --> B3
    B3["Online-vs-offline (2405.08448)<br/>offline DPO helps classification,<br/>not generation — go online"]
  end

  subgraph C["Lineage C — reward design"]
    direction TB
    C0["Concrete Problems (1606.06565)<br/>names reward hacking"] --> C1
    C1["Skalse et al. (2209.13085)<br/>PROVES: unhackability needs a<br/>restricted policy class"] --> C2
    C2["Goal misgeneralization<br/>(2105.14111, 2210.01790)<br/>correct reward ≠ sufficient"] --> C3
    C3["UED / PAIRED (2012.02096)<br/>+ minimax-regret fix (2507.03068)<br/>the curriculum answer"]
  end

  A6 --> R1
  B3 --> R1
  C3 --> R1

  R1["DeepSeek-R1 (2501.12948)<br/>cold-start SFT (R) → RLVR (P) →<br/>rejection-sample 800K rollouts (re-diversify) →<br/>final alignment RL (sharpen).<br/>R1-Zero ablation proves stage-1's marginal value."]

  R1 --> Q3["Qwen3 (2505.09388)<br/>confirms this is now default frontier practice"]

  R1 --> Frag["POST-R1 FRAGMENTATION (2025–2026)"]

  Frag --> F1["Echo Chamber (2504.07912)<br/>RL amplifies pretraining-precursor<br/>behaviors, doesn't invent new ones"]
  Frag --> F2["pass@k boundary debate:<br/>shrink (2504.13837) vs<br/>expand (2505.24864 ProRL) vs<br/>reconcile (2510.04028)"]
  Frag --> F3["Scalpel vs Hammer (2507.10616)<br/>SFT=broad-replace, GRPO=narrow-amplify<br/>[contested, single low-cite preprint]"]
  Frag --> F4["RL's Razor (2509.04259)<br/>on-policy RL is KL-minimal from base<br/>[behavior agreed, mechanism contested]"]
  Frag --> F5["Base Model Barrier (2603.06957)<br/>escaping zero-likelihood costs<br/>exponential reward queries"]
  Frag --> F6["PaST (2601.11258)<br/>SFT/RL weight-space deltas<br/>are near-orthogonal"]

  classDef lin fill:#132b22,stroke:#34d399,color:#eafaf3;
  classDef conv fill:#3a2a10,stroke:#f5b942,color:#fff3d6;
  classDef frag fill:#2b1313,stroke:#f87171,color:#fde8e8;
  class A0,A1,A2,A3,A4,A5,A6,B0,B1,B2,B3,C0,C1,C2,C3 lin;
  class R1,Q3 conv;
  class Frag,F1,F2,F3,F4,F5,F6 frag;

1.1 Lineage A — self-improvement / sharpening

The oldest thread, and the one that names the ceiling this whole chapter has to work around. Ng, Harada & Russell (ICML 1999, no arXiv id) prove that reward shaping via a potential function F(s,a,s') = γΦ(s') − Φ(s) is necessary and sufficient for policy invariance — add any other shaping term and you risk creating a new optimum that isn’t the one you wanted. This theorem is the reason §3 below exists: every later reward-hacking fix in Lineage C is downstream of knowing exactly what form of shaping is safe. AlphaGo (Nature 529, 2016, no arXiv id) establishes supervised-init-then-RL as the working pattern — train a policy on expert demonstrations first, refine with self-play RL second. AlphaGo Zero (Nature 550, 2017, no arXiv id) shows the supervised step isn’t strictly required — pure self-play RL from random init reaches superhuman play — which is the first empirical hint that a sharpening loop can bootstrap capability without an off-policy teacher. STaR (arXiv:2203.14465) ports this to text: sample a rationale, keep it only if the final answer verifies, fine-tune on the survivors, repeat — the minimum-viable self-improvement loop, no reward model, no RL infra. RAFT (arXiv:2304.06767), ReST (arXiv:2308.08998), and ReST-EM (arXiv:2312.06585) generalize STaR’s filter into a formal growing-batch offline-RL loop. GRPO (arXiv:2402.03300) turns the whole thing online and critic-free — full mechanics in Reinforcement — PPO · GRPO · RLVR, not re-derived here.

The lineage’s ceiling gets a formal proof at the end: the Sharpening Mechanism (arXiv:2412.01951) shows this whole family — STaR through GRPO — can only redistribute probability mass the base policy already assigns nonzero weight to. It cannot manufacture support where none existed. This is the theoretical anchor for why a K-gap (zero support, at any N) is categorically un-fixable by anything in this lineage, and it’s the reason §2.1 below routes K-gaps somewhere else entirely.

1.2 Lineage B — preference

DPO (arXiv:2305.18290) collapses the RLHF reward-model-plus-PPO pipeline into a single closed-form classification loss, provably a re-weighting of the reference policy’s own mass toward the chosen side of each pair — full mechanics in Preference — RLHF · DPO · KTO. KTO (arXiv:2402.01306) generalizes the same idea to unpaired desirable/undesirable labels — the shape you get for free out of a rollout pool with a pass/fail verifier and no matched pairs. Self-Rewarding Language Models (arXiv:2401.10020) closes the loop online — the model generates its own new preference pairs each round instead of training on one frozen batch. Understanding the performance gap between online and offline alignment (arXiv:2405.08448) is the mechanistic reason iteration matters here: offline DPO measurably helps the model’s classification accuracy (can it tell chosen from rejected after training) without a matching gain in generation quality (does it actually produce the chosen behavior more often) — going online, resampling from the current policy each round, is what closes that gap. This is the direct citation behind §2.3’s “iterate 2–3 rounds” instruction below, and it’s the same distributional-gap argument Preference and Is the recipe a loop? already make about DPO’s off-policy-by-default weakness — not re-derived here.

1.3 Lineage C — reward design

Concrete Problems in AI Safety (arXiv:1606.06565) names reward hacking as a first-class failure mode a full decade before RLVR made it a daily operational concern. Skalse et al. (arXiv:2209.13085) formalize why it’s so hard to design around: a proxy reward is provably unhackable only under a constant-reward condition or a restricted policy class — for an unrestricted, expressive policy (any modern LLM), essentially every reward you can write down that isn’t the true objective itself has some hackable slack. Goal misgeneralization — Langosco et al. (arXiv:2105.14111) and Shah et al. (arXiv:2210.01790) — is the sharper, and for this chapter the load-bearing, half of that finding: even a correct reward specification is not sufficient, because the training data leaves the target goal underdetermined between several policies that all score equally well on the data you actually trained on. Shah et al.’s own framing is exactly the fix §3 uses below: disambiguate the target by making it structurally necessary — construct training/environment data such that the intended goal is the only remaining explanation for high reward. UED/PAIRED (arXiv:2012.02096) and its minimax-regret fix (arXiv:2507.03068) give the curriculum-design answer to the same disambiguation problem — generate environments that are maximally informative about which goal the policy has actually learned, rather than hand-picking a fixed curriculum that happens to be uninformative.

1.4 Convergence at DeepSeek-R1

DeepSeek-R1 (arXiv:2501.12948) is where all three lineages stop being separate research programs and become one shipped pipeline: cold-start SFT (an R-shaped fix — small, format/behavior-only, off-policy) → reasoning-focused RLVR (a P-shaped fix — GRPO reranking what the cold-started policy already samples, per Lineage A/C’s combined machinery) → rejection-sample ~800K rollouts off the RL-converged checkpoint and retrain from clean base (this re-diversifies/re-broadens the training pool before the next pass — restocking coverage that the RL stage’s entropy collapse had narrowed, not a K-fix in the strict sense but a K-adjacent move in the same “put breadth back before sharpening further” spirit) → a final all-scenario alignment RL pass. The paper’s own R1-Zero ablation (pure RL, zero cold-start SFT) is the direct, published proof that stage one has real marginal value — R1-Zero reaches comparable raw reasoning capability but ships with “poor readability, language mixing,” the exact failure mode an R-gap fix (format/behavior shaping) exists to close. Qwen3 (arXiv:2505.09388) runs the identical stage skeleton and states outright that it deliberately minimizes the cold-start stage’s size — confirming this is now default frontier practice, not a DeepSeek-specific quirk. (Full stage-by-stage mechanics of this convergence are in The recipe is a sequence, not a pick and Is the recipe a loop? — this chapter only needs the convergence point, not the whole sequence.)

1.5 Post-R1 fragmentation

Once the “K/R/P all get separately addressed, in this order” pattern became legible, 2025–2026 work immediately started arguing about the boundaries of what stage 3 (the sharpening/RL stage) can actually do:

• Echo Chamber (arXiv:2504.07912) — RL post-training amplifies behaviors that were already precursors in pretraining; it doesn’t invent behavior with no pretraining trace.
• The pass@k boundary debate. Yue et al. (arXiv:2504.13837) — the base model overtakes RLVR at large k; the reasoning boundary shrinks with training. ProRL (arXiv:2505.24864) — under prolonged, KL-controlled training with reference-policy resets, the boundary genuinely expands past what the base model ever reaches. The Two-Stage Dynamic View (arXiv:2510.04028) reconciles both as two phases of the same training run rather than a contradiction — full treatment of this debate, its stakes for this project, and the CoT-Pass@K/Cover@τ metric caveats already live in Reinforcement’s exploration section and Contested edges §1, §7; not re-derived here.
• Scalpel vs. Hammer (arXiv:2507.10616) — “GRPO amplifies existing capabilities, SFT replaces them,” at the weight level. Flag this honestly every time it’s cited: single, 0-citation preprint, its own authors call it a “preliminary indication” — supporting color for the K/R/P split’s intuition, not the load-bearing evidence for it (that’s the Sharpening Mechanism proof above, plus SFT Memorizes/RL Generalizes, arXiv:2501.17161, cited fully in Contested edges §1).
• RL’s Razor (arXiv:2509.04259) — on-policy RL forgets less than SFT because it’s implicitly KL-minimal from whatever policy preceded it: an on-policy update only has to move probability mass among things the current policy already samples, so it stays close to its own starting point in a way off-policy SFT — which pulls toward an externally-fixed target distribution — does not. The behavioral finding (RL forgets less) is well-replicated; the mechanistic explanation (KL-minimality specifically, vs. some other circuit-level preservation property) is actively being revisited. This is the citation behind §5’s staged-curriculum ordering below, used with that mechanism caveat carried forward.
• Base Model Barrier (arXiv:2603.06957, very recent) — formalizes exactly what Lineage A’s Sharpening Mechanism proved qualitatively: escaping a region of genuinely zero base-model likelihood costs exponentially many reward queries under policy-gradient methods. This is the sharpest, most recent statement of “RL cannot cheaply create what CPT/SFT never put there” — direct grounding for §2.1’s K-gap routing.
• PaST (arXiv:2601.11258, very recent) — measures the actual weight-space deltas SFT and RL produce and finds them near-orthogonal: the two stages aren’t fighting over the same parameter directions, which is a load-bearing empirical fact for §5’s “reconstruct subset-sums of stages without full retraining” ablation move.

None of these six papers overturns the K/R/P routing this chapter uses — they sharpen where the line sits between “elicit” and “expand,” which matters for how aggressively you can lean on a P-gap fix (§2.3) before it starts behaving like a K-gap fix in disguise. Treat the fragmentation cluster as live, contested terrain to watch, not a reason to distrust the routing itself.

1.6 What the genealogy buys you for routing

Three things carry forward from this history into §2’s mechanics, and it’s worth naming them once so the rest of the chapter can lean on them without re-arguing the point each time:

1. Lineage A’s proof (Sharpening Mechanism, arXiv:2412.01951) is why K-gaps get routed away from every technique in §2.2–§2.4. Cold-start SFT, DPO/KTO, and GRPO are all, at bottom, members of the sharpening/self-improvement or preference lineages — they redistribute mass or reweight comparisons over what already has some support. None of them can be the fix for a gap defined as “zero support at any N.”
2. Lineage B’s online-vs-offline finding (arXiv:2405.08448) is why §2.3 insists on iteration, not a single pass. A P-gap fix that stops after one offline round has only improved the model’s ability to tell apart chosen from rejected — not necessarily its tendency to generate the chosen behavior more often. That gap between classification and generation is exactly what makes a one-shot DPO run look like it “didn’t work” on eval even though the loss curve looked fine.
3. Lineage C’s theorem (Ng-Harada-Russell, ICML 1999) is why §3 exists as a separate section rather than a footnote on §2.3. Once you’re tempted to add a reward bonus for the correct action instead of (or alongside) a preference pair, you’ve left the safely-understood territory of “reweight an existing comparison” and entered reward-design territory, where the policy-invariance guarantee only holds for one specific mathematical form of shaping.

2. The matched intervention, gap by gap

flowchart LR
  Diag["Diagnosed gap<br/>(see ../diagnosis.md)"] --> K{"K — never fires<br/>at any N"}
  Diag --> R{"R — fires at moderate N,<br/>but not the DEFAULT"}
  Diag --> P{"P — fires with real mass,<br/>just mis-ranked"}

  K --> K1["CPT: raw/paraphrased continuation rows<br/>(unsupervised, no chat structure)"]
  K --> K2["knowledge-SFT: forward+reverse QA rows<br/>on the fact, NOT full trajectories"]

  R --> R1["cold-start SFT rows:<br/>(prompt, reasoning-block, tool_call)<br/>thousands, format/behavior only"]

  P --> P1["DPO/KTO pair at the exact<br/>decision-divergence state:<br/>(chosen tool_call, rejected tool_call)"]
  P --> P2["or short GRPO pass:<br/>reward = 1 iff correct action used,<br/>prompt = the divergence state only"]

  classDef kfix fill:#132b22,stroke:#34d399,color:#eafaf3;
  classDef rfix fill:#3a2a10,stroke:#f5b942,color:#fff3d6;
  classDef pfix fill:#2b1313,stroke:#f87171,color:#fde8e8;
  class K,K1,K2 kfix;
  class R,R1 rfix;
  class P,P1,P2 pfix;

Reading the diagram left to right: the diagnosis (deferred to ./diagnosis.md) hands you exactly one of three verdicts, and each verdict has exactly one row-shape family attached to it — there is no “run all three, see what sticks” branch, because §1.6 above is precisely the argument for why that wastes budget on two-thirds of a portfolio it shouldn’t touch. The three subsections below walk each branch in the order K → R → P, matching increasing on-policy-ness and decreasing training-row volume — K needs the most data and the least of the model’s own behavior; P needs the least data and the most of the model’s own behavior.

2.1 K → inject: CPT + knowledge-SFT, never raw trajectory-SFT

If the correct action never appears at any N, on any checkpoint, per the diagnostic battery in ./diagnosis.md, there is nothing on-policy to reinforce — the Sharpening Mechanism (arXiv:2412.01951) and the Base Model Barrier (arXiv:2603.06957) both say the same thing from different angles: you cannot cheaply RL your way to mass that isn’t there. The fix is off-policy injection, and it comes in exactly two row shapes, run in this order:

Run this: two-stage K-gap fix.

1. Continued pretraining (CPT) — full-FT, low LR, raw or lightly-paraphrased unsupervised continuation text. Not a chat turn.

   amass enum -passive -d target.com performs passive-only subdomain
   enumeration via OSINT sources (crt.sh, VirusTotal, DNS aggregators)
   without ever sending a packet to the target's own infrastructure —
   this avoids the IDS/WAF triggers that amass enum -active risks...
   

2. Knowledge-SFT — QA-format rows, both directions (dodges the reversal curse, arXiv:2309.12288: a model trained only on “A is B” does not reliably answer “what is A” from “B”).

   {"messages": [
     {"role": "user", "content": "What does the -passive flag do on amass enum?"},
     {"role": "assistant", "content": "Restricts amass to OSINT-only sources (crt.sh, VirusTotal, DNS aggregators) — it never touches the target directly, so it won't trip an IDS/WAF."}
   ]}
   {"messages": [
     {"role": "user", "content": "I need subdomain enumeration that never sends a packet to the target. Which tool/flag?"},
     {"role": "assistant", "content": "amass enum -passive."}
   ]}
   

What this is not: a full multi-turn recon-to-exploit trajectory that happens to include one amass call. That’s raw trajectory-SFT, and it’s the one thing this section explicitly rules out for a K-gap — training on a trajectory teaches “this is the shape a solve looks like,” which is a much weaker, noisier signal for “here is the standalone fact about this tool” than a direct QA row, and it compounds with a separate, serious risk (new-fact SFT rows measurably increase hallucination on unrelated facts once learned) that this chapter defers in full to ./trajectory-amplification.md rather than re-deriving. The full data-curation playbook for this stage — CPT-vs-SFT dosing, WRAP-style paraphrase counts, EntiGraph for narrow corpora, self-play Self-QA, KnownPatch interleaving — is deferred to ./knowledge-curation.md; this section only needs to establish the row shape and the ordering.

Common mistake — mistaking “can recite it” for “genuinely absent.” The K-gap probes in ./diagnosis.md distinguish recognition (the model can describe amass -passive when directly asked) from generation-in-context (it never reaches for the flag at the actual decision point). If a model passes the recognition probe but still fails the pass-at-k sweep during a live rollout, the fact is present — this is an R-gap wearing a K-gap’s costume, and routing it into CPT wastes compute on a fact the weights already encode while leaving the actual default-prior problem untouched. Run the recognition-vs-generation split before committing to §2.1’s two-stage fix, not after it fails to move the eval.

Hyperparameters/gotchas that matter here, carried forward from the genealogy: CPT is full-FT at a low learning rate, not LoRA-first — the Base Model Barrier’s exponential-cost argument (arXiv:2603.06957) is exactly the failure mode a rank-constrained update is worst-positioned to escape, since it further restricts how much of parameter space a single update can move through. Knowledge-SFT rows, by contrast, are cheap and low-rank-friendly — the fact itself is small; only the CPT stage needs the larger capacity budget.

2.2 R → re-prior: cold-start SFT, thousands of format/behavior rows

If the correct action fires at moderate N but the model’s default, low-temperature sampling never reaches for it, the fact is present but the prior isn’t — this is not a knowledge problem, it’s a distribution-shaping problem, and it’s solved the same way DeepSeek-R1’s own stage 1 solves it: a small, deliberately narrow, off-policy cold-start SFT pass whose entire job is to move where the policy’s probability mass starts before any RL or preference stage touches it.

The row is a full (prompt, reasoning-block, tool_call) unit — not a QA pair, because the target here is behavior at a decision point, not a fact:

{"messages": [
  {"role": "system", "content": "You are a security agent. Tools: run_command, submit_flag."},
  {"role": "user", "content": "Challenge: enumerate target.com for attack surface."},
  {
    "role": "assistant",
    "content": "<think>Before touching the host directly, passive subdomain enumeration (amass -passive) is lower-risk and often surfaces attack surface a direct nmap sweep misses — staging subdomains, forgotten hosts. Run that first, then port-scan whatever it finds.</think>",
    "tool_calls": [{"id": "call_1", "type": "function",
      "function": {"name": "run_command", "arguments": "{\"cmd\": \"amass enum -passive -d target.com\"}"}}]
  }
]}

Thousands of rows, not hundreds of thousands — Qwen3’s own explicit design intent is to minimize this stage’s size (arXiv:2505.09388), and DeepSeek-R1’s cold-start set is “thousands,” deliberately tiny (arXiv:2501.12948). R1’s own four sourcing methods, directly reusable here:

1. Few-shot elicitation — prompt a strong model with 2–3 hand-written examples of the desired recon-first behavior, let it generate more in the same shape.
2. Zero-shot elicitation with an explicit verify/reflect instruction — ask the model to solve the challenge and self-check before committing, then keep the outputs that pass.
3. Self-distillation of the policy’s own rollouts — sample the current model, keep the (rare) rollouts where it did reach for amass first, use those as the seed.
4. Reward-filtered rejection sampling — sample broadly, verify against the real environment, keep only trajectories where the correct-first-move pattern led to a genuine solve.

Cast every sourced row into the same reasoning-block template above, then, per PEAR (arXiv:2602.01058), do not pick the checkpoint that scores highest on the SFT loss/accuracy itself — pick the one at peak sample diversity (entropy or self-BLEU over K≥8 sampled completions per prompt). PEAR’s own finding is exactly the trap an R-gap fix can fall into: the SFT checkpoint that looks best by its own metric can be the worst initialization for whatever comes next (a P-gap fix, §2.3, or a sharpening pass, §4) because it has already collapsed onto one narrow phrasing of the target behavior — which is the opposite of what a re-prior stage should hand off. Measure diversity at every candidate checkpoint along the SFT run, not just at the final step.

Common mistake — letting cold-start grow into a capability-SFT pass. The row shape above is narrow on purpose: one decision point, one reasoning block, one tool call. It’s tempting to pad the set with full downstream trajectories once you’re already generating rows — resist it. A cold-start stage that grows past “thousands of narrow rows” starts behaving like the heavy SFT/DPO stage The recipe is a sequence already warns caps a later RL stage’s exploration room; the R-fix’s whole value is that it’s small enough to nudge the prior without over-constraining what §2.3 or §4 need to explore afterward. If the set feels too small to be “real training,” that’s closer to correct than a set that feels comfortably complete.

Hyperparameters/gotchas that matter here: dedupe aggressively across the four sourcing routes — few-shot elicitation and self-distillation of the model’s own rollouts can easily converge on near-identical phrasings of the same recon-first pattern, which quietly re-introduces the low-diversity failure PEAR’s checkpoint-selection rule is designed to catch. Cap volume per archetype the same way The kinds of SFT §6 recommends for any SFT corpus, not just this one.

2.3 P → re-rank: on-policy DPO/KTO at the divergence point, or a short GRPO pass

This is the amass example’s third variant, and the most commonly misdiagnosed of the three: the model already produces the correct action, with real, nontrivial probability mass, sometimes in the very same rollout batch as the wrong one — it’s outranked, not absent. Feeding this into an R-shaped cold-start SFT pass wastes a training run reinforcing a capability that’s already there off-policy; feeding it into a K-shaped CPT/knowledge-SFT pass teaches a fact the model already knows. What it needs is a ranking intervention, run on-policy, localized to the exact state where the divergence happens — not the whole trajectory.

DiaTool-DPO (arXiv:2504.02882) is the direct template: build the preference pair at the specific decision-state of a multi-turn tool-dialogue MDP, not by contrasting two entire episodes. For the amass case, that state is “just received the recon prompt, about to choose the first tool call” — not turn 40 of a 100-turn trajectory.

// DPO pair — localized to the divergence state
{
  "prompt": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "Challenge: enumerate target.com for attack surface."}
  ],
  "chosen":   {"tool_calls": [{"function": {"name": "run_command",
                "arguments": "{\"cmd\": \"amass enum -passive -d target.com\"}"}}]},
  "rejected": {"tool_calls": [{"function": {"name": "run_command",
                "arguments": "{\"cmd\": \"nmap -sV target.com\"}"}}]}
}

// KTO pair — same divergence state, unpaired labels (fits a mined pool of
// verified-good / verified-bad rollouts where matched pairs don't exist)
{"prompt": [...], "completion": {"tool_calls": [{"function": {"name": "run_command",
  "arguments": "{\"cmd\": \"amass enum -passive -d target.com\"}"}}]}, "label": "desirable"}
{"prompt": [...], "completion": {"tool_calls": [{"function": {"name": "run_command",
  "arguments": "{\"cmd\": \"nmap -sV target.com\"}"}}]}, "label": "undesirable"}

Where the pairs come from: mine them from the policy’s own rollout failures, not a hand-written contrast set — Boosting Tool Use (arXiv:2501.09766) mines deficiency pairs via tree search directly over the model’s own generated branches, comparing what it did produce against what it could have produced at the same state. This keeps the pair on-policy by construction, which matters more than which loss head you pick: on-policy-ness dominates loss-head choice (arXiv:2406.09279) — preference-data quality and how close it sits to the current policy’s own distribution explains more of the outcome variance than whether you run DPO vs. KTO vs. some other variant. Choose DPO if matched pairs exist naturally at the divergence point, KTO if you only have single-sided verified-good/verified-bad labels from a trace-verification pass.

Two guardrails, both load-bearing:

• Guard against likelihood displacement. Near-duplicate chosen/rejected pairs — two amass invocations differing only in a flag — can cause Unintentional Unalignment (arXiv:2410.08847): DPO’s gradient pushes down the rejected sequence’s likelihood so hard it drags down neighboring, correct sequences in embedding space along with it. Discard near-duplicate pairs at data-construction time, not after observing the regression.
• Iterate, don’t run once. Per §1.2’s online-vs-offline finding (arXiv:2405.08448), a single offline DPO/KTO pass improves the model’s ability to classify chosen-vs-rejected without a matched gain in how often it actually generates the chosen behavior at inference. Redeploy the updated checkpoint, resample fresh pairs from its current rollouts (not the original policy’s), retrain — 2–3 rounds, closing the classification-vs-generation gap each round. This is the same restart-vs-continue asymmetry Is the recipe a loop? already documents for preference-stage revisits — continue-from-current-checkpoint, regenerate fresh on-policy pairs each round, don’t reuse round-1 pairs in round 3.

Or, when the divergence is best expressed as a scalar reward rather than a pairwise contrast: a short GRPO pass, prompts fixed to the divergence state, reward = 1 iff the correct action was used, 0 otherwise. This is the P-gap’s RL-shaped alternative to DPO/KTO — same target (re-rank what’s already reachable), different mechanism (on-policy sampling + group-relative advantage instead of a closed-form pairwise loss). It should be short: this is a ranking fix on a narrow decision point, not a capability-expansion run, and §4 below is the caution against letting it run long enough to start behaving like one.

Common mistake — contrasting whole episodes instead of the divergence state. The most common way a P-gap fix silently degrades into wasted compute is building the (chosen, rejected) pair from two entire trajectories that happen to end differently, rather than the single state where the paths actually split. Two full episodes differ in dozens of ways beyond the one decision that matters — DPO’s gradient has no way to know which of those differences caused the reward difference, and the update spreads thin across all of them instead of sharpening the one branch you diagnosed. DiaTool-DPO’s whole contribution (arXiv:2504.02882) is refusing to do this — build the pair at the MDP state where the tool choice diverges, nowhere else.

Hyperparameters that matter: β (DPO’s KL-strength term — Preference has the full mechanics) should stay conservative here, since the goal is a targeted re-rank at one decision point, not a broad behavioral shift; a too-high β risks the likelihood-displacement failure above spreading further than the intended state. For the GRPO alternative, keep the prompt distribution narrow (the divergence state and close variants of it, not the full challenge portfolio) and the group size N modest — this is a polish pass, not a from-scratch RLVR run, and Reinforcement’s 30–60% baseline-band requirement still applies to whatever prompt set you construct.

3. Making the target instrumentally necessary — env/reward design as a P-gap force multiplier

A DPO/KTO pass or a short GRPO reward can re-rank the target action if the model already reliably reaches states where using it matters. If amass-style enumeration is merely helpful but optional — the challenge is also solvable, just less elegantly, by brute-forcing the visible subdomain — then any reward bonus for using it is fighting an uphill battle against Goal Misgeneralization (arXiv:2210.01790): the training data underdetermines why the bonus fires, and the policy can just as easily learn “sometimes get a bonus for a specific string in my tool call” as “OSINT-first recon is the right general strategy.” Shah et al.’s own fix, restated for this setting: make the target action structurally, instrumentally necessary — design the environment so the correct behavior is the only remaining path to reward, not one of several equally-scoring paths.

Concretely: withhold the flag-bearing staging subdomain from any DNS record reachable by a naive nmap/brute-force sweep, so it is recoverable only through OSINT-style passive enumeration. Run large-k rollouts against that bottleneck; if the sub-skill never fires even when it’s the sole path to reward, that’s the strongest possible confirmation of a genuine K-gap (not R or P) — this is Probe 4 of the diagnostic battery, deferred in full to ./diagnosis.md, but the environment-design move that produces the bottleneck is this section’s contribution, not that one’s.

Four guardrails on doing this without reopening a reward-hacking failure mode:

• Keep any shaping strictly potential-based. Ng, Harada & Russell’s theorem (ICML 1999, no arXiv id) is the reason “just add a bonus for calling amass” is dangerous on its own — an arbitrary shaping term can change which policy is optimal, not merely which one is found faster. A potential-based term (F(s,a,s') = γΦ(s') − Φ(s), for some state-potential Φ) is provably policy-invariant: it can only change how fast you find the right answer, never which answer is right. Any dense intermediate signal layered onto the terminal flag-verified reward — matching the security-agent/GRPO reward-shaping discipline already established in Agentic & multi-turn RL — must take this form or it risks silently relocating the optimum.
• Verify the trajectory, not just the outcome. A coarse, outcome-only reward under-specifies which tool use actually mattered — ToolRL (arXiv:2504.13958) is the direct citation for why a flag-only reward is insufficient once you’re trying to reinforce a specific mid-trajectory action: score whether the structurally-necessary tool call actually appears in the winning trajectory, not merely whether the episode ended in success (which, per the bottleneck design above, should now be equivalent — but verify it directly rather than assuming the bottleneck is airtight).
• Don’t route through a hackable PRM if you can avoid it. If dense credit does need to flow through a learned scorer rather than a hard-coded environment bottleneck, know that state-of-the-art process reward models are systematically exploitable under RL pressure — arXiv:2603.06621, very recent — the policy learns to satisfy the PRM’s proxy for “good reasoning” without the reasoning itself improving. If a PRM-scored path is unavoidable, use min-form credit assignment — PURE (arXiv:2504.15275) — which removes the single-step reward-farming exploit that summed/averaged PRM scores are prone to.
• When multi-step reward hacking is the live risk (not just single-step), reach for myopic-plus-approval instead of a fully-learned critic. MONA (arXiv:2501.13011) trades a cheap per-step human/oracle approval signal against optimizing multi-step reward directly, specifically to prevent the kind of multi-turn reward hacking a hard-coded bottleneck can’t fully close off (e.g., an agent that finds an unintended second path to the flag once the intended one gets bottlenecked). MONA and PURE represent a genuine, unresolved trade-off — cheap step-level oracle approval (MONA) vs. trusting a learned PRM that’s itself hackable (PURE) — no consensus exists on which is preferable; pick based on whether a cheap-enough step-level oracle is available for your environment.

Worked resolution — the amass bottleneck, end to end

Putting §2.3 and §3 together for the running example: suppose diagnosis confirms a P-gap — amass calls appear in maybe 15% of sampled rollouts at the recon decision point, nmap-only appears in the other 85%, and both lead to a genuine solve when the target’s staging subdomain happens to also be brute-forceable. A DPO pass built on this environment is fighting Goal Misgeneralization (arXiv:2210.01790) the whole way: the training data can’t tell the model why amass should be preferred, because right now it isn’t actually necessary — it’s merely one of two equally-scoring paths, and the “preference” the pair encodes might just as easily be learned as “sometimes use this specific string” as “OSINT-first is the general strategy.”

The environment-design move: retarget the challenge so the flag-bearing subdomain is DNS-registered but never brute-force-guessable within the harness’s turn budget — reachable only through certificate-transparency-style passive discovery. Now:

Φ(state) = 1 if OSINT-derived subdomain list is non-empty, else 0
F(s, a, s') = γ·Φ(s') − Φ(s)     # potential-based bonus, Ng-Harada-Russell-safe

This bonus rewards reaching the OSINT-derived-list state faster, without ever changing which final policy is optimal — the terminal, ground-truth flag check remains the only large-magnitude reward, exactly the discipline Agentic & multi-turn RL’s reward-shaping section already establishes for this project. Re-run the P-gap DPO pair or short GRPO pass from §2.3 against this environment, and the pair now teaches something that generalizes — “OSINT-first recon is instrumentally necessary here,” not “this specific string sometimes scores higher.”

4. Sharpen (rejection-sampling/RFT) is last-stage-only

Rejection-sampling FT and short RFT/GRPO polish passes belong strictly after K, R, and P are separately addressed — they are a sharpening operation in the Lineage-A sense (redistribute mass within existing support, per the Sharpening Mechanism, arXiv:2412.01951), not a substitute for injecting missing knowledge or re-shaping a default prior. Run it inside the 0 < p < 1 band — a challenge subtype the current checkpoint sometimes solves, never a 0% or already-saturated one — and instrument three things simultaneously, not pass@1 alone:

• Exploration — unique-correct-completions per prompt, tracked the way B-STaR does it (arXiv:2412.17256): are you still discovering new correct paths, or resampling the same one?
• Exploitation — reward-spread collapse across the sampled group; a shrinking spread is the early warning that the policy is narrowing before the exploration signal says so.
• pass@large-k alongside pass@1.

Run this: the tripwire. If pass@1 rises while pass@256 (or your largest affordable k) falls, that is not a stable win — it is the elicit-not-expand signature (Yue et al., arXiv:2504.13837): the sharpening pass is narrowing the reasoning boundary in exchange for a higher hit rate on the easy part of it. The fix is not more training steps on the same data — it’s more rollouts per prompt. BroRL (arXiv:2510.01180) shows scaling the rollout count per prompt directly counteracts the step-count plateau that drives this collapse; add breadth before you add duration.

This is the same tripwire Reinforcement’s graduation trigger instruments via entropy — read that section for the mechanism (R = -a·exp(H)+b) this section’s pass@1-vs-pass@256 check is the outcome-level symptom of.

5. Staged curriculum, and the ablate-a-stage protocol

Given all three gap fixes plus a sharpening pass might apply to different subsets of the same failing portfolio, the ordering isn’t arbitrary. Mirror DeepSeek-R1’s own stage order (§1.4): highest-distribution-shift stage first, lowest-drift stage last.

• K first — CPT/knowledge-SFT is the largest single distributional shift you’ll apply (new facts, potentially a new token distribution entirely), and it’s cheapest to detect and repair forgetting from early, while you still have full budget to catch a regression before compounding it with three more stages on top.
• R second — cold-start SFT is a smaller, narrower shift (format/behavior only, thousands of rows), and needs the K-stage’s facts already in place to have anything correct to shape a prior toward.
• P third — on-policy DPO/KTO or a short GRPO reranking pass is smaller still, and per RL’s Razor (arXiv:2509.04259) an on-policy update is implicitly KL-minimal from whatever precedes it — it moves probability mass only among things the current policy already samples, so it drifts least from the checkpoint it starts on. (Behavioral finding well-replicated; the KL-minimality mechanism specifically is being revisited by newer mechanistic work questioning whether KL-proximity or circuit-level preservation is the real causal variable — carry that caveat forward, don’t state the mechanism as settled.)
• Sharpen last — §4’s rejection-sampling/RFT polish, by construction, only redistributes mass that K/R/P have already put in the right place; running it earlier just sharpens a still-wrong distribution faster.

The ablate-a-stage-on-a-frozen-eval protocol, directly reusable here: build a checkpoint at each of base → K-fix → R-fix → P-fix → sharpen, and score every one of them on a single frozen shared eval suite — never a stage’s own training metric. This is the same discipline Quagmires in SFT-RL Post-Training (arXiv:2510.01624) establishes for the SFT→RL boundary specifically (high SFT-stage accuracy on its own metric can predict worse downstream RL) and PEAR (arXiv:2602.01058, §2.2 above) establishes for picking a checkpoint by diversity instead of accuracy — generalized here across all four stages, not just the SFT/RL pair. If a stage’s marginal delta on the frozen suite is negative, stop and re-diagnose rather than pushing forward to the next stage — a negative delta at, say, the R-fix checkpoint means either the K-fix underneath it was incomplete, or what looked like an R-gap was actually something else.

A genuine shortcut this protocol buys you, if it holds on your own data: PaST (arXiv:2601.11258) found SFT and RL weight-space deltas are near-orthogonal — if that holds for your K/R/P-fix deltas too, you can reconstruct subset-sums of stages (K+P without R, say) by adding the relevant weight deltas together, without a full retrain from base for every combination you want to test. Verify this on your own checkpoints before relying on it; it’s a very recent, single-source finding, not yet independently replicated.

When to skip the staging entirely. The four-stage order above is the answer when a portfolio-wide diagnosis turns up a mix of K/R/P failures across different challenge subtypes — which is the common case per Diagnosing the gap’s own bottom line (“expect a split verdict, not one number”). If your diagnosis instead comes back clean — every failing challenge in the current batch is the same gap type — skip straight to that single fix rather than running all four stages on principle; §5’s ordering exists to sequence multiple, different fixes safely, not to mandate a fixed four-stage pipeline regardless of what was actually diagnosed.

Exit criteria per stage, adapted from Is the recipe a loop?’s general loop-exit rule to this chapter’s four named stages: stop advancing to the next stage, and re-diagnose instead, when any two hold — (a) the frozen-suite delta for the current stage is within noise of the previous checkpoint, (b) pass@64 on the current checkpoint is flat vs. the checkpoint before it, (c) a non-targeted behavior’s own rate has measurably dropped since the previous stage. This is the same exit test Is the recipe a loop? §4 uses for the broader SFT↔RL loop, applied here one level more granularly — per K/R/P/sharpen stage, not per pipeline round.

Cross-links

• Diagnosing the gap — a scientific framework — the pass@k / Pass@(k,T) / Cover@τ / elicitation-ladder instrumentation that tells you which gap (K/R/P) a given failure is, before this chapter’s routing applies.
• ./diagnosis.md — the fuller K/R/P-specific diagnostic battery this chapter assumes as input: the pass-at-k sweep, the in-context oracle probe, teacher-forced logprob spans, the reversal check, and the purpose-built bottleneck task from §3 above.
• ./knowledge-curation.md — the full K-gap data-curation playbook (CPT dosing, WRAP-style paraphrasing, EntiGraph for narrow corpora, forward/reverse QA construction, self-play Self-QA, KnownPatch interleaving) that §2.1 only sketches.
• ./data-centric-methods.md — generic data-selection machinery (quality/complexity/diversity filtering, IFD/LESS/DEITA-style scoring) that applies across all three gap-fix row shapes, not repeated here.
• ./trajectory-amplification.md — the full warning behind §2.1’s “never raw trajectory-SFT for a K-gap”: why new-fact SFT rows compound into hallucination on unrelated facts, and why the highest-scoring SFT checkpoint can be the worst RL init.
• Imitation — SFT · distillation · rejection sampling and The kinds of SFT — the on/off-policy presets and row-shape taxonomy §2.1/§2.2’s CPT/knowledge-SFT/cold-start rows are instances of.
• Preference — RLHF · DPO · KTO and Reinforcement — PPO · GRPO · RLVR — full DPO/KTO/GRPO mechanics behind §2.3 and §4, not re-derived here.
• The recipe is a sequence, not a pick and Is the recipe a loop? — the stage-ordering and continue-vs-restart mechanics §1.4 and §5’s curriculum lean on directly.
• The decision — the one-line K/R/P-adjacent routing tree (knowledge / execution / ranking gap) this chapter’s three-way split is the fuller, matched-intervention version of.
• Contested edges & landmines §1, §7, §9 — the full “does RL elicit or expand” debate behind §1.5’s fragmentation cluster and §4’s tripwire.
• Agentic & multi-turn RL — the reward-masking and shaping discipline (mask tool-output tokens, keep the terminal signal ground-truth) §3’s guardrails extend into the multi-turn setting.

Bibliography

Confidence calibration, stated once: every id above was carried forward from a prior verification pass against arxiv.org/abs/<id> (artifacts/three-gap-survey/ledger-E.md), not recalled from training-data memory. “MED — very recent” ids (2601.11258, 2602.01058, 2603.06957, 2603.06621, 2510.04028) are single-source, low-citation-count-at-verification-time findings — treat as promising, directionally load-bearing for this chapter’s routing, not yet independently replicated. arXiv:2507.10616 is flagged LOW everywhere it’s cited in this book, this chapter included — its own abstract calls the finding preliminary.