The three gaps — knowledge · prior · policy

The question this chapter answers: your agent doesn’t do the expert thing. On a fresh challenge it fires a scattered curl or dig instead of running a recon methodology; it reaches for subfinder but never amass; and when it does run subfinder it uses the defaults, not your flag-set. Before you spend a single GPU-hour on SFT, DPO, or GRPO, you have to answer one question: which of three gaps is this? Because the three have different, non-substitutable fixes, and picking the wrong lever doesn’t just waste the run — for one of them it actively makes the model worse.

This short chapter is the map. The six chapters after it are the territory, each fully cited.

Bottom line up front

Your symptom decomposes into three candidate gaps, and the whole game is telling them apart:

The load-bearing fact that makes K genuinely distinct from R/P — and this is a theorem, not a heuristic — is the closed-form of every KL-regularized objective. DPO’s optimum is π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β) arXiv:2305.18290, and RLVR is formally bounded to the base model’s support arXiv:2507.14843. Any RL or preference method can only redistribute probability mass the reference policy already assigns somewhere. If π_ref(y*) ≈ 0, the gradient has nothing to grab. That is precisely why “just RL it” fails on a knowledge gap and why the diagnosis must come before the training run.

The headline: your tool/flag failure is almost certainly a P-gap

You told me the failure is universal across frontier models — every one of them reaches for the salient tool and the default flags. That universality is itself the diagnosis: it’s a policy-prior signature, not a weak base model. Generic instruction+agentic post-training optimizes plausible general helpfulness, which sharpens the default distribution toward the single most-salient action. Expert procedure (the full tool-set, the full flag-set, a disciplined sequence) is long-tail behavior that generic RLHF under-rewards. So no lab’s post-training taught it — which reframes your fix from “get a better model” to “supply the data + reward that make expert methodology the default policy.”

Two 2026 mechanistic results make this concrete for exactly your example (both recent/low-citation — promising, not yet broadly validated — but strikingly direct):

• Akgül et al., It’s Sparse Policy Selection, Not Capability Learning arXiv:2605.06241: RL changes only 1–3% of token positions, concentrated at high-entropy decision points, and the token it promotes is always already in the base model’s top-5 logits. Operationalized: if the correct tool/argument token sits in the base model’s top-5 at the decision step, you are in a checkable P-gap and RL/DPO is well-targeted.
• Chen, Looking Is Not Picking arXiv:2606.16364: on real tool-call failures, per-candidate attention shows the model attends to the correct tool 80% of the time (vs 21% chance) yet still calls the wrong one. Prompt-side fixes (reordering/duplicating the tool in the schema) recover ≤23% of failures; decision-readout-layer interventions recover 59–91%. The model sees amass; it mis-picks at the readout. That’s a P-gap, and it tells you where the fix lands (the decision layer, i.e. on-policy preference/RL) — not in the knowledge.

The deep version of this taxonomy and its three-literature genealogy is in the taxonomy chapter.

The diagnosis, on one screen

You don’t guess which gap — you run a cheapest-first probe battery. The full runnable version (with numbers, pseudocode, and readouts) is Diagnosing which gap; here is the skeleton:

flowchart TD
  S["Symptom: agent won't run amass /<br/>won't use expert flags / no opening plan"] --> PK["pass@k sweep: grep 50 rollouts<br/>for the expert action (temp 0.7-1.0)"]
  PK -->|"appears sometimes"| RP["in-support → R or P gap"]
  PK -->|"never, even at k=256<br/>across ≥10 rephrasings"| ORC["in-context ORACLE probe:<br/>put the answer/flags in the prompt,<br/>re-run"]
  ORC -->|"now it does it"| Kd["was surfacing, not absent →<br/>R/P gap after all"]
  ORC -->|"still can't, even handed the answer"| K["KNOWLEDGE gap →<br/>inject (curation chapter)"]
  RP --> TOK["token-level: is the expert token<br/>in the base model's top-5?<br/>(arXiv:2605.06241)"]
  TOK -->|"yes"| P["POLICY gap →<br/>on-policy DPO / GRPO<br/>(1-3% of tokens move)"]
  TOK -->|"in-support but buried"| R["PRIOR gap →<br/>cold-start SFT re-prior"]
  classDef fix fill:#132b22,stroke:#34d399,color:#eafaf3;
  class K,P,R fix;

The two probes that carry the most weight, in plain terms:

pass@k = run the same task N times (say 50) at temperature ~0.8 and ask “does the expert action ever appear?” — score it with the unbiased estimator 1 − C(N−c,k)/C(N,k), never naïve c/k. Ever appears → in-support → R/P. Never, even at k=256 across ≥10 rephrasings → a knowledge-gap candidate. (Caveat: for pure factual recall a small k can false-positive a K-gap arXiv:2605.07153 — push k high, and on a multi-turn agent use Pass@(k,T), not Pass@k.)

in-context oracle probe = put the answer in the prompt ("expert workflow: subfinder -d T -all -recursive AND amass enum -passive -d T") and re-run. If it now executes correctly, the machinery was there and only the default policy failed to deploy it → R/P gap. If it still can’t even when handed the answer → genuine K gap. This is the single cleanest K-vs-P splitter you can run in an afternoon, no training.

The warning you already sensed: trajectory SFT amplifies an unfixed gap

Your instinct — “everyone jumps to trajectory-level [SFT]; maybe that’s been amplifying the problem” — is correct and citable. If you SFT trajectories that use knowledge the base model doesn’t actually have, you teach it to imitate the surface form of expert tool-use without the grounding: new-fact SFT rows are learned slower and, once learned, linearly increase hallucination on previously-known facts arXiv:2405.05904. Layer that on an unfixed K-gap and you manufacture a confident fabricator. The ordering rule that falls out: diagnose first → fix knowledge/prior → trajectory-SFT last. The full evidence chain is Does trajectory SFT amplify an unfixed gap?.

How to read this section

Cross-links

• Diagnosing the gap — a scientific framework — the complementary framing (knowledge / execution / exploration gap). The mapping is exact, not a loose overlap: execution = R ∪ P — the same continuous in-support-but-mis-ranked axis this chapter’s taxonomy defines, at two granularities (R = macro/plan-level, P = micro/decision-point); exploration is not a fourth gap but a training-dynamics modifier that attaches to an R/P failure when its winning path is sequentially-gated rather than single-shot. taxonomy.md is canonical for this mapping — read it before reconciling the two vocabularies.
• The kinds of SFT — it is the data, not the algorithm — the SFT data-shape taxonomy the curation chapters build on; its §4 (synthetic-authoring → confabulation) is the twin of trajectory-amplification.
• The one axis that predicts everything — the on/off-policy distinction underneath every “inject vs amplify” decision here.
• Contested edges & landmines — the elicit-vs-expand debate that most of the diagnosis rests on is genuinely unresolved; treat it as contested.

Bibliography (ids verified live in the survey pass, 2026-07-02)

Standing rule (all chapters in this section): no load-bearing claim rests on an academic cybersecurity-LLM training/benchmark paper — every citation is general ML/RL theory or frontier-lab evidence. The cyber running example is the motivation, not the basis.