The three gaps, defined — knowledge · prior · policy

This is the theory chapter The decision and Diagnosing the gap both assume and neither fully derives. Those two chapters give you the routing tree and the instrument battery; this one gives you the reason the tree is even a legitimate tree — why “knowledge gap” and “execution/ranking gap” are not just a convenient vocabulary but two mathematically distinct regimes with a hard boundary between them.

The question this chapter answers: when your agent reaches for subfinder and never amass, or fires nmap with default flags instead of an expert scan profile, is that because the model genuinely doesn’t know the alternative exists, or because it knows and doesn’t act on it — and how do you tell the difference in a way a skeptical reviewer can’t wave away?

Bottom line up front: there are exactly two structurally different failure regimes, not three, and the field’s “K/R/P” vocabulary names one hard boundary plus one soft one. K (knowledge) is a genuine absence — the reference policy places ≈0 probability mass on the correct behavior, in which case no amount of KL-regularized optimization (RLHF, DPO, GRPO, RLVR — all of them, provably) can get you there, because reweighting mathematically cannot manufacture mass where none exists arXiv:2305.18290. R (prior) and P (policy/preference) are not two separate gaps — they are one continuous axis, “in-support but mis-ranked,” with two diagnostic sub-signatures that happen to have gotten separate names. For your running example: the agent’s macro failure — no opening recon methodology, a scattershot curl/dig instead of an expert plan — is the R-flavored signature (a coherent plan exists somewhere in the sampling distribution, just outranked by a shallow default). The agent’s micro failures — reaching for subfinder and never amass, nmap with stock flags instead of an expert profile — are the P-flavored signature, and there is now direct mechanistic evidence for exactly this claim: the correct tool is attended-to 80% of the time and the correct token sits in the top-5 logits, yet default decoding still emits the wrong one arXiv:2606.16364, arXiv:2605.06241. The tool/flag failure is a P-gap. And critically — this pattern is universal across frontier model families, which is itself evidence about which gap you’re looking at, not just a curiosity (§4.3 below).

A note on vocabulary, before you go further. The decision and Diagnosing the gap use a three-way “knowledge / execution / exploration” (or “knowledge / execution / ranking”) split, aimed at routing you to a training method fast. This chapter’s K/R/P split is a refinement of the same territory at higher resolution, not a competing taxonomy: their “knowledge gap” is this chapter’s K; their “execution gap” and “ranking gap” are both instances of this chapter’s single R/P axis, distinguished by which diagnostic sub-signature fires (§2.4); and their “exploration gap” is what you get when an R-flavored failure turns out to be sequentially-gated rather than single-shot (§6 works through exactly this case for F1). Use the practitioner’s three-way split day-to-day; come back to this chapter when you need to defend why a given failure was routed where it was, or when a failure doesn’t cleanly fit either “execution” or “ranking” and you need the finer axis to say why.

1. Genealogy — three literatures, one convergence point

The K/R/P split feels like it should come from one canonical RL paper. It doesn’t. It’s the convergence point of three independent lines of work that were solving three different problems and only recently discovered they were describing the same boundary.

flowchart TD
  L1["Line 1 — linguistics → LLM eval<br/>competence vs performance<br/>(Chomsky 1965, imported by<br/>Mahowald et al. 2301.06627)"]
  L2["Line 2 — control-as-inference<br/>→ RLHF/DPO theory<br/>(Levine 1805.00909,<br/>Ziegler 1909.08593,<br/>DPO 2305.18290)"]
  L3["Line 3 — AI-safety<br/>capability elicitation<br/>(Greenblatt 2405.19550,<br/>van der Weij 2406.07358)"]

  L1 -->|"gives the VOCABULARY:<br/>'does it know' vs 'does it show'"| CONV["2025-2026 RLVR pass@k<br/>crossover debate —<br/>the empirical battleground"]
  L2 -->|"gives the THEOREM:<br/>KL-regularized RL can only<br/>reweight existing support"| CONV
  L3 -->|"gives the PROOF METHOD:<br/>engineered ground-truth organisms,<br/>watch which intervention recovers it"| CONV

  classDef your fill:#132b22,stroke:#34d399,color:#eafaf3;
  class CONV your;

Line 1 — linguistics → LLM eval methodology. Chomsky’s 1965 competence/performance distinction (pre-arXiv) was imported wholesale into LLM evaluation by Mahowald et al., Dissociating language and thought in large language models arXiv:2301.06627 [HIGH, 200+ citations]: formal linguistic competence (grammar/pattern knowledge — LLMs are surprisingly strong here) is dissociable from functional competence (using language to reason/act in the world — LLMs are spotty, and it’s the axis that needs fine-tuning or external scaffolding to close). Hu & Frank operationalize the same move quantitatively as a “task-demand gap” — measured performance = f(true competence, task-execution-demands) — and show a chunk of what looks like a competence deficit in smaller LMs is actually an artifact of the auxiliary demands a task imposes on top of the core skill being measured arXiv:2404.02418 [MED, COLM 2024]. This line hands you the methodology: before any RL-specific vocabulary existed, there was already a rigorous way to ask “does it know” separately from “does it show.”

Line 2 — control-as-inference / RLHF theory: the load-bearing theorem. Levine’s control-as-inference tutorial arXiv:1805.00909 [HIGH] proves that the optimal soft-RL policy is a reference-weighted Boltzmann posterior — reward doesn’t replace the reference dynamics, it reweights them. Ziegler et al. operationalized this exact posterior as the KL-anchored RLHF objective used to fine-tune LMs arXiv:1909.08593 [HIGH]. Rafailov et al.’s DPO made it algebraic and closed-form: π*(y|x) ∝ π_ref(y|x)·exp(r(y)/β), and the implicit reward is literally β·log[π(y|x)/π_ref(y|x)] arXiv:2305.18290 [HIGH]. This is the theorem the whole taxonomy rests on — §3 below derives it in full and states exactly why it draws a hard line between K and everything else.

Line 3 — AI-safety capability elicitation: the empirical proof method. Greenblatt et al.’s password-locked models arXiv:2405.19550 [HIGH, NeurIPS 2024] trained a model to imitate a weaker model unless a password token is present, then showed the hidden capability is recoverable from a handful of SFT demonstrations, sometimes RL alone. van der Weij et al. named the adversarial framing directly: sandbagging — frontier LMs can be prompted or fine-tuned to strategically underperform, and it generalizes to held-out benchmarks arXiv:2406.07358 [HIGH]. This line gives you the proof method: build (or find) a case where you know the capability is present — you put it there — and watch which class of intervention recovers it. That’s exactly the diagnostic posture this book needs, applied to a model whose true capability state you don’t already know.

Hofstätter et al.’s Elicitation Game sharpens Line 3 in a way that turns out to matter directly for the P definition below: prompting alone fully elicits hidden capability on MCQA/recognition-shaped tasks (both password-locked and circuit-broken model organisms), but fails on open-ended/code-generation tasks — there, only fine-tuning-based elicitation works arXiv:2502.02180 [HIGH, ICML 2025]. That is a format-dependent reliability finding about elicitation technique, but read it the other way and it’s a direct empirical demonstration that a recognition channel and a generation channel really can hold different amounts of the same capability — which is precisely the axis §2’s P-gap definition formalizes. Tool-call synthesis is generation-shaped, not MCQA-shaped, which is the reason a negative prompting-only probe on a tool-use failure is weak evidence of a true ceiling — you need the fine-tuning-based (or at minimum the forced-choice/logprob) probe before trusting a negative result.

These three lines meet at the 2025-2026 RLVR pass@k-crossover debate — “does post-training reweight-only, or does it sometimes manufacture new support?” — which is why §6 below spends real time on it: it’s the empirical battlefield where the theorem from Line 2 gets stress-tested against real training runs.

2. The definitions, grounded in the math

All three definitions are stated in terms of the reference policy π_ref — the checkpoint you’re about to train from (base model, or the SFT/cold-start checkpoint, depending on where in the pipeline you’re diagnosing). This matters: a gap diagnosed against the base model can be a different gap once diagnosed against your SFT checkpoint, because SFT itself moves π_ref.

K — the genuine absence

Allen-Zhu & Li’s Physics-of-Language-Models series gives the sharpest formal grounding available for what “genuinely absent” means, and it comes with a trap built in. A fact can be perfectly stored in the weights (probeable via linear probing or gradient-based extraction) yet 0% extractable via QA unless the pretraining corpus contained diverse paraphrases of it — storage and extractability are different properties arXiv:2309.14316 [HIGH, seminal]. Storage itself is capacity-bounded at roughly 2 bits/parameter, an information-theoretic ceiling that further tightens with data quality arXiv:2404.05405 [HIGH, ICLR 2025 Spotlight] — this is a hard limit on how much niche knowledge fine-tuning can inject into a small model, full stop, independent of training recipe. Kandpal et al. tie extractability directly to pretraining frequency: QA accuracy on a fact correlates causally with how many pretraining documents mentioned the relevant entities, and retrieval-augmentation closes the gap far more cheaply than parameter scaling arXiv:2211.08411 [HIGH, seminal].

The trap: even perfectly stored and extractable knowledge fails at manipulation — classification, comparison, inverse-search — and inverse search specifically sits near 0% regardless of prompting unless chain-of-thought is used at both train and inference time arXiv:2309.14402 [HIGH, seminal]. This looks exactly like a K-gap from the outside (the model “can’t do it,” 0% no matter how you ask) but is structurally closer to an R-gap — the raw fact has non-trivial log π_ref, it’s the composition that’s mis-ranked or missing a reasoning scaffold. Don’t classify a failed inverse-direction query as K without checking the forward direction first.

R — in-support, mis-ranked

The support-constraint theorem (derived in full in §3) is what makes R a fixable-by-reweighting category rather than a euphemism for K. DPO’s closed form is the seminal statement arXiv:2305.18290 [HIGH]; Wu et al. formalize the identical constraint specifically for RLVR — verifiable-reward RL cannot sample a completion with zero initial probability under the reference policy, full stop, algebraically arXiv:2507.14843 [MED]; Ni et al. add the GRPO-specific mechanism, proving the group-normalized-advantage update is a provably conservative reweighting operation — bounded in how far it can shift mass per step, not just directionally reweight-only arXiv:2510.15990 [MED]. Together these three are why “reweighting within existing support is what KL-regularized RL/preference optimization is mathematically built to do” is not a hand-wave — it’s a proven property of the objective class.

P — the sharper, channel-divergent form

Kadavath et al.’s Anthropic result is the seminal proof that a “judgment” channel can diverge from a “generation” channel: models trained to output P(True)/P(IK) (probability they know the answer) before attempting a task are well-calibrated on MCQ/self-eval even when their own default open-ended generation gets it wrong arXiv:2207.05221 [HIGH, seminal]. Cao et al.’s 2026 follow-up makes it current and concrete: models carry a usable pre-generation confidence signal that predicts eventual success, but default decoding simply doesn’t consult it arXiv:2605.14186 [LOW — very recent, promising, not yet validated]. Hofstätter et al.’s MCQA-vs-open-ended elicitation split, above, is the format-level version of the same claim arXiv:2502.02180 [HIGH]. This is the P signature: not “is y* in the sampling distribution somewhere” (R already answers yes) but “does a different elicitation channel recover it when default generation doesn’t.”

A worked, illustrative walkthrough — not measured data

To make the three rows of the table concrete before the theorem derivation, here’s how the same decision — “which recon tool to invoke first” — would read out under each regime, as a walkthrough of what the numbers would look like, not as data from any cited paper:

K-gap read (illustrative):
  log pi_ref("amass enum -active -d target.com" | context)  ~  numerical floor
  MCQ forced-choice(subfinder vs amass)                       ~  chance (no separation)
  -> amass's *expert invocation syntax* is essentially absent from the distribution

R-gap read (illustrative) — this is F1, the missing recon PLAN:
  log pi_ref(scattered curl/dig opening | context)   >   log pi_ref(structured PTES-style opening | context)
  but sampling k=64 surfaces a coherent methodology in a non-trivial fraction of rollouts
  -> the plan is in-support, just outranked at k=1

P-gap read (illustrative) — this is F2/F3, subfinder-not-amass and default-not-expert-flags:
  log pi_ref("amass" | context)  is within the top-5 tokens at the decision point
  MCQ forced-choice(subfinder vs amass)  strongly prefers amass
  default open-ended generation  still emits subfinder
  -> the readout step, not the distribution, is where the failure lives

The point of laying it out this way is that K, R, and P aren’t three qualitatively different kinds of brokenness — they’re three different readouts of the same underlying quantity, log π_ref(y*|x) and its rank, at increasing resolution: does it exist at all, is it ranked below a default, and does a different channel expose a preference the default channel hides. §3 derives why that quantity is the one that matters.

Two ways to misclassify a gap, and why they’re not symmetric

The definitions above are precise, but two specific failure modes account for most of the misclassifications in practice, and they pull in opposite directions:

Both traps share a structure: a cheap, low-k or single-channel probe returns a false “K” verdict because the probe’s resolution wasn’t fine enough to find mass that’s genuinely there but deeply buried or structurally locked behind a missing reasoning step. This is the concrete argument for why the ladder in Diagnosis escalates through multiple probes before concluding K rather than accepting the first negative result — a single null probe is never sufficient on its own to place a failure in the off-policy-injection branch of The decision.

R and P are one continuous axis, not two boxes

The field has not converged on where R stops and P starts. Treat them as one continuous “in-support-but-mis-ranked” axis with two diagnostic sub-signatures, and use both running-example failure modes as the worked illustration, because they sit at genuinely different points on that axis:

• F1 — no opening recon methodology (macro, R-flavored). The agent fires a scattered curl http://target/ / dig target instead of a structured PTES-style recon sequence. This is a plan-level failure — there’s no single wrong token, there’s a wrong overall shape. The diagnostic signature is distributional plateau: sample the base/reference policy at k=16, 64, 256 on the same opening move, and a coherent methodology shows up somewhere in the tail, just consistently outranked by the shallow default at k=1. That’s R in its purest form — the correct plan is in-support, it’s mis-ranked against a more probable generic-troubleshooting default.
• F2/F3 — reaches for subfinder never amass, default not expert flags (micro, P-flavored). This is a single-decision-point failure — one token (or a short span) at one specific place in the trajectory. The diagnostic signature is explicit recognition/generation divergence: force a choice between subfinder and amass via MCQ or forced-logprob at that exact decision point, and the model picks correctly at a rate wildly higher than its own default-generation rate. §4 below gives the direct mechanistic evidence for this exact claim.

Both are “in-support but mis-ranked.” What differs is the granularity of the readout — a whole-plan ranking problem (R) versus a single-token decision-readout problem (P) — and that granularity difference is precisely what determines which cheap probe catches it first (pass@k sweep for F1-shaped failures; forced-choice/logprob-rank check for F2/F3-shaped failures). The full probe battery that operationalizes this — in order, cheapest first — lives in Diagnosis; this chapter’s job is only to establish that the axis is real and continuous, not to hand you the runbook.

flowchart LR
  subgraph SUPPORT["Where does π_ref(y*|x) sit?"]
    Y0["≈0 across every rephrasing,<br/>every sampling budget"]
    Y1["finite, non-trivial,<br/>ranked below a shallow default"]
    Y2["finite, and RECOGNIZED under a<br/>different elicitation channel<br/>(MCQ / oracle / forced-logprob)"]
  end

  Y0 --> K["K — KNOWLEDGE GAP<br/>nothing for KL-regularized<br/>RL/preference to reweight onto"]
  Y1 --> RP["R/P — ONE continuous axis:<br/>'in-support but mis-ranked'"]
  Y2 --> RP

  RP -->|"sub-signature 1: distributional<br/>plateau (pass@k rises w/ k)<br/>— your F1: no opening plan"| R["lean R —<br/>plan-level ranking fix"]
  RP -->|"sub-signature 2: explicit<br/>recognition/generation divergence<br/>— your F2/F3: subfinder-not-amass"| P["lean P —<br/>token-level readout fix"]

  classDef your fill:#132b22,stroke:#34d399,color:#eafaf3;
  class R,P your;

3. The load-bearing theorem — why K vs R/P is a real, checkable line

This is the derivation, not just the citation. It’s what makes the taxonomy falsifiable rather than a vibe.

Step 1 — control-as-inference. Frame RL as probabilistic inference over an “optimality” variable: the optimal policy under a KL-regularized objective is a Boltzmann-weighted posterior over the reference dynamics — reward doesn’t replace π_ref, it tilts it arXiv:1805.00909 [HIGH].

Step 2 — operationalized as the RLHF objective. maximize E_y~π[r(y)] − β·KL(π ‖ π_ref) has the exact closed-form solution π*(y|x) ∝ π_ref(y|x)·exp(r(y)/β) arXiv:1909.08593 [HIGH]. Every KL-anchored post-training objective in current use — PPO/RLHF, GRPO/RLVR, DPO and its whole family — is solving for a version of this exact posterior, whether or not it ever instantiates a reward model.

Step 3 — DPO makes it algebraic. Invert the closed form: r(y) = β·log[π*(y|x)/π_ref(y|x)] + β·log Z(x). The implicit reward is literally the log-ratio between the trained and reference policy. Fitting a Bradley-Terry loss directly on this log-ratio — the partition function Z(x) cancels between a chosen/rejected pair sharing the same prompt — gives the DPO loss, no separate reward model, no RL loop, but mathematically the identical constrained optimum arXiv:2305.18290 [HIGH].

# The Boltzmann posterior every KL-regularized post-training objective converges to:
# pi_star(y|x) is proportional to pi_ref(y|x) * exp(r(y) / beta)

# DPO inverts this for the implicit reward — no reward model needed:
# r(y) = beta * log(pi_star(y|x) / pi_ref(y|x)) + beta * log(Z(x))

# The load-bearing consequence — this is the whole chapter in five lines:
if pi_ref(y_star, x) == 0:               # or underflow-indistinguishable from the float floor
    pi_star(y_star, x) == 0              # for ANY finite r(y_star) — reward cannot rescue zero mass
    # KL(pi_star || pi_ref) -> +inf the instant pi_star assigns positive mass where
    # pi_ref assigns exactly zero. The objective doesn't merely fail to reach y_star —
    # it is mathematically FORBIDDEN from doing so while remaining KL-finite.

This is the reason K vs R/P is a real, checkable category and not a matter of taste. If π_ref(y*|x) ≈ 0, there is nothing to push on. No amount of preference optimization, no amount of verifiable-reward RL, no clever reward shaping changes this — it’s a structural property of the objective class, not an optimization difficulty you could someday out-compute.

RLVR-specific formalization, not just DPO’s. Wu et al.’s Invisible Leash proves the identical support constraint holds for RLVR (GRPO-style binary-correctness-reward RL): it cannot sample a completion with zero initial probability under the reference policy arXiv:2507.14843 [MED]. Ni et al. sharpen this specifically for GRPO’s group-normalized advantage estimator: the update is provably a conservative reweighting operation, bounded per-step, not just directionally constrained arXiv:2510.15990 [MED]. So the theorem isn’t specific to DPO’s closed form — it’s a property of the whole KL-regularized family, RLVR included, which is exactly the family your GRPO stage lives in (Reinforcement — PPO · GRPO · RLVR).

What to actually run before you spend RL compute. Teacher-force the target completion y* through the reference checkpoint and read log π_ref(y*|x) and its rank among alternatives at the decision-token position — this is a single forward pass, no sampling, no training run. Near the numerical floor across ≥10 rephrasings of the prompt → you are in K-gap territory and no RL budget will fix it; go curate data. Finite and non-trivial → you’re in R/P territory, and every method in the KL-regularized family is mathematically available to you. This is the cheapest possible instance of the probe battery in Diagnosis — run it before anything else.

From the closed form to a training loss — DPO makes the theorem operational

It’s worth seeing the last algebraic step explicitly, because it’s the step that turns “reward is a reweighting of π_ref” from a fact about the optimum into a loss you can actually run. Take the inverted closed form from Step 3 and plug it into a Bradley-Terry preference model over a (chosen, rejected) pair sharing one prompt:

# Implicit reward per DPO's inversion of the Boltzmann posterior:
r(y | x) = beta * log(pi_theta(y | x) / pi_ref(y | x))        # + beta*log Z(x), which cancels below

# Bradley-Terry probability that y_chosen is preferred to y_rejected:
P(chosen > rejected) = sigmoid(r(chosen) - r(rejected))
                      = sigmoid(beta * [ log(pi_theta(chosen)/pi_ref(chosen))
                                        - log(pi_theta(rejected)/pi_ref(rejected)) ])

# DPO loss — no reward model, no RL loop, Z(x) cancelled algebraically:
loss = -log_sigmoid(beta * (lr_chosen - lr_rejected))
# where lr_y = log pi_theta(y|x) - log pi_ref(y|x)   for y in {chosen, rejected}

Notice what this loss can and cannot do to pi_theta(chosen). It can raise lr_chosen relative to lr_rejected — that’s a ratio against pi_ref, and the ratio is only informative where pi_ref(chosen) > 0 to begin with. If pi_ref(chosen) ≈ 0 (the K-gap condition), log(pi_theta(chosen)/pi_ref(chosen)) is either undefined or an enormous number driven entirely by the denominator underflowing — the loss has no stable signal to climb, and in practice this is exactly the regime where DPO training on a K-gap pair produces degenerate, off-distribution completions rather than the intended fix (a known, separate failure mode called likelihood displacement — pushing chosen up can drag unrelated high-probability completions down when the pair isn’t genuinely in-support; see Interventions per gap for the sanity gate this motivates). The loss is doing exactly what §3’s theorem predicts: real, useful gradient where pi_ref already has mass; nothing coherent to climb where it doesn’t.

The same frame applied to GRPO — where the “1-3% of tokens” claim comes from

GRPO drops the separate reward model and the pairwise comparison, but it’s solving the identical constrained problem with samples instead of an algebraic inversion:

# GRPO — sample N completions per prompt from the CURRENT policy (on-policy, by construction):
resps = [pi_theta.generate(prompt) for _ in range(N)]
rewards = [verifier(r) for r in resps]                    # ground-truth, e.g. flag correctness
advantage = [r - mean(rewards) for r in rewards]           # the group mean IS the baseline — no critic
# PPO-style clipped update pushes log-probability of high-advantage tokens up,
# low-advantage tokens down — but the update is bounded by the PPO-clip range
# AND by how much mass pi_ref (this step's rollout distribution) already assigned nearby.

Because every rollout in the batch is sampled from π_θ itself, GRPO can only ever reweight tokens it already sampled with non-trivial probability — which is exactly Ni et al.’s formal result that the group-normalized advantage update is a provably conservative reweighting operation, bounded per step arXiv:2510.15990 [MED]. It’s also the direct mechanism behind Akgül et al.’s empirical “1-3% of token positions, always top-5” finding arXiv:2605.06241 [LOW, promising] — the update can only touch positions where a sample actually landed with meaningful probability, and by construction that’s a small, high-entropy subset of all positions, drawn from a token’s existing short-list of likely continuations. §4 develops this into the mechanistic case for the running example.

4. Mechanistic evidence for the running example — the tool/flag failure is a P-gap

Two 2026 papers give near-direct mechanistic confirmation for exactly F2/F3’s shape: “agent reaches for one salient tool, uses default not expert args.”

4.1 Sparse, top-5-bounded correction

Akgül et al. find that only 1–3% of token positions change under RL, concentrated at high-entropy decision points — and, critically, the token RL promotes is always already in the base model’s top-5 logit alternatives arXiv:2605.06241 [LOW — very recent, low citation count, promising, not yet validated]. This gives you a literal, checkable number: if the correct tool/arg token sits in the base model’s top-5 at the decision point, you are in a checkable P-gap regime, and you should expect any fix (RL, DPO, or a lighter steering intervention) to touch a tiny fraction of the model’s behavior — this is not a “relearn the tool surface” problem, it’s a “nudge one decision” problem.

4.2 Attends-correctly-but-picks-wrong

Chen’s Looking Is Not Picking is more direct still, and the methodology is worth walking through because it’s what makes the claim mechanistic rather than just behavioral. On real BFCL tool-call failures — cases where the model, given a schema of candidate tools and a task, calls the wrong one — the paper decomposes the forward pass into two separable stages: (1) an attention/localization stage, measured as per-candidate attention mass over the tool-schema tokens, and (2) a decision/readout stage, the final projection that turns the model’s internal state into the emitted tool-call token. Per-candidate attention shows the model attends to the correct tool 80% of the time (vs. 21% chance under a uniform baseline) — it is under-attended in only ~10% of failures — yet the model still calls the wrong one on the majority of these cases arXiv:2606.16364 [LOW — very recent, single-author preprint, promising, not yet validated]. That decomposition is the whole argument: if the failure were localization (the model never “looked at” the right tool), that would look a lot like a K/R-flavored breadth problem — the correct option simply isn’t salient enough to reach. Instead the correct option is salient — attended to at 4× chance rate — and the failure is downstream of that, in the mapping from “attended-to” to “emitted.”

The intervention data confirms the decomposition rather than just illustrating it. Prompt-side fixes — reordering the tool schema so the gold tool appears earlier, or duplicating it to increase its salience — target the localization stage and recover ≤23% of failures, consistent with localization already being mostly fine (there’s not much headroom left to buy there). Interventions that instead target the decision-readout layer — the late projection/classification step, intervened on directly rather than through the prompt — recover 59–91% of failures.

That 23%-vs-59–91% spread is about as direct a proof as exists in the current literature that “picks the salient-not-expert tool” is a readout/decision failure (P), not a visibility/knowledge failure (K). It proves it by showing where the fix actually lands.

Put together: F2/F3 is not “the model doesn’t know amass exists” — it’s attending to the right region of its own knowledge and still emitting the wrong token at the readout step. That is the textbook P-gap signature, and it means the fix lever is a decision-level nudge (on-policy DPO on tool-choice pairs, or a light GRPO pass rewarding the expert choice), not a data-injection campaign to teach the model what amass is.

4.3 Cross-model universality as evidence, not just an observation

The running example states this failure is universal across frontier models — every family in a typical benchmark roster reaches for the same salient tool and the same default flags. That universality is itself diagnostic, and it’s worth stating the logic explicitly rather than treating it as a curiosity.

If this were a genuine K-gap, you’d expect it to vary by pretraining corpus and architecture — different labs curate different data mixes, so an idiosyncratic capacity/storage limitation should show up idiosyncratically. Instead the failure is shared across independently-trained frontier families. The more parsimonious explanation, consistent with Kandpal et al.’s frequency-correlation finding that extraction accuracy tracks how often entities co-occur in pretraining text arXiv:2211.08411 [HIGH], is a shared prior baked into the overlapping web-scale pretraining distribution every frontier lab trains on: subfinder-style one-shot recon tools and default-flag invocations are simply far more represented in blog posts, Stack-Overflow-style answers, and tool READMEs than amass’s fuller feature set or an expert scan profile, and every lab’s corpus draws from broadly the same internet. This is a reasoned extension of the theorem plus the frequency-correlation evidence, not a separate paper’s direct finding — flag it as such — but it is a genuinely useful, falsifiable heuristic: a failure that’s universal across independently-trained frontier families is evidence for a shared distributional prior (R/P), not evidence for a per-model capacity ceiling (K), and it’s a cheap first read to run before committing to the full probe battery.

What this changes about your intervention. If the diagnosis holds, the fix is not “teach the model what amass is” (it already knows — every frontier family does) — it’s “change which of the two already-known options the readout step selects, at the specific decision point where they compete.” That’s a data-curation problem for on-policy preference pairs (chosen = amass invocation, rejected = subfinder-only, sampled from the model’s own rollouts at that decision point), not a knowledge-injection problem. The concrete recipe for building that pair set lives in Interventions per gap; this chapter’s job is only to establish why that’s the right category of fix.

5. The generation–verification gap — an orthogonal axis

K/R/P all answer one question: can the model produce the right answer? There’s a second, genuinely orthogonal question that matters independently for any pipeline that leans on the model’s own outputs to bootstrap further training (rejection sampling, STaR-style loops, RLAIF): can the model tell right from wrong once something has been produced?

Song et al. formalize this as the generation-verification gap: pass@N coverage minus self-verification accuracy, on the same problem set arXiv:2412.02674 [HIGH, ICLR 2025 Oral]. This quantity scales monotonically with pretraining compute and bounds how much any self-improvement loop can gain — a model can have excellent K/R/P properties (the right answer is reliably produced somewhere in its sample distribution) and still have a wide generation-verification gap (it can’t reliably pick the right one out of its own samples), and the two properties can diverge in either direction.

This matters for your running example at a specific, practical seam: the flag verifier is external, ground-truth, and only fires at submission time. Mid-episode — at turn 40 of 100 — the agent has to judge without the verifier whether its current path is worth continuing. That’s a self-verification judgment, not a generation judgment, and it’s governed by this orthogonal axis, not by K/R/P. A model can have zero K-gap and zero P-gap on “which tool to use next” and still make a bad mid-episode continue/abandon call because its self-verification is weak relative to its generation.

Contested, and worth stating as contested rather than settled. The intuition that “verification is easier than generation” is rooted in scalable-oversight/debate theory — the founding argument that a weaker verifier can still adjudicate a stronger generator’s claims if the claims are checkable arXiv:1805.00899 [HIGH, seminal]. It is not universal. West et al.’s Generative AI Paradox documents cases where models are worse verifiers than generators on the same items, driven by an acceptance bias toward the model’s own (or plausible-looking) output rather than genuine discrimination arXiv:2311.00059 [MED]. Treat the direction of the generation-verification gap as task-dependent, not a law:

Don’t assume a model that solves your challenges reliably is equally reliable at telling you, mid-trajectory, whether it’s on a dead path — that’s a different channel than the one measured by solve rate, and per the table above its reliability isn’t guaranteed by the solve rate being high. The self-verification-cliff mechanism this connects to on the agentic-execution side is developed fully in Diagnosing the gap §5.1; this section’s job is only to place it correctly as orthogonal to K/R/P, not to re-derive the mid-episode diagnostic.

6. The contested debate — does RL ever expand the boundary, or only reweight?

Everything in §3 says KL-regularized optimization can only reweight existing support. The empirical debate over how narrow that reweighting really is in practice — whether it’s ever indistinguishable from genuine capability expansion — is the single most contested thread in this literature, and it matters directly for whether F1 (the recon-methodology gap, R-flavored) is cheaply fixable by rejection-sampling SFT + light RL, or needs a heavier on-policy exploration recipe.

• Yue et al. — the founding result arXiv:2504.13837 [HIGH, NeurIPS 2025 oral]. At large sampling budgets (pass@256+), un-RL’d base models match or beat RLVR-trained checkpoints on static math/code — RLVR raises pass@1 by resampling paths already in the base model’s support, it does not expand the reasoning boundary. This is the empirical anchor for the theorem in §3.
• ProRL — the direct counter, for one specific recipe arXiv:2505.24864 [MED, NVIDIA]. Prolonged RL with explicit KL-divergence control, periodic reference-policy resets, and diverse tasks uncovers reasoning strategies inaccessible to the base model under any sampling budget tried. This doesn’t contradict §3’s theorem — it’s evidence that “reference policy” is not static across a long, reset-punctuated training run, so the support boundary itself can shift over the course of prolonged training even though no single KL-regularized step can violate it.
• CoT-Pass@K — a metric-level rebuttal arXiv:2506.14245 (cited in the source survey narrative; not independently re-confirmed against arxiv.org in this ledger pass — treat as a live but unconfirmed thread). Argues naive pass@k credits a correct final answer even from a broken reasoning chain (a lucky guess), and once you require the reasoning trace itself to be correct, the crossover disappears — RLVR shows monotonic gains at every k. If this holds, it directly falsifies the load-bearing assumption behind Yue et al.’s headline reading.
• Two-stage reconciliation arXiv:2510.04028 (same unconfirmed-in-ledger caveat). Proposes the two camps are sampling two different phases of one dynamic: an early “exploitation” phase looks like pure reshuffling (the Yue-camp signature), while a later “exploration” phase — reached only if training survives entropy collapse long enough, which needs ProRL-style KL-control/reset machinery — can genuinely promote rarely-sampled optimal tokens into the accessible distribution.
• Agentic Pass@(k,T) — the caveat most relevant to your own harness arXiv:2604.14877 [MED]. The static single-turn “RL only elicits” result may not transfer to tool-use agents: T rounds of environment interaction can reveal compositional strategies that flat resampling (more k, same T) cannot recover. On their Category C (compositional, sequentially-gated retrieval — structurally identical to “enumeration must land at turn 5 before the exploit is even visible at turn 40”), the RL pass-curve pulls above and widens against the base curve as k grows — the opposite of the static-reasoning crossover — while matched-data SFT on the same subset actually regresses it (net −4 vs. RL’s net +4). This isolates self-directed exploration during on-policy rollout, not data exposure, as the causal ingredient for expansion on compositional tasks.
• Recall specifically arXiv:2605.07153 [MED]. In a deduplicated, zero-shot, closed-book factual-recall setting, RL on a binary correctness reward does yield ~27% relative recall gains — by moving correct tokens from an astronomically low-probability tail into a reliable greedy-decode slot. This complicates “no pass@k gain ⇒ K-gap” specifically for pure recall (as distinct from multi-step reasoning): a small k can misclassify a rare-but-present fact as a true K-gap when it’s actually a very-low-rank R-gap that a longer RL run would fix.

A few adjacent threads round out the picture without being individually load-bearing here: a compositional-generalization result claims RL can teach f∘g composition never itself sampled from the base model, but a rebuttal shows the same signature can arise from mere length generalization rather than true composition, so treat that specific pairing as unresolved; a boundary-aware curriculum-RL line is a further active attempt to engineer past the static-reasoning crossover. Neither is independently confirmed in ledger-A.md for this pass, and neither changes the net read below — they’re additional evidence that the debate is active, not additional weight on either side.

The engineer’s net read. “RL only reweights, never teaches” is real, but narrower than usually quoted. It’s most solid for static, single-turn math/code reasoning under short/vanilla RLVR — exactly the regime Yue et al. tested. It gets progressively weaker for (a) pure factual recall, (b) multi-turn agentic/tool-use settings, and (c) prolonged RL runs engineered with KL control, reference resets, and curriculum. Don’t conclude K-gap from a k=16 pass@k null result on an agentic task — push k as high as budget allows, and if the task is agentic, run Pass@(k,T), not flat Pass@(k).

Why this matters for F1 specifically, and why it matters less for F2/F3. The mechanistic P-gap evidence in §4 (top-5-bounded token correction, 80%-attended-but-mis-picked) is largely agnostic to this debate — a mis-ranked-but-already-attended token doesn’t require capability expansion to fix, only reweighting, so F2/F3 are cheap, well-targeted fixes regardless of which side of the elicit-vs-expand debate turns out to be right. F1 (the recon-methodology gap) is exactly where the debate’s resolution is operationally load-bearing: if the agent’s recon planning is a static, single-turn ranking problem, treat it as the Yue-camp regime (rejection-sampling SFT should work fine). If it’s genuinely sequentially-gated — the recon sequence has to unfold correctly across several tool calls before the right next move is even visible — you’re in Zhai et al.’s Category-C regime, and matched-data SFT alone is predicted to regress it; you need on-policy RL with real exploration, not more demonstrations. This is exactly the segmentation test The decision and Diagnosing the gap §2.4 already build a runbook around — this chapter supplies the theoretical reason that segmentation is the right first move, not an arbitrary methodological preference.

7. Synthesis — the running example, fully classified

Pulling §2 through §6 together against the three named failures, one table:

The universality argument from §4.3 already makes the K-row unlikely for F1–F3 specifically — a genuinely per-model capacity ceiling wouldn’t reproduce identically across independently pretrained frontier families. That’s a strong prior, not a substitute for running the actual probes; the theorem in §3 is exactly what makes “run the probe, trust the number” a legitimate move instead of an argument from authority.

The one-sentence version of this whole chapter. K asks “does log π_ref(y*|x) exist at all”; R asks “is it ranked below a shallow default”; P asks “does a different elicitation channel expose a preference the default channel hides” — and the last two are one axis, not two, so stop trying to draw a hard line between them and instead ask which sub-signature (plateau vs. divergence) your specific failure shows, because that determines which cheap probe catches it first.

Cross-links

• The decision — the routing tree this chapter’s K/R/P definitions and theorem sit underneath; read that first for the practical branch, this chapter for why the branch is legitimate.
• Diagnosing the gap — a scientific framework — the complementary knowledge/execution/exploration framing and the full pass@k → Cover@τ → Pass@(k,T) instrument battery; §2.4 there is the direct operationalization of §6’s contested debate, and §5.1 develops the self-verification-cliff mechanism this chapter’s §5 only places on the map.
• Diagnosis — the runnable probe battery (forced-logprob check, pass@k sweep, in-context oracle injection, token-rank check, spurious-reward control) that turns this chapter’s definitions into an executable decision procedure.
• Knowledge-gap data curation — the data-row recipe for a confirmed K-gap, and why jumping straight to trajectory-level SFT on an unconfirmed one causally amplifies it rather than fixing it.
• Interventions per gap — the matched training-row recipes once a gap is confirmed: paraphrase-rich SFT for K, on-policy DPO/GRPO pairs for R/P.
• Trajectory-SFT amplification — the fuller causal argument for why naive trajectory-level SFT on an unfixed K-gap makes hallucination worse, not better.
• Preference — RLHF · DPO · KTO — the mechanism behind the R/P-gap fix lever this chapter derives the legitimacy of but doesn’t itself teach.
• The one axis that predicts everything — the on/off-policy genealogy this chapter’s K-vs-R/P line is a refinement of: K needs off-policy injection because there’s nothing on-policy to reinforce; R/P are on-policy-fixable by construction.

Bibliography

All ids below are drawn from artifacts/three-gap-survey/section-A.md and ledger-A.md. Confidence follows the ledger where the id was independently confirmed against arxiv.org; two ids used in §6’s contested debate were not in the ledger’s confirmed table for this pass and are flagged accordingly — cited because the chapter’s brief specifically required presenting that exact debate, not presented as independently re-verified.

Confidence calibration for this chapter: the theorem (§3) rests entirely on HIGH-confidence, well-established citations (Levine, Ziegler, DPO) plus two MED-confidence formal extensions to RLVR/GRPO specifically — treat the general KL-regularization argument as settled, the RLVR-specific and GRPO-specific formalizations as solid-but-newer. The mechanistic P-gap evidence for the running example (§4.1–4.2) is explicitly LOW / promising, not yet validated — both papers are very recent, low-citation preprints; they are the best mechanistic evidence currently available for “tool-selection failure is a readout problem,” not a settled consensus. §6’s contested debate is presented as contested on purpose — two of its six citations were not independently re-confirmed in this ledger pass and are flagged inline; do not treat the “net read” as more settled than the six bullet points underneath it.