Diagnosing which gap — the runnable probe battery

The question this chapter answers: given one failing behavior — the agent never runs amass, always runs subfinder with default flags, and never front-loads a recon methodology before it starts poking things — what is the cheapest sequence of runnable probes that sorts this into a Knowledge (K), Prior/Policy (R/P), or Exploration-collapse gap, with numbers, not vibes?

BLUF. Run nine probes, cheapest-first, each costing at most what the previous one bought you: (0) a free grep of your existing corpus, (1) one forced-logprob forward pass, (2) a pass@k sampling sweep, (3) in-context oracle injection — the single sharpest K-vs-P splitter in the whole battery — (4) a generation-vs-recognition split, (5) a token-level top-5 rank check, (6) a spurious-reward control gate before you trust any RLVR result, (7) Pass@(k,T) segmented by compositional structure, (8) a minimal-parameter/few-shot-SFT elicitation probe, and (9) the escalating elicitation ladder that synthesizes 0–8 into one verdict. No individual probe is trusted alone — every family below ships with a documented confound, and the real confidence signal is agreement across three or more independent probe families, not a single clean plot. Close with a purpose-built bottleneck CTF that turns the whole battery into one number you can read off a solve-rate band.

Every arXiv id below is CONFIRMED in artifacts/three-gap-survey/ledger-A.md or ledger-B.md (project research pass, verified live against arxiv.org/abs/<id>). Confidence tags: [HIGH] peer-reviewed/heavily reproduced, [MED] coherent preprint not yet contested, [LOW] single small-N or very-recent preprint — “promising, not yet validated.” Ids flagged UNCERTAIN or absent from either ledger are not used here, per the standing project rule against inventing or citing unverified ids.

0. This chapter vs. Diagnosing the gap — a scientific framework: same territory, two vocabularies

The framework chapter sorts a failure into Knowledge / Execution / Exploration, grounded in the linguistics competence/performance split (Firestone PMC7604508 [HIGH]; Mahowald et al., arXiv:2301.06627 [HIGH]) and built around one core instrument (pass@k → Cover@τ → Pass@(k,T)). taxonomy.md sorts the same failure into the finer-grained K / R / P — Knowledge, Prior, Policy — and its own §0 is the canonical statement of how the two vocabularies map onto each other: “their ‘knowledge gap’ is this chapter’s K; their ‘execution gap’ and ‘ranking gap’ are both instances of this chapter’s single R/P axis… and their ‘exploration gap’ is what you get when an R-flavored failure turns out to be sequentially-gated rather than single-shot.” This chapter doesn’t re-derive that mapping — it takes it as given and adds the missing piece: a runnable test (Probe 7, §9) for the one part of the mapping that isn’t a static lookup — whether a given R/P-flavored failure is also sequentially-gated, and therefore an Exploration-collapse case rather than a plain R/P-gap.

The correspondence — restated compactly here, canonical derivation in taxonomy.md §0:

Practically: this chapter is the how — the runnable protocol with concrete numbers, thresholds, and pseudocode, operationalizing taxonomy.md §0’s mapping rather than re-deriving it. Framework.md is the why — the theory of why pass@k, Cover@τ, and Pass@(k,T) are valid instruments at all, including the contested edges (§2.2’s crossover vs. its CoT-Pass@K rebuttal). Read taxonomy.md §0 for the mapping and framework.md for the instrument derivations; this chapter assumes both and tells you what to actually type into a terminal.

1. The running example: three decision points, one battery

Three co-occurring symptoms, universal across the project’s benchmark roster (5 model families — xai, gemini, deepseek, glm, qwen, none privileged, per lessons/evals/pd-bench-benchmark-family-roster.md):

• F1 — no opening methodology. Turn 1 dives straight into a scan/exploit attempt instead of a structured recon-first plan.
• F2 — subfinder, not amass. Subdomain enumeration reaches for the tool with the narrower default passive-source list.
• F3 — default, not expert, flags. Whatever tool gets called, it’s invoked with its bare defaults, not the flag combination (-passive -config <file> for amass; --min-rate 5000 -p- -sV -sC for nmap) that a human operator would reach for.

The reason to run the same battery on all three: each is a decision point — a place in the trajectory where the model chooses among alternatives — just at different granularity. F2 and F3 are single-token/single-argument decisions (clean, cheap to probe up to k=256). F1 is a structural decision about the shape of the opening move, closer to framework.md §6’s planning/state-management axis — grounded there in the METR long-horizon study and DAgger’s compounding-error theory, not re-cited here — probeable with the same instruments, but its verdict is less often clean, and a confirmed non-K reading on F1 should route to framework.md §6.1’s per-turn fault labeling for the deeper cut, not stop here.

Run this first, for free: before any of the probes below, confirm the symptom is real and not a sampling artifact — pull 10-20 raw transcripts and read them. A “gap” that turns out to be one bad temperature setting or a truncated context window is not a research question.

2. Probe 0 — free triage before you spend anything

Two zero-cost checks that route you into the ladder; neither is a verdict.

Corpus grep. Your run-trace corpus already exists — events.jsonl per run, preamble carrying meta/tool_schemas/system_prompt/user_message, per this project’s harness observability contract (lessons/security-agent/harness-observability-contract-2026-06.md, always-loaded handbook rule #17). Grep it before generating a single new token:

grep -l '"amass"' runs/*/events.jsonl | wc -l     # e.g. 2 / 50 existing rollouts
grep -l '"subfinder"' runs/*/events.jsonl | wc -l # e.g. 47 / 50
grep -c '"-passive"' runs/*/events.jsonl          # expert-flag string, near 0 across the corpus

Readout: rare-or-absent at existing N tells you where to look, nothing more. Family 1’s own caveat applies immediately: absence in 50 already-collected rollouts at whatever temperature they were sampled at is not evidence of a K-gap — under-sampling collapses pass@k toward pass@1 and manufactures false K-gap readings (section-B, Family 1). Route to Probe 1.

Verbalized confidence, free. Ask the policy, before it acts: “List every subdomain-enumeration approach you could use here, and rate 0-100 your confidence you’re not missing a stronger passive-source option.” Kadavath et al. arXiv:2207.05221 [HIGH] established that models are reasonably calibrated on this kind of pre-generation self-report even when open-ended generation is wrong — a high self-reported completeness score that turns out to be false (the model never mentions amass) is itself a P-gap-flavored signature worth carrying forward, not a verdict on its own.

3. Probe 1 — forced-logprob + recognition (one forward pass)

What you run. Teacher-force the exact expert string through the policy at the decision token — no sampling, no gradient:

target = "amass enum -passive -d target.com -config ~/.config/amass/config.ini"
logps  = teacher_force(policy, prefix=trajectory_up_to_tool_call, target=target)
nll_mean   = -mean(logps)                       # whole-target perplexity
min_k_stat = -mean(sorted(logps)[: int(0.15 * len(logps))])   # worst 15% of tokens — Shi et al.

Min-K% Prob — the mean of the lowest-probability slice of tokens rather than the whole-sequence mean — catches a single catastrophically bad token (e.g. the -passive flag) hiding inside an otherwise fluent, plausible-looking command. Shi et al., arXiv:2310.16789 [HIGH], 700+ citations, the mainstream instrument for exactly this.

Two mandatory adjustments, or you’ll manufacture a false K-gap:

• Sweep 3-5 paraphrasings of the system/task prompt and take the minimum perplexity across them (equivalently, max logprob) — a single unlucky framing of the task is only a lower bound on what the model “knows,” per Jiang et al.’s prompt-sensitivity finding, arXiv:1911.12543 [HIGH].
• Normalize for surface-form competition if more than one valid rendering of the target exists (-passive vs --passive-only; amass enum vs amass intel) — raw target-string probability underestimates knowledge when synonymous forms split probability mass. Holtzman et al., arXiv:2104.08315 [HIGH].

In parallel, run the forced-choice recognition twin: present a 4-option MCQ (“which command maximizes passive subdomain source coverage: A) subfinder -d target.com B) amass enum -passive -config ... C) … D) …”), score selection accuracy across the same paraphrasings.

Readout:

Don’t commit to trajectory-level SFT on the strength of a single-prompt logprob read. Gekhman et al., arXiv:2405.05904 [HIGH, EMNLP 2024]: SFT rows the pre-training policy classifies as genuinely novel (via k-sampling) are fit dramatically slower, and once fit, linearly increase hallucination on unrelated, previously-known facts. Ghosal et al., arXiv:2406.14785 [MED]: SFT on lesser-known facts teaches the model to ignore the conditioning subject/context and emit a generic default — i.e. naive SFT on an unconfirmed K-gap can strengthen the “reach for the salient tool” default rather than fix it. This is the reason Probe 1 is a screen, not a commit signal — proceed to the oracle-injection step (§5) before writing a single SFT row.

4. Probe 2 — the pass@k sweep, and what the SHAPE tells you

What you run. The unbiased combinatorial estimator — never naive c/k — first defined for exactly this purpose by Chen et al.’s Codex/HumanEval paper, arXiv:2107.03374 [HIGH]:

def pass_at_k(n, c, k):
    """n = samples generated, c = number correct, k = budget. Numerically stable form."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

Sample n ≥ 4·k_max completions per decision point at temp 0.7-1.0 (never greedy — under-sampling collapses pass@k toward pass@1 and manufactures a false ceiling), across ≥10 rephrasings of the task prompt, same format for every checkpoint you compare. Score c = number of rollouts where the target action (the amass invocation with the expert flag set, specifically, not “any working subdomain enumeration”) actually fires. Sweep k = {1, 4, 16, 64, 256} and plot pass@k vs. log(k).

Cost reality check: n ≥ 4·k_max at k_max=256 means n≥1024 — expensive for a full T≈100-turn agentic rollout. This is affordable for F2/F3 because they’re single-decision-point resamples (re-roll from the same trajectory prefix up to the tool call, not the whole episode) — cheap, parallel, no environment cost. Reserve full-episode resampling at large k for after you’ve narrowed the search with the cheap single-decision version; full-trajectory Pass@(k,T) sweeps (§9) are reserved for smaller k, larger T instead.

What the SHAPE means:

• Flat near-zero across all k up to 256, across all 10 rephrasings → K-gap candidate, proceed to oracle injection (§5) before finalizing — per Family 1’s convergent caution, a null pass@k result on a pure recall-flavored decision like “which tool has broader source coverage” can still be a false K-gap at small k; Yang et al. show RL on binary correctness reward yields real recall gains in a closely analogous zero-shot factual-recall setting by moving a correct token out of an astronomically low-probability tail — cited via section-A’s ladder as [arXiv:2605.07153] but note this specific id is present only in ledger-A’s confirmation and is not independently re-verified in ledger-B; treat as [LOW], corroborating not load-bearing.
• Rises sharply and plateaus well above pass@1’s floor by k≈16-64 (e.g. pass@1 = 3%, pass@64 = 55%) → R/P-gap: the correct action is in-support, just not the default top choice. Proceed to §5.
• If you have both a base and an SFT/RL-trained checkpoint: plot both curves together — this is framework.md §2.2’s crossover test, not re-derived here. Trained wins at k=1, base catches up/exceeds by k=64-256 → elicitation-only (Yue et al., arXiv:2504.13837 [HIGH]); contested by requiring reasoning-path correctness, not just final-answer match (Wen et al., arXiv:2506.14245 [MED]) and by prolonged-RL-with-KL-control counter-evidence (Liu et al., ProRL, arXiv:2505.24864 [HIGH]). State the crossover as contested when you report it — see framework.md §2.2 and Contested edges §1.
• Discrete/low-cardinality caution. A flag string is exactly the kind of discrete, low-cardinality answer space where pass@k at large k is dominated by guessing rather than genuine reliability (Cover@τ, framework.md §2.3, not re-derived here — Dragoi et al., arXiv:2510.08325 [MED]). Report Cover@τ (τ≈0.3) alongside any pass@64+ number you’d otherwise be tempted to read as a clean win.

Mandatory control for any few-shot variant of this probe: if you’re also sweeping in-context demonstration count (dose-response per Many-Shot ICL, Agarwal et al., arXiv:2404.11018 [HIGH]), always run a shuffled-label control alongside the correct-demo condition — Min et al., arXiv:2202.12837 [HIGH]: randomizing demonstration labels barely hurts accuracy across 12 models, meaning most naive few-shot lift is format/label-space calibration, not a learned mapping. Only the delta between correct-demo and shuffled-demo few-shot is real evidence of an R/P-gap; if correct-demo ≈ shuffled-demo, the zero-shot gap was a schema-calibration artifact, fixable with a system-prompt nudge, not training data. Zhao et al., arXiv:2405.19874 [MED-HIGH]: ICL underperforms fine-tuning as an elicitation technique and the gap widens at higher capability — a negative few-shot-only probe on a frontier-scale model is not, by itself, license to conclude K-gap.

5. Probe 3 — in-context oracle injection: the K-vs-P splitter

This is the single sharpest instrument in the whole battery, and the direct answer to “does putting the answer in context and re-running settle it.”

What you run. Take the exact prompt that produces F2/F3, and append the missing knowledge verbatim into the system prompt or tool description:

Addendum to system prompt:
"For subdomain enumeration, prefer `amass enum -passive -d {target} -config amass.ini`
over `subfinder` — amass's default passive source set includes {source X}, which
subfinder's default list omits. Chain to `httpx` for liveness confirmation afterward."

Re-run N=32-50 rollouts, cold start, same challenge, same temperature as your Probe 2 baseline.

Readout — the split itself:

• Fixes it near-zero-shot (correct-tool-use rate jumps from ~2% to 70%+) → CONFIRMED K-gap. The policy machinery to use the information was already there; only the declarative fact was missing. Route to data curation (./knowledge-curation.md), not raw trajectory-level SFT — see Probe 1’s caution above on why that specific move backfires on an unconfirmed K-gap.
• Doesn’t fix it, even with the answer handed over verbatim → CONFIRMED P-gap — a readout/decision-layer failure, not a knowledge failure. This is exactly the mechanistic signature Chen documents directly on tool-call failures: per-candidate attention finds the correct tool 80% of the time (vs. 21% chance) yet the model still calls the wrong one; prompt-side fixes (reordering/duplicating the gold tool in the schema — which is structurally what oracle injection is) recover ≤23% of failures, while decision-readout-layer interventions recover 59-91% (Chen, arXiv:2606.16364 [LOW, single-author preprint — promising, not yet validated]; drawn from section-A’s ladder). Route to Probe 5 / RL-DPO fixes, not more demonstrations.

Teacher Flip Rate — three privileged-context levels, not one. Build a graduated version of the oracle instead of a binary present/absent:

• L1 — abstract hint. Self-generated by the same model, conditioned on the gold trajectory, asked to produce “a hint containing the core knowledge needed” — no distribution shift from an external author.
• L2 — concrete-but-partial. amass --help excerpt, or one worked example — tests the K-gap question specifically.
• L3 — full oracle. The exact gold command in context (what you just ran above).

Sample G=8-16 rollouts unprivileged, keep only the wrong ones, re-sample the same wrong prompts at each level (inference only, no weight update). TFR(level) = fraction that flip to correct. Mandatory control: also compute TFR against a semantically-empty, length-matched context (“think step by step, be thorough”) — if TFR is elevated there too, your verifier has a leak and you should distrust the whole probe. This is the OPSA framework’s Teacher Flip Rate, Fu et al., arXiv:2605.15239 [MED, recent].

Readout table:

Two caveats before you trust a positive TFR: (1) revealing the full gold answer as a hint can actually underperform an abstract hint — Chen, Peng et al.’s NuRL, arXiv:2509.25666 [MED, ICLR 2026 accepted poster], found full-info hints cause shortcut-copying rather than genuine reactivation, so don’t skip straight to L3 and assume it’s the most informative signal. (2) a hint that creates learning signal under hinted rollouts does not guarantee the deployed hint-free policy improves — Xia et al.’s HiLL, arXiv:2604.00698 [LOW, promising], is the mandatory downstream control: always re-validate any fix completely hint-free before declaring victory.

Run this on F1 too. For the “no opening methodology” symptom, the oracle is a structured checklist: “Standard operating order: (1) passive recon, (2) active recon, (3) enumerate, (4) exploit, (5) verify.” If injecting the checklist alone fixes turn-1 compliance, that’s a K-gap (the model didn’t know to front-load recon as a declarative procedure) — curate data. If it still skips steps even with the checklist explicitly in context, that smells like the attention/decision-readout class of failure, or a genuine planning/state-management limitation — route to framework.md §6 and its per-turn fault taxonomy (§6.1) for the deeper cut; this chapter’s single-decision-point probes aren’t built to fully resolve a structural planning failure.

6. Probe 4 — generation-vs-recognition (MCQ) split

Why this is a separate probe from Probe 1’s recognition check: Probe 1 asks whether the correct answer is ranked competitively. This probe asks whether the model’s own judgment channel (pick the best of several options, one of which is its own default) disagrees with its generation channel (what it actually does by default). West et al.’s Generative AI Paradox names the theoretical basis: generation and discrimination/understanding are dissociable capacities, and the dissociation can run in either direction — arXiv:2311.00059 [HIGH].

What you run. Build the candidate set from the model’s own rollouts, not hand-written strawmen — this matters, because a distractor set the model would never actually produce tells you nothing about its failure mode:

state s = trajectory prefix up to the enumeration decision
C = { a_expert:  "amass enum -passive -d {t} -config amass.ini"  (verified-correct),
      a_shallow: <the model's OWN greedy pass@1 rollout at s>,
      a_dist_1, a_dist_2: <the model's OWN other sampled rollouts at s> }

Present s + C in randomized position order (position bias is real — Zheng et al., arXiv:2306.05685 [HIGH], 432 citations), ask the model to (a) select the best option and (b) justify by citing the differentiating feature. Run a discriminative control variant too (single-token choice / logprob-over-options, no justification) — if it disagrees with the generative select-then-justify answer, the justification may be post-hoc rationalization, not the real reason.

Readout:

This is the Elicitation Game’s exact signature transplanted onto a tool-choice: Hofstätter et al., arXiv:2502.02180 [MED, ICML 2025 poster], found prompting/steering suffice to elicit hidden capability on MCQA-shaped probes but fail on generative/code-shaped tasks — a null result from Probe 4’s recognition side is much more trustworthy than a null result from raw open-ended prompting alone (§7’s ladder makes this ordering explicit).

7. Probe 5 — token-level top-5 logprob-rank check

What you run. At the exact generation step where the wrong tool name gets chosen — the token position where subfinder beats amass, or the default flag string beats the expert one — pull the base model’s top-5 logprobs at that position:

logits_at_decision = policy.forward(prefix_up_to_decision_token)
top5 = torch.topk(logits_at_decision, 5)
rank_of_amass_token = (logits_at_decision.argsort(descending=True) == amass_token_id).nonzero()

Readout:

Cross-check against Probe 3’s mechanistic finding: if attention already localizes the correct tool 80% of the time (per Chen, arXiv:2606.16364 [LOW]) but the readout still picks wrong, and the token IS in the top-5 here, you have two independent signals converging on the same P-gap verdict — exactly the “agreement across 3+ families” bar this chapter’s BLUF asks for.

8. Probe 6 — the spurious-reward control, before you trust any RLVR result

Why this probe exists: a GRPO run that appears to fix F2/F3 is not, by itself, evidence the reward was doing discriminative work. It might just be amplifying whatever the sampling temperature already surfaces more of.

What you run. Train the identical GRPO setup twice, a few hundred steps each, on the same prompts:

• (a) Real reward: +1 if the trajectory used amass with the expert flag set AND the flag verified correct, 0 otherwise.
• (b) Shuffled/random reward: the same prompts, but the reward for each rollout is assigned by a coin flip independent of what the rollout actually did.

Readout:

• (b) also moves the target metric substantially (amass-usage-rate climbs under the random-reward run too) → you were mostly amplifying a pre-existing high-prior behavior, not doing genuine discriminative reward-shaping. This is a real, confirmed finding, not a hypothetical — Shao, Li, Xin, Geng et al.’s Spurious Rewards paper, arXiv:2506.10947 [MED — the finding is confirmed but was demonstrated on a specific model family; verify it replicates on your own roster before generalizing across all five families]. Invest in exploration/coverage instrumentation, not more reward engineering.
• (a) works, (b) doesn’t → the reward is doing real discriminative work on an in-support-but-mis-ranked behavior — clean P-gap, RL is well-targeted.
• Neither moves the metric → K-gap; stop spending RL compute, go curate data.

Run this as a gate, not a postmortem. Before committing a full GRPO budget to “fix” F2/F3, run both arms for ~300 steps first — cheap relative to a full training run, and it directly determines whether the rest of your RL compute is well spent.

9. Probe 7 — Pass@(k,T) for the agentic/compositional case

This is where the Exploration-collapse dynamic actually gets diagnosed, and it’s the one probe in this battery that requires framework.md’s full derivation (§2.4) — not re-derived here, only operationalized.

Segment the portfolio first. F2/F3 in isolation (does turn-1 use amass with expert flags — a single decision, not gated by anything upstream) is the simple/independent-retrieval segment. The compositional case is where the flag is reachable only because amass’s passive enumeration surfaces a non-obvious subdomain that the agent must then correctly interpret and pivot to — a chained, sequentially-gated decision, structurally identical to Zhai et al.’s “Category C,” arXiv:2604.14877 [MED].

What you run. Same estimator as pass@k, with a second axis T = max tool-calls/turns:

PASS@(k,T)(q, π) = 1 - C(n - c_T, k) / C(n, k)     # c_T = correct at interaction depth T specifically

Sweep T ∈ {1, 2, 4, 8, 16, 32} at fixed k=1 first — does giving the agent more turns/self-correction rounds alone, no training change, recover the compositional path? Then sweep k ∈ {1, 4, 16, 64} at your deployment T. Run this separately on the simple and compositional segments, before and after your rejection-sampling-SFT checkpoint.

The falsifiable prediction, per section-B’s synthesis of Zhai et al.: on the simple segment, expect the static crossover to hold (SFT works, further RL may plateau — a clean R/P-gap, cheaply closed, per Yue et al. arXiv:2504.13837 [HIGH]). On the compositional segment, expect matched-data SFT on the exact winning trajectories to regress the capability boundary while on-policy RL widens it — a net −4 vs +4 pattern in Zhai et al.’s own numbers, isolating self-directed exploration during RL, not data exposure, as the causal factor.

Readout:

One more control before you trust a “regression” reading as genuine exploration-collapse: the pass@k-minus-pass@1 gap mechanically shrinks as a policy’s sampling distribution concentrates during training — independent of any real change in underlying capability. Yu, arXiv:2511.16231 [LOW, single preprint]. Corroborate the regression reading with an entropy/diversity check on the same checkpoints (framework.md §7’s RL-PLUS “capability boundary collapse,” not re-derived here) before concluding the SFT itself caused the regression, rather than the training run simply narrowing its own sampling distribution for unrelated reasons.

10. Probe 8 — the minimal-parameter / few-shot-SFT elicitation probe

What this probe answers that the others don’t: a quantitative recovery-cost number, not just a direction.

What you run — two variants, pick based on weight access:

• Minimal-parameter unfreeze. Freeze the model; unfreeze a random parameter subset starting at ~100 parameters, scale up to ~10,000; fine-tune briefly on the task’s own verifiable labels (no curated demonstrations needed — the labels come from your own flag verifier).
• Few-shot LoRA. Fine-tune a rank-1 to rank-4 LoRA on 5-20 demonstration rows — self-distilled from the model’s own best-of-N successful rollouts if no external teacher exists (this is a diagnostic use of light SFT, not the production data-curation pipeline — that pipeline is kinds-of-sft.md’s territory, deferred here).

Readout:

This mirrors an unpublished-but-confirmed-on-OpenReview result: 10-100 randomly chosen trained parameters recovering roughly half the full-fine-tune gap (NeurIPS 2025 poster, OpenReview id Dkgx2pS4Ww — no arXiv mirror exists for this one; cite cautiously per the ledger’s own note). More load-bearing: Greenblatt et al.’s password-locked calibration methodology, arXiv:2405.19550 [HIGH] — a handful of high-quality SFT demonstrations is often sufficient to fully elicit a locked capability.

Build ground truth before trusting this probe on your real ambiguous case. Take a capability you already know the base model has (verified via Probe 2’s high-k pass@k). SFT a clone: with a password token, train on 20-50 oracle-quality trajectories (full expert behavior); without the password, train it to imitate the observed shallow default. Sweep re-elicitation budget {4, 16, 64, 256, 1024 rows} on this clone, and separately run a noise-injection pre-check — Tice et al., arXiv:2412.01784 [MED, NeurIPS 2025 poster]: sandbagging/suppressed clones show a non-monotonic accuracy bump under mild Gaussian weight noise (σ ~ 0.001-0.01 relative to per-layer std), vs. monotonic degradation for a genuinely absent capability. Record how much budget it took to recover the known-present capability, then run the identical protocol on your real F2/F3 case and compare recovery cost against that calibration curve.

11. Probe 9 — the elicitation ladder: synthesizing 0-8 into one verdict

The AI-safety capability-elicitation literature solved almost exactly this problem for a different reason (sandbagging/eval-integrity), and its protocol is directly reusable — framework.md §5 covers the full genealogy (Greenblatt, Hofstätter, van der Weij, Ryd), not re-derived here. The ladder itself, made concrete for F2/F3:

Probe 3 (oracle in context)  →  few-shot (2-3 examples)  →  Probe 8 (light SFT, 10-50 rows)  →  Probe 6-gated GRPO  →  (still nothing) → confirmed K-gap
      (free)                       (cheap)                        (cheap-medium)                     (expensive)

• Step A — inject the answer (Probe 3). Fixed near-zero-shot → stop, K-gap confirmed, route to data curation.
• Step B — few-shot, 2-3 correct-usage examples, shuffled-label controlled (Probe 2’s control). Fixed → pure elicitation/prompting gap, no training needed at all.
• Step C — light SFT on 10-50 rows (Probe 8). Fixed → P/R-gap via light elicitation, matches Greenblatt’s finding directly.
• Step D — Probe-6-gated GRPO (never skip the spurious-reward control). Fixed and durable across checkpoints (re-check Pass@(k,T), Probe 7, for regression on the compositional segment) → confirmed P/R-gap, real discriminative fix.
• Step E — nothing works, even at large parameter/data budgets, even with the full oracle in context. Before concluding a hard ceiling, rule out deliberate or contextual suppression — van der Weij et al., arXiv:2406.07358 [HIGH]: frontier LMs can be prompted or fine-tuned to strategically underperform on recognizable-as-eval inputs while performing normally elsewhere. If ruled out, this is the genuine ceiling — a K-gap that survives even oracle injection is architecturally different from one that a light nudge fixes.

Order matters at Step D. Neither SFT nor RL alone reliably elicits held-back performance from a degenerate policy; SFT-on-weak-demonstrations-first, then RL, is what fully elicits it — RL-first from a degenerate starting point “almost always leads to reward hacking rather than genuine improvement.” Ryd et al., arXiv:2604.22082 [MED, 2026]. This is the reason the ladder above puts light SFT before GRPO even when both are on the table.

12. The purpose-built bottleneck CTF — a controlled instrument, not just an observational probe

Every probe above reads an existing decision point. This section builds one on purpose, so the verdict isn’t confounded by whichever real challenges happen to be in the portfolio.

Design. Author a challenge (a recommendation for challenge-builder’s seat, not something this chapter builds) where the flag is reachable only via the full methodology — no partial-credit path:

• A target with a subdomain (e.g. internal-api.<target>) that is discoverable only through a passive certificate-transparency-adjacent source that amass’s default source configuration queries and subfinder’s default source list omits — verify this asymmetry against each tool’s actual current default source list before building on it, source lists drift.
• The flag lives behind that specific vhost, reachable only after the correct enumeration → correct pivot → correct probe chain.
• The task prompt is deliberately light: “Find the flag.” No hint toward amass, no hint toward expert flags — leaking Probe-3-style oracle information into the baseline prompt would contaminate the very solve-rate you’re trying to read.

Run it. Pass@k sweep, k = {1, 4, 16, 64, 256}, across the full 5-family roster, same estimator as §4.

Read the solve-rate band — this reuses the project’s own locked GRPO go/no-go rule (llmresearch-handbook.md rule #7: baseline must hit 30-60%; below 5% no signal; above 80% no headroom) as a diagnostic read, not just a training gate:

Run this as a portfolio-level check, not a single-challenge anecdote. τ-bench’s own leaderboard shows exactly this kind of number can mislead if read only at pass@1: GPT-4o’s 61% pass@1 on retail tool-use collapses to under 25% pass^8 (all-k-succeed), Yao et al., arXiv:2406.12045 [HIGH]. A single high-k solve on a single bottleneck challenge is not the same claim as reliable multi-trial coverage across the portfolio — report the full curve, not one point on it.

13. The routing flowchart

flowchart TD
  Sym["Symptom: agent skips methodology (F1),<br/>calls subfinder not amass (F2),<br/>uses default not expert flags (F3)"] --> P0["Probe 0: grep run-trace corpus<br/>for amass / expert-flag strings<br/>across N existing rollouts"]

  P0 --> Freq{"Rare or absent<br/>at existing N?"}
  Freq -->|"Common already"| NotAGap["Not a gap at this decision point —<br/>check a different turn/challenge"]
  Freq -->|"Rare / absent"| P1["Probe 1: teacher-force the expert<br/>command, read logprob + rank;<br/>MCQ recognition, ge3 paraphrases"]

  P1 --> LP{"Logprob near floor AND<br/>MCQ recognition also fails,<br/>across paraphrasings?"}
  LP -->|"Yes"| P2a["Probe 2: pass@k sweep,<br/>k=1..256, ge10 rephrasings"]
  LP -->|"No — finite logprob<br/>or MCQ succeeds"| P3["Probe 3: in-context oracle<br/>injection + Teacher Flip Rate"]

  P2a --> Shape{"Correct action ever<br/>appears, any k, any phrasing?"}
  Shape -->|"Never, even k=256"| P3
  Shape -->|"Rises, plateaus above<br/>pass@1's floor"| P3

  P3 --> Fix{"Injecting the expert<br/>command / flags fixes it<br/>near-zero-shot?"}
  Fix -->|"No — still fails even<br/>with the answer handed over"| Know["KNOWLEDGE gap:<br/>curate paraphrase-rich SFT rows<br/>-&gt; knowledge-curation.md"]
  Fix -->|"Yes, but TFR(L1) low,<br/>only TFR(L3) is high"| P4["Probe 4: generation-vs-<br/>recognition (MCQ) split"]
  Fix -->|"Yes, and TFR(L1)<br/>already high too"| P5["Probe 5: token-level<br/>top-5 logprob-rank check"]

  P4 --> Disc{"Discrimination accuracy<br/>much greater than<br/>generation accuracy?"}
  Disc -->|"Yes"| P5
  Disc -->|"No, both low"| Know

  P5 --> Rank{"Expert token in the<br/>base model's top-5?"}
  Rank -->|"Yes"| P6["Probe 6: spurious-reward<br/>control before trusting RLVR"]
  Rank -->|"No, not even top-20"| Know

  P6 --> Spur{"Random reward ALSO<br/>moves the metric?"}
  Spur -->|"Yes — false positive"| ReDiag["Distrust the RLVR signal;<br/>re-run Probes 1-5"]
  Spur -->|"No — real discriminative work"| P7["Probe 7: Pass@(k,T),<br/>simple vs compositional segment"]

  P7 --> Comp{"Matched-data SFT REGRESSES<br/>the compositional segment's<br/>boundary (entropy-checked)?"}
  Comp -->|"Yes"| Explore["EXPLORATION-COLLAPSE dynamic:<br/>on-policy RL with diversity<br/>preservation, not more SFT<br/>-&gt; agentic-rl.md"]
  Comp -->|"No regression"| RP["PRIOR / POLICY gap:<br/>light SFT (Probe 8) or DPO/GRPO<br/>on this exact contrast<br/>-&gt; intervention-per-gap.md"]

  classDef verdict fill:#132b22,stroke:#34d399,color:#eafaf3;
  class Know,Explore,RP verdict;

14. The protocol, in order

Refuse to collapse this into one sentence. Per framework.md §8’s own conclusion — restated here because it’s exactly as true at the single-decision-point granularity as at the challenge-portfolio granularity — report the verdict per decision point, segmented by whether it’s a simple or compositional structure. A skeptical reviewer will catch “F2/F3 is an execution gap” stated as a flat claim; “F2 is a confirmed P-gap (Probe 3 fixed it, Probe 5 confirms top-5, Probe 6 confirms real discriminative work) and F1 is unresolved pending framework.md §6.1’s fault labeling” is the defensible version.

Cross-links

• Diagnosing the gap — a scientific framework — the theory this chapter operationalizes: full pass@k/Cover@τ/Pass@(k,T) derivations (§2), the elicitation-ladder genealogy (§5), the planning/state-management axis for F1-style symptoms (§6), and the RL-PLUS entropy-collapse mechanism this chapter’s Probe 7 leans on (§7).
• ./taxonomy.md — the precise K/R/P definitions this chapter assumes; §0 is the canonical K/R/P ↔ knowledge/execution/exploration mapping this chapter’s §0 points to rather than re-derives.
• ./intervention-per-gap.md — what to actually build once a probe returns a verdict: the R/P-gap DPO/GRPO row shape, the K-gap SFT-curation recipe.
• ./knowledge-curation.md — the full K-gap data-row recipe (paraphrase volume, directional pairs, entity-connective text) once Probe 3 confirms a K-gap; not re-derived here.
• ./trajectory-amplification.md — the forgetting/hallucination-amplification mechanics behind Probe 1’s caution against premature SFT on an unconfirmed K-gap.
• The decision — the one-line version of the routing question both this chapter and framework.md expand on.
• Contested edges & landmines §1, §7 — the crossover-test’s own contested status, referenced in §4 rather than re-argued.
• Agentic & multi-turn RL — where the Exploration-collapse fix (on-policy RL, entropy preservation) is implemented once Probe 7 diagnoses it.
• The kinds of SFT — the production data-curation pipeline Probe 8’s diagnostic light-SFT is explicitly not a substitute for.
• Foundations: the one axis that predicts everything — the on/off-policy distinction underlying why oracle-injection (§5) and DPO/GRPO pairs must stay on-policy to be trustworthy.

Bibliography

Cited but not counted as arXiv: OpenReview id Dkgx2pS4Ww (Donoway, Joren, Somani, Sleight, Michael et al., Quantifying Elicitation of Latent Capabilities in Language Models, NeurIPS 2025 poster) — §10, no confirmed arXiv mirror, cite cautiously.

Confidence calibration, restated: every id above is CONFIRMED in ledger-A.md or ledger-B.md; none is invented. Where section-A’s narrative referenced an id absent from either ledger (Verifier Gain, RankAlign, and two OpenReview-only ids with no ledger row), that material was dropped from this chapter rather than cited unverified — the load-bearing generation-vs-recognition claim in §6 rests on West et al. (confirmed, HIGH) instead.