Teaching a tool, teaching recon — the data behind a skill

You want to teach a model two different things: (a) a specific tool with a fixed signature — grep, the Vercel CLI, a REST endpoint — and (b) a search methodology — “explore many things to find some things,” a grounded decision tree that fans out and prunes. Both are “tool use” in the loose sense. They are not the same training problem, and — this is the crux — for neither one does the answer route through Q&A. This chapter is about the DATA OBJECT: what shape of row you write to disk (or generate online), how it gets made, whether the tool was actually executed to produce it, and which technique it pairs with.

Framing: general research reference, drawn from seven sources (Toolformer, ToolLLM/ToolBench, APIGen/xLAM, Gorilla/DocPrompting, terminal-CLI-agent literature, search/recon-RL literature, and a procedural-vs-declarative synthesis). Every arXiv id below was cross-checked live 2026-07-02 against the source notes; only ids present in those notes appear here.

1. The axes, and the up-front verdict

Three bodies of vocabulary are in play, and disambiguating them is the first move:

• Tool learning / tool-use training / function-calling / tool-integrated reasoning (TIR). The literature for teaching a model to call a fixed-signature tool correctly — right name, right arguments, right moment, right reaction to the result. This is the “teach grep” problem.
• Search / information-seeking RL. The literature for teaching a model to explore — issue a query, read what comes back, decide whether to search again or answer, with no single correct path. This is the “teach recon” problem. It is tool-use RL’s cousin, not tool-use RL itself: the object being learned is a branching strategy, not a call signature.
• Procedural vs. declarative knowledge. The cognitive-science distinction that explains why the first two need different data than a knowledge base does. Declarative knowledge is “facts you can state” (what a flag does); procedural knowledge is “a skill you can execute” (when to reach for that flag, how to recover when it returns nothing). Training data that only states facts caps out at declarative recall no matter how much of it you have.

The up-front verdict, unanimous across all seven sources: Q&A (Q: "What does grep do?" A: "...") teaches facts about a tool. It does not teach skill with a tool — sequencing, argument selection under a real task, reading an observation to decide the next action, recovering from a failed call. Every source that runs the comparison (ToolBench, APIGen/xLAM, Gorilla/DocPrompting, procedural-vs-declarative) lands on the same conclusion: trajectories — ideally executed — are necessary for procedural transfer; Q&A is a legitimate thin seed for call-syntax, never the primary signal. Toolformer is the instructive edge case: it isn’t Q&A and isn’t a full multi-step trajectory either — it’s a self-supervised single (call, result) insertion into raw text, and it lands on the executed-trace end of the spectrum because the keep/discard filter is grounded in the real API output’s effect on next-token loss, not in a hand-written answer.

This is the same principle this book states elsewhere as “execution is the engine” — see kinds-of-sft.md for the general argument that whether a trajectory was executed against a real environment (rather than synthetically authored/narrated) is the single biggest lever on whether SFT teaches a transferable skill or teaches confabulation. This chapter is that argument applied specifically to tool-calling and search.

2. Teaching A TOOL (grep, the Vercel CLI, any fixed-signature callable)

2.1 The data-object ladder

Quality increases strictly down this ladder — each rung is necessary, none is sufficient on its own:

Rung 1 — doc-grounding / retrieval. Gorilla (arXiv:2305.15334) trains SFT with the oracle API doc concatenated straight into the prompt, so the model learns to read a doc rather than memorize one — this is what drops its hallucination rate to 6.98% vs. GPT-4’s 36.55% on TorchHub. DocPrompting (arXiv:2207.05987) goes one step further and trains a retriever + FiD generator jointly over (NL_intent, top-k_doc_paragraphs, reference_code). Neither executes the tool during data creation — the doc is trusted as ground truth. DocPrompting’s own ablation is the sharpest argument for why this rung exists at all: retrieving docs (versus retrieving worked examples) roughly doubles unseen-function recall (9.03% → 18.30%), because a doc generalizes to argument combinations an example never showed, but an example doesn’t. This rung is necessary — it’s where hallucinated signatures die — but it’s still one-shot code/call generation, not a decision loop.

Rung 2 — tool-call trajectories. ToolLLM/ToolBench’s canonical training row (arXiv:2307.16789):

{
  "instruction": "<user task>",
  "relevant_apis": ["API_1", "API_2"],
  "solution_path": [
    {
      "step": 1,
      "thought": "<reasoning about next action>",
      "api_name": "SearchByTitle",
      "parameters": {"title": "Batman", "country": "us"},
      "api_response": { "...real JSON response from actual API execution..." }
    },
    {
      "step": 2,
      "thought": "<reasoning based on step 1 response>",
      "api_name": "GetStreamingAvailability",
      "parameters": {"movie_id": "12345"},
      "api_response": { "...real response..." }
    }
  ],
  "final_answer": "<final output to user>"
}

Every API call in the 126,486-instance dataset is annotated by DFSDT (depth-first search with decision tree) — the annotator model is allowed to branch, backtrack on a failed call, and keep only working paths — which is what makes this trajectory-shaped rather than a single-shot call. The API documentation (name, params, example response) is attached at every step, so the model learns API semantics alongside call syntax, not call syntax alone. All 469,585 API calls behind ToolBench were made against real RapidAPI endpoints — not simulated REST mocks, which is exactly the gap ToolBench closes relative to prior single-tool, no-execution datasets like APIBench and ToolAlpaca.

Rung 3 — executed + verified traces. This is where every source converges as the non-negotiable floor for CLI-grade reliability. APIGen’s row (arXiv:2406.18518), using grep directly as the worked example:

{
  "query": "Check if the word 'error' appears in the log file at /var/logs/app.log",
  "function_calls": [
    { "name": "grep", "arguments": { "pattern": "error", "file": "/var/logs/app.log", "case_insensitive": false } }
  ],
  "observations": [
    { "type": "tool_output", "content": "[Line 42] ERROR: Connection timeout\n[Line 88] ERROR: Auth failed", "execution_time_ms": 14 }
  ],
  "expected_action": "The tool executed successfully; report the found errors.",
  "parallel_calls_supported": true
}

And xLAM/APIGen-MT’s multi-turn extension (arXiv:2409.03215, arXiv:2504.03601) — the worked example is literally the Vercel CLI:

{
  "conversation_id": "conv_001",
  "turns": [
    { "user_query": "Deploy my Vercel site",
      "function_calls": [ { "name": "vercel_list_projects", "arguments": {} } ],
      "tool_output": { "projects": [{"name": "my-app", "id": "proj_123"}] } },
    { "user_query": "Now deploy it to production",
      "function_calls": [ { "name": "vercel_deploy", "arguments": { "project_id": "proj_123", "environment": "production" } } ],
      "tool_output": { "deployment_url": "https://my-app.vercel.app", "status": "ready" } }
  ],
  "blueprint": {
    "task": "Deploy a web app to production via Vercel",
    "ground_truth_actions": ["list_projects", "deploy"],
    "dependencies": ["project_id must come from list_projects output"]
  }
}

Note the state-threading in the blueprint: project_id in turn 2 must literally be the value that came back from turn 1’s real execution — this is only checkable because the tool was actually run. APIGen’s gate that produces rows like this is a 3-stage verifier: (1) format check — does the call parse and match the schema (does the grep call have pattern/file, is case_insensitive a bool); (2) execution — actually run it in a sandbox (real grep, real Vercel SDK against a test project, real HTTP request) and capture stdout/exit code/latency; (3) semantic check — an LLM judge asks “does this (query, call, observation) triple actually make sense,” which is what catches a call that executed successfully but whose logged observation was hallucinated rather than the tool’s real return. From 3,673 executable APIs this produced 60,000 verified samples. Toolformer’s loss-reduction filter (arXiv:2302.04761) is the self-supervised sibling of the same idea — keep an inserted <API> call → result </API> block only if the real API result measurably lowers next-token loss (L_i^- − L_i^+ ≥ τ_f) — grounded in execution, gated by a different (automatic, LM-internal) signal rather than an LLM judge. Models trained on rung-3 data (APIGen/xLAM, 1B–70B) top BFCL and beat GPT-4o/Claude-3.5 on τ-bench — the empirical payoff for paying the execution cost.

2.2 How each rung gets MADE

2.3 The pipeline, end to end

flowchart LR
    A["Docs / --help / man pages\n(declarative floor)"] --> B["LLM proposes candidate\ninstruction + call"]
    B --> C["EXECUTE the tool for real\n(sandbox / test project / live CLI)"]
    C --> D{"Verify\nformat -> execution -> semantic\n(or: loss-reduction filter)"}
    D -- fail --> B
    D -- pass --> E["SFT on (instruction, trajectory,\nreal_observation) rows"]
    E --> F{"Plateau on\nheld-out tasks?"}
    F -- no, good enough --> G["Ship"]
    F -- yes --> H["Optional RL refinement\n(decomposed reward, on-policy\nexecution each rollout)"]

2.4 Worked recipe: teaching grep / the Vercel CLI

Step 1 — Docs-in-context (declarative floor)
  Data: {tool_name, signature, flags[], description, example_invocations} from `man`/`--help`/API docs.
  Purpose: kill hallucinated flags. Gorilla-style: concatenate the retrieved doc into the training
  prompt so the model learns to READ docs, not memorize them.

Step 2 — Executed CLI trajectories (procedural, grounded from the start — do not skip execution)
  Data object: {instruction, [{thought, command, real_stdout, real_stderr, exit_code}]_1..n, final_answer}
  How made: propose a command for a realistic task ("find all TODOs under src/ modified this week");
  RUN IT in a real/sandboxed filesystem or a test Vercel project; capture the ACTUAL output — never
  simulate. This is APIGen's execution stage and ToolBench's "real responses, not simulated"
  requirement, applied to a CLI instead of a REST API.
  Diversity axes: vary flags, vary phrasing, include multi-command pipes (`grep | sort | uniq`),
  include distractor tools (`find` vs `locate` vs `grep`), include failure cases (wrong flag, 0
  matches) so the model sees real error text, not invented text.

Step 3 — Verifier-filtered SFT
  Filter: APIGen's 3-gate check (format-valid -> executes without error -> semantically answers the
  instruction, LLM-judge or exact-match) OR Toolformer's loss-reduction filter (keep only if the
  executed result measurably helps predict the continuation).
  Train: standard next-token CE loss on (system, user, assistant-with-tool-call, tool_result) tuples.
  500-1000 verified trajectories is APIGen/xLAM's working scale for a single tool; ToolBench's ratio
  (~4 reasoning turns/instruction) is a reasonable trajectory-depth target.

Step 4 — Optional RL (only if Step 3 plateaus, or exit-code/efficiency matters)
  Reward: dense, decomposed (ToolRL) — R_format (valid syntax) + R_execution (ran without error) +
  R_answer (task solved) — NOT one sparse outcome reward, which collapses past ~4 steps for
  compositional CLI tasks. GRPO on-policy, executing the sampled command for real each rollout.
  Structurally this is imitation -> RL refinement; rejection sampling is the simplest version —
  canonical home: methods/imitation.md. Reach for full RL only when SFT plateaus.

Q&A’s only legitimate role in this recipe: a small (~50-100), format-only seed (Q: find TODOs in python files -> A: grep -r TODO --include='*.py') to bootstrap valid call syntax before Step 2 — never the primary signal.

3. Teaching RECON — search / information-seeking RL

Recon is not a longer tool-call trajectory. It’s a search tree with an outcome-only reward, and the winning recipe is structurally different from §2:

methodology-seed (thin SFT, format-only)  ->  RL discovers the branches
(cold-start: teaches the CALL FORMAT,          (Search-R1 / R1-Searcher / DeepResearcher / ToolRL:
 NOT the decision tree)                         outcome reward on REAL execution, retrieved-token
                                                 masking, GRPO — the tree structure emerges)

3.1 The data object

(query, trajectory=[(action, real_observation)]_1..n, final_answer, outcome_reward) — crucially, the intermediate steps are not supervised; only the terminal outcome is rewarded. This is what forces the model to discover which branches are worth exploring rather than imitate one hand-authored path. Search-R1’s worked row (arXiv:2503.09516):

{
  "question": "Who won the 2024 US presidential election?",
  "trajectory": [
    {
      "step": 1,
      "reasoning": "I need to search for the 2024 US election results",
      "search_query": "2024 US presidential election winner",
      "observation": "<retrieved_passage_text>",
      "answer": "Harris won the election"
    }
  ],
  "final_answer": "Kamala Harris",
  "reward": 1.0
}

Note the intermediate answer field is allowed to be wrong — it’s not supervised, only the terminal one is. This row is generated online, during training, not pre-computed: for each task, sample N rollouts against a live/sandboxed retriever (Search-R1 used online BM25 over 2018 Wikipedia), each a real cold-start → search → reason → answer sequence, reward = exact-match on the final answer.

3.2 Retrieved-token masking — the mechanism that makes the strategy transfer

During the GRPO backward pass, loss is computed only on model-generated tokens (the query text, the reasoning) — never on the retrieved/observed passage tokens. This stops the model from overfitting to one retrieval corpus’s exact phrasing and is exactly why the learned strategy — when to search again, when to stop — transfers to a new corpus or a new tool, whereas a model trained by imitating fixed retrieval snippets would not. The same masking rule generalizes past web search: shell stdout, an HTTP response body, a scan’s raw output — anything that came back from the environment rather than from the model gets masked the same way. Full mechanics of this masking rule inside the broader tool-integrated-reasoning RL literature (ReTool, ToRL, Search-R1) live in methods/rl-long-horizon-exploration.md §1.8 — not re-derived here.

3.3 Why SFT-first-then-RL is contested, not settled

Nemotron-Research-Tool-N1 (arXiv:2505.00024) found pure RL from a cold start outperformed SFT-then-RL on their benchmark — a thick SFT seed on distilled reasoning traces overfit to that reasoning style and hurt downstream RL. R1-Searcher (arXiv:2503.05592) runs no cold-start SFT at all, using a two-stage reward instead: stage 1 rewards invoking retrieval in the right format, regardless of correctness (teaches the call shape); stage 2 rewards getting the final answer right (teaches using the retrieved info correctly). WebGPT (arXiv:2112.09332), by contrast, used human demonstrations (behavior cloning) followed by reward-model-guided rejection sampling — and that worked well precisely because its action space (arbitrary web clicks) is far larger and less constrained than a small fixed API surface, closer to the terminal-agent action spaces below.

Working rule for this book: a thin, format-only SFT seed — teaching valid call syntax, not a decision tree — is safe and typically improves sample efficiency; a thick SFT seed on full expert reasoning traces risks exactly the Nemotron overfitting failure. This is the same trade-off this book covers in depth for the general hint-guided case — see hint-guided-bootstrapping.md — apply that chapter’s risk register before deciding how thick your recon seed should be.

3.4 Let branching emerge; do not author it

DeepResearcher’s outcome-only RL (arXiv:2504.03160) produces self-reflection, cross-validation, and honesty (“I can’t find X”) as emergent behaviors — none were hand-coded into the reward or the seed data. This is the empirical case for seed-then-RL-discovers over hand-authoring a decision tree: a fixed methodology (“always whois, then DNS, then web”) is brittle, because real recon has too many valid orderings. TIER (arXiv:2605.16790) makes the negative case concrete: a trajectory-supervised reward (reward the model for matching one authored path) collapses past depth 4–6, because it penalizes perfectly valid alternative orderings the annotator didn’t happen to take. Execution-grounded, outcome-only reward doesn’t have this failure mode — it rewards any trajectory that reaches the correct terminal state, at whatever depth. If a curriculum is needed at all, order by task difficulty (easy targets first) — R1-Searcher found that specifically including hard questions prevents early convergence and under-exploration.

Full RL mechanics — GRPO, PPO-clip, group-relative advantage, long-horizon credit assignment at depth — are this book’s own territory: methods/rl-long-horizon-exploration.md and methods/agentic-rl.md. Not re-derived here.

3.5 Worked recipe: teaching recon

Step 1 — Methodology seed (thin SFT, NOT a full decision tree)
  Data: a small number (tens, not thousands) of demonstrated (query, action, real_observation, ...,
  outcome) trajectories showing the CALL FORMAT for the recon tools available (search, grep,
  nmap-equivalent, whois, etc.) — teach "here is how you emit a well-formed action," not "here is the
  one correct order to explore in." Over-specifying (full expert reasoning traces) risks the
  Nemotron-Tool-N1 failure: the model memorizes the demonstrated path instead of learning to search.

Step 2 — RL with outcome-only reward + retrieved-token masking
  Data object generated ONLINE during training (not pre-computed): for each task, sample N rollouts,
  each a real sequence of (action, real_observation) pairs against a live/sandboxed target, terminating
  in an answer; reward = binary or graded task success (flag found / answer exact-match / vulnerability
  confirmed) — intermediate steps unsupervised.
  Mask the loss to generated tokens only (actions + reasoning), never retrieved/observed text.
  GRPO update per batch of rollouts.

Step 3 — Let branching emerge; do not author it
  Do not hand-write a fixed scaffold ("first whois, then DNS, then web") — it is brittle. RL with a
  good outcome reward reliably discovers cross-validation, backtracking-on-dead-end, and stopping
  conditions on its own. If curriculum is needed, order by difficulty (easy targets first) to avoid
  early convergence / lack of exploration.

4. Consolidated table: skill-type → data object → how made → executed? → technique

5. Two concrete recipes, side by side

6. Confidence and what’s contested

• High confidence, convergent across all 7 sources: the Q&A-vs-trajectory verdict (§1), the executed > synthetic ordering for tool data (§2.1), and retrieved-token masking as the mechanism that lets a learned search strategy generalize (§3.2). These are not single-paper claims.
• Contested, stated as such by the sources themselves: whether recon needs any SFT seed before RL (Nemotron-Tool-N1’s pure-RL-wins result vs. R1-Searcher’s two-stage-reward-no-SFT vs. WebGPT’s imitation-then-rejection-sampling for a much larger action space) — see §3.3. Don’t present “thin seed then RL” as settled; present it as this book’s working rule given the evidence on both sides.
• Working from the source notes, not a fresh independent search this session — the arXiv ids above were verified live during the research pass the notes themselves record (2026-07-02); this chapter synthesizes rather than re-verifies.

Cross-links (canonical homes — referenced, not re-derived here)

• Why execution is the training engine, generally: kinds-of-sft.md.
• Rejection sampling / STaR-family imitation-refinement (the RL-lite mechanism behind the optional RL step in §2.4, and WebGPT’s rejection-sampling stage): methods/imitation.md.
• Full RL mechanics for search/exploration and long-horizon agentic credit assignment (GRPO, outcome-only reward at depth, retrieved-token masking’s place in the wider RL literature): methods/rl-long-horizon-exploration.md + methods/agentic-rl.md.
• STaR / hint-guided bootstrapping (thin-seed-then-let-the-model-discover, and the risk of an over-thick seed collapsing exploration): hint-guided-bootstrapping.md.
• On-policy vs. off-policy as the organizing axis underneath “executed at rollout time” vs. “pre-computed and replayed”: foundations/on-off-policy.md.
• General method→data framing: method-to-data.md.

Verified arXiv IDs referenced in this chapter

Toolformer 2302.04761 · ToolLLM/ToolBench 2307.16789 · APIGen 2406.18518 · xLAM 2409.03215 · APIGen-MT 2504.03601 · Gorilla 2305.15334 · DocPrompting 2207.05987 · Search-R1 2503.09516 · R1-Searcher 2503.05592 · ToolRL 2504.13958 · Nemotron-Research-Tool-N1 2505.00024 · DeepResearcher 2504.03160 · WebGPT 2112.09332 · TIER 2605.16790. All cross-checked live 2026-07-02 per source notes.