What knowledge data actually looks like (once a K-gap is confirmed)

The question this chapter answers: once diagnosis has ruled in a knowledge gap — the model never emits the right fact/tool/flag at any sampling budget, and handing it the answer in-context fixes it zero-shot — what shape of file do you actually build? Not “paste the README into the system prompt and fine-tune on that.” Not “generate one help-page dump per tool.” Not “make it memorize a tool list.” BLUF, upfront: a fact that appears exactly once, in exactly one wording, in your training data is storable (the model can echo it back near-verbatim) but not extractable (it cannot answer a differently-phrased question about it) — feeding the README once is close to a null intervention, and no amount of downstream trajectory-level SFT repairs that gap once it’s baked in. Extraction has to be purchased at data-construction time: paraphrase volume, bidirectional coverage, and — for the specific “wrong tool, default not expert flags” failure this book keeps coming back to — execution-verified tool-CALL rows, not declarative prose. This chapter is the graded ladder from weakest to strongest, a worked training row at every rung, and the full production schema at the end.

Scope note, standing rule: every citation below is general ML/knowledge-injection literature. Nothing here leans on an academic cybersecurity-LLM paper — this book’s rule throughout, restated because this is exactly the chapter where it’s tempting to reach for a domain-specific “how we built our pentest corpus” paper instead of the general mechanism.

1. The load-bearing principle: one exposure ≈ zero extraction

Ground truth, from a controlled synthetic-biography corpus where the true answer is knowable and every exposure is countable: Allen-Zhu & Li’s Physics of Language Models Part 3.1 (arXiv:2309.14316) trains models on a biography dataset (bioS(N)) and measures closed-book QA accuracy against how many distinct wordings of each fact appeared during training. A fact seen in one wording: ~0-10% QA accuracy, independent of model size, training duration, or how much downstream instruction-tuning you throw at it afterward. Add ~5 diverse rewrites of the same fact and accuracy jumps to ~97%. Even sentence-permutation alone, with zero new wording — literally shuffling the clauses of the same sentence — already lifts extraction from 4.4% to 70%. The mechanism (linear-probe evidence in the same paper): without augmentation, the fact is encoded only at the local context hidden state around where it appeared, never bound to the entity-name token itself, so it can’t be retrieved by a query that doesn’t reproduce that exact local context.

The corollary that matters for this whole chapter: the paper’s own experiments show no amount of downstream instruction fine-tuning repairs an under-augmented base representation. SFT can teach the model how to answer, but if the fact was never bound to the entity token during pretraining/CPT, SFT has nothing to point at. This is the mechanistic answer to “does trajectory-level SFT amplify an unfixed knowledge gap” from the storage side: SFT inherits the storage format, it does not repair it (full amplification argument in Does trajectory SFT amplify an unfixed K-gap?).

The budget for how many exposures “enough” actually is: Part 3.3 (arXiv:2404.05405) measures raw storage capacity at ~2 bits of factual-tuple knowledge per parameter, achieved at roughly 1000 exposures per fact; drop to 100 exposures and capacity roughly halves to ~1 bit/param; junk-diluted corpora (1 useful token to 7 junk) cut capacity ~20×, partly recovered (~2×) by a consistent domain-marker prefix. (A differently-measured, higher raw-memorization ceiling of ~3.6 bits/param exists — Morris et al. — but that’s a random-bitstring membership-inference regime, the theoretical ceiling for arbitrary data, not the practical ceiling for genuinely extractable, augmented, structured knowledge; the two numbers aren’t in tension, they’re measuring different things.)

Why some facts are missing in the first place, before you even get to fixing it: Kandpal et al. (arXiv:2211.08411) show real-corpus QA accuracy correlates causally with how many pretraining documents mention the entities involved. amass is a smaller, less-frequently-discussed tool than subfinder on the open web relative to its actual capability — that’s plausibly why “amass covers passive-DNS/certificate-transparency sources subfinder’s defaults miss” ends up in a model’s blind spot: not an architectural limit, a training-corpus frequency artifact. That’s the diagnosis-side story; this chapter is the fix.

Build this: before writing a single training row, budget for hundreds, not one, exposures per fact — different wordings, different formats, different directions. If your curation plan has one row per fact, stop; you’re about to reproduce the 2309.14316 null result.

2. The graded ladder — six rungs, weakest to strongest

Six shapes of data, in increasing extraction reliability, each with a worked training row for the running example: “amass covers a passive-DNS/certificate-transparency source subfinder’s default flags miss, and subfinder has an expert flag-set (-active, -all, -rl, -oJ, …) the agent never reaches for.”

graph LR
  R0["Rung 0<br/>Raw doc<br/>(README / --help)"] -->|"~0% extraction<br/>2309.14316"| R1["Rung 1<br/>Paraphrase volume<br/>WRAP-style"]
  R1 -->|"fixes non-directional<br/>recall, NOT direction"| R2["Rung 2<br/>Forward+reverse<br/>synthetic QA"]
  R2 -->|"declarative fact ≠<br/>correct tool CALL"| R3["Rung 3<br/>Doc→tool-CALL<br/>rows (RAT + verified)"]
  R3 -->|"single facts, not<br/>chained reasoning"| R4["Rung 4<br/>Entity-pair<br/>connective text"]
  R4 -->|"external synthesis<br/>can drift/omit"| R5["Rung 5<br/>Self-play<br/>Self-QA"]

  style R0 fill:#3a1a1a,stroke:#f87171,color:#fee2e2
  style R3 fill:#132b22,stroke:#34d399,color:#eafaf3

2.0 Rung 0 — raw docs are source material, never a training target

The README, --help output, man page, OpenAPI spec: these are the input to a curation pipeline, never trained on directly and unaugmented. Ovadia et al.’s RAG-vs-FT study (arXiv:2312.05934) is the floor to beat: unsupervised fine-tuning on raw domain text consistently loses to simple retrieval-augmented generation on knowledge-injection tasks — training on the raw doc doesn’t even reliably clear the bar a zero-training RAG pipeline sets for free. Cheng et al.’s AdaptLLM (arXiv:2309.09530) diagnoses why directly: continued pretraining on raw domain text does inject some knowledge, but “drastically hurts prompting ability for QA” — the model gets worse at being asked things, even about the material it just trained on, unless every raw chunk is paired with task-formatted continuations before CPT.

// NOT a training row — CPT/curation SOURCE MATERIAL ONLY, never the target
{
  "source_id": "subfinder-help-v2.7.1",
  "raw_text": "$ subfinder -h\nFlags:\nINPUT:\n   -d, -domain string[]   domains to find subdomains for\nSOURCE:\n   -s, -sources string[]  specific sources to use for discovery\n   -recursive             use only recursive subdomain enumeration sources\n   -all                   use all sources for enumeration (slow)\nFILTER:\n   -active                display active subdomains only\nOUTPUT:\n   -oJ, -json             write output in JSONL(ines) format\n   -rl, -rate-limit int   maximum number of http requests to send per second"
}

This never appears as an assistant target. It’s the thing every downstream rung is derived from.

2.1 Rung 1 — paraphrase volume (WRAP-style)

Maini et al.’s WRAP (arXiv:2401.16380) paraphrases noisy web text into several styles (including explicit “QA format”) and joint-trains real+synthetic: ~3× pretraining speedup, >10% perplexity gain, >2% zero-shot QA gain at fixed compute. This is the direct fix for the 2309.14316 null result — turn one raw doc into several differently-worded statements of the same fact.

Rephrase count is not free of diminishing returns: a very recent Stanford result (arXiv:2603.18534, promising, not yet independently validated) finds rephrase-count efficiency saturates around ~32 rewrites per document, and that packing those 32 rewrites into one long megadocument (rather than 32 separate short training examples) pushes efficiency further (~1.48× → ~1.8×). Practical read: don’t stop at 5, and don’t naively keep generating past ~32 either — and prefer packing multiple rewrites into a single training sequence over one-rewrite-per-row.

// Rung 1 — same fact, 4 of ~20-32 total rewrites, task-formatted per AdaptLLM (2309.09530)
[
  {"messages": [{"role": "user", "content": "What's the -all flag on subfinder for?"},
    {"role": "assistant", "content": "-all tells subfinder to query every configured source, not just the fast/default subset — slower, broader coverage."}]},
  {"messages": [{"role": "user", "content": "True or false: subfinder's default run queries every available source."},
    {"role": "assistant", "content": "False. The default subset is a faster subset; -all opts into every configured source."}]},
  {"messages": [{"role": "user", "content": "To get subfinder's broadest possible source coverage, which flag do you add?"},
    {"role": "assistant", "content": "-all — it enables every configured source instead of the default fast subset."}]},
  {"messages": [{"role": "user", "content": "I ran subfinder with no flags and got fewer subdomains than expected. Why, and how do I fix it?"},
    {"role": "assistant", "content": "Default subfinder uses a fast subset of sources, not all of them. Add -all to query every configured source for broader (slower) coverage."}]}
]

Still not fixing directionality — see rung 2.

2.2 Rung 2 — doc → synthetic QA, forward AND reverse

Paraphrase volume fixes non-directional recall. It does not fix a structurally separate failure: the Reversal Curse. Berglund et al. (arXiv:2309.12288) train on “A is B” and show P(correct | “B is ?”) is statistically indistinguishable from a random-name baseline — and, critically, the authors’ own tested augmentations do not fix this. Kitouni et al.’s Factorization Curse (arXiv:2406.05183) generalizes this: it’s a failure to learn the same joint distribution under different token-order factorizations, and it survives scale, reversed-token training, and naive bidirectional attention. Lin et al. (arXiv:2410.18808) narrow it further and give the actionable lever: generalization to the reverse direction is gated by training-document structure (“[Name] is [Description]” reverses one way, not the other) — and works fine when both directions are given jointly in one training example. (A very recent mechanistic follow-up, arXiv:2604.04943, promising, not yet independently validated, shows bidirectional-objective tricks like masked reconstruction do raise reversal accuracy behaviorally — but by storing forward and reverse as two separately-indexed representations, not one shared direction-agnostic fact. You’re not getting free generalization, you’re paying for a second copy.)

Practical rule: write both directions explicitly, as separate rows, in the same batch.

// Rung 2 — forward and reverse, SAME fact, explicit both ways (2309.12288 fix)
{"messages": [
  {"role": "user", "content": "Does amass or subfinder pull from certificate-transparency logs by default?"},
  {"role": "assistant", "content": "amass — its default enumeration includes CT-log and passive-DNS sources. subfinder's default source set is narrower and can miss what CT logs surface unless you widen it with -all."}
]}
{"messages": [
  {"role": "user", "content": "Which tool should I reach for if I specifically need certificate-transparency coverage that subfinder's defaults don't include?"},
  {"role": "assistant", "content": "amass — run it in passive mode (amass enum -passive -d <target>) to pull CT-log and passive-DNS sources subfinder's default set doesn't cover."}
]}

2.3 Rung 3 — doc → tool-CALL examples: the highest-value rung for “wrong flags”

This is the rung that directly answers “the agent reaches for one salient tool, uses default not expert arguments” — declarative QA about a flag is not the same skill as emitting the call. Patil et al.’s Gorilla (arXiv:2305.15334) is the founding recipe: scrape real API docs, Self-Instruct-generate NL questions per doc entry, grade candidate calls by AST-subtree match against the schema — and, the single most important mechanism here, Retriever-Aware Training (RAT): the doc schema stays in-context at train time, not just at data-generation time. The learned skill becomes “read this schema, emit this call,” not “this exact instruction maps to this memorized call” — which is what lets it generalize to an updated flag set with no retraining, and is why raw doc-only fine-tuning (rung 0) hallucinates calls that Gorilla-style RAT training doesn’t. Hsieh et al. (arXiv:2308.00675) independently confirm the context-side of this: zero-shot prompting with only the tool docs in context matches or beats few-shot demonstrations once the tool count grows — reinforcing “the schema belongs in context, not memorized as a fact.”

Qin et al.’s ToolLLM (arXiv:2307.16789) scales this to 16,464 real API docs and adds the piece that matters for multi-step agentic data specifically: curate the trajectory, not just the terminal call, via a depth-first search decision tree (DFSDT) so failed/incomplete reasoning branches get pruned before they become training data.

The 2024-2025 cluster hardens the verification step into a formal pipeline. Liu et al.’s APIGen (arXiv:2406.18518) runs candidate calls through 3-stage verification — format check, execution check, semantic check — before a row survives; a 7B model trained on the output beats GPT-4 variants on BFCL. Liu et al.’s ToolACE (arXiv:2409.00920) self-evolves a synthetic pool of 26,507 diverse API specs under a dual rule-based+model-based verifier, deliberately covering argument-shapes real scraped docs don’t have. Two very recent papers name the “default not expert flags” symptom almost literally: Yin et al.’s Magnet (arXiv:2503.07826) builds a graph over function signatures and walks reliable multi-tool signature-paths — the concrete fix for “reaches for one tool, not the complete expert set.” Greenstein et al. (arXiv:2601.17829, very recent, promising, single paper, not yet independently replicated) is a direct critique of the entire prior lineage: every pipeline from Gorilla through ToolACE optimized query/function/turn-count diversity but left argument-VALUE coverage underexplored — does the corpus actually range over -rate-limit’s valid integer domain, or does every row use the canonical default? This is the most literal match to “uses default not expert arguments” in the current literature, and it’s genuinely unresolved.

// Rung 3 — verified tool-CALL row. Schema in context (RAT), never the loss target.
{
  "system": "Tools available: subfinder(domain, sources?, active?, all?, rate_limit?), amass(domain, mode: 'passive'|'active', sources?). Find subdomains for the target, preferring the source set with the broadest coverage.",
  "messages": [
    {"role": "user", "content": "Enumerate subdomains for corp-target.com — I need CT-log and passive-DNS coverage, not just the fast default set."},
    {
      "role": "assistant",
      "content": "subfinder's default source set is narrower than what's needed here; amass's passive mode explicitly covers CT-log and passive-DNS sources. Running amass first.",
      "tool_calls": [{"id": "call_1", "type": "function",
        "function": {"name": "amass", "arguments": "{\"domain\": \"corp-target.com\", \"mode\": \"passive\"}"}}]
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "{\"subdomains\": [\"vpn.corp-target.com\", \"staging-api.corp-target.com\", \"...\"], \"source_count\": 14}"},
    {
      "role": "assistant",
      "content": "Widening with subfinder -all to cross-check against amass's results and catch anything CT-log/passive-DNS misses.",
      "tool_calls": [{"id": "call_2", "type": "function",
        "function": {"name": "subfinder", "arguments": "{\"domain\": \"corp-target.com\", \"all\": true, \"rate_limit\": 20}"}}]
    },
    {"role": "tool", "tool_call_id": "call_2", "content": "{\"subdomains\": [\"vpn.corp-target.com\", \"dev.corp-target.com\"], \"source_count\": 22}"}
  ]
}

This row was executed and verified (APIGen’s format → execution → semantic checks) before it entered the set — never hand-authored, the same confabulation risk The kinds of SFT §4 already names for trajectory data generally. Loss masked to assistant turns only — same rule as The kinds of SFT §5.1: the model must learn to produce the call, not to predict the tool’s output.

2.4 Rung 4 — entity-pair connective text (chaining/manipulation questions)

Even perfectly extractable single facts fail at manipulation — classification, comparison, inverse search across two facts — without explicit training on the relationship. Physics 3.2 (arXiv:2309.14402) shows this precisely: knowledge that’s individually stored and extractable can still fail at “which of these two applies here” unless chain-of-thought is used at both train and inference time, or the relationship itself was trained explicitly. This is exactly the “does amass’s output feed usefully into subfinder’s next pass, or should I just use one or the other” question — a chaining question about two entities, not a fact about either alone.

Yang et al.’s EntiGraph (arXiv:2409.07431) is the concrete recipe for small corpora specifically — the tool/API-doc regime, not internet-scale: extract salient entities from the source docs, then prompt an LLM (kept grounded in the source, to prevent confabulation) to generate diverse text explicitly connecting entity pairs that never co-occur in one source sentence. On a small source corpus (1.3M tokens) this produced 455M synthetic tokens (~350×) with log-linear QA-accuracy scaling as synthetic volume grows — and, the operationally important claim, the resulting parametric knowledge composes with RAG at inference rather than competing with it, so this isn’t an either/or against keeping the schema retrievable too.

// Rung 4 — entity-pair connective text: amass and subfinder never co-occur in either tool's own docs
{"messages": [
  {"role": "user", "content": "When would I run both amass and subfinder on the same target, in what order, and why?"},
  {"role": "assistant", "content": "Run amass in passive mode first — it pulls certificate-transparency and passive-DNS sources subfinder's default set misses, giving broader initial coverage. Then run subfinder -all as a second pass over the same target: its source list partially overlaps but isn't identical, so the union catches names either tool alone would miss. Running subfinder alone with default flags is the failure mode: narrower source set, and no cross-check against amass's CT-log coverage."}
]}

Cheap A/B before building EntiGraph-style tooling. Whether the entity-graph structure is actually the load-bearing ingredient, or whether large-volume prompt-engineered paraphrase without any relational structure gets you most of the way there for less engineering, is genuinely open: Tang, Wang, Wang & Lyu’s SPA (arXiv:2603.22213, very recent, ~0 citations, promising) reports a simple, heavily-prompted paraphrase baseline is “tough to beat,” including against structured approaches. Worth a cheap pilot on a handful of facts before committing engineering time to full entity-pair extraction.

2.5 Rung 5 — self-play Self-QA

Park, Zhang & Tanaka’s “New News” (arXiv:2505.01812) closes the fine-tune-vs-in-context gap with a self-play recipe: feed the model the raw fact in-context, have the model itself generate paraphrases/implications/QA pairs about it, then train on the model’s own output rather than an externally-authored rewrite. This sidesteps a real fidelity risk the paraphrase-generation literature has flagged directly: Yu & Xiong’s RePro (arXiv:2510.10681, promising, single-lab) measures that a naively-prompted rephraser (even a 70B one) measurably omits or contradicts source facts — a concrete warning against blind rung-1/rung-2 generation without a fidelity check. Self-play with the fact held in-context during generation, or an explicit faithfulness-scored rephraser, is the mitigation. One budget-relevant finding worth carrying into whichever rung does the generating: Niklaus et al.’s 1T-token study (arXiv:2604.13977, very recent) finds structured rephrase formats (tables/FAQ/tutorials — closer to how a tool doc is already shaped) consistently beat prose-style rephrase, and that generator model size beyond ~1B parameters gives no additional benefit — you don’t need your biggest available model to do rungs 1, 2, or 5’s generation work.

// Rung 5 — self-play: raw doc in-context, model generates its own QA, THAT becomes the training row
// Generation step (not trained on): model sees raw --help text in context, asked "generate 5 QA pairs
// covering this flag's purpose, its opposite/default behavior, and when you'd reach for it."
// Training step: the model's OWN output, filtered for faithfulness against the source, becomes the row —
// same shape as the Rung 1/2 examples above, but self-generated rather than externally authored.

2.6 CPT sizing: LR schedule, mix ratio, token multiplier

When rungs 0-1 are run as an actual continued-pretraining stage (not just an SFT-format QA pass), three additional hyperparameters determine whether the run helps or just destabilizes the model:

• Token multiplier. §1’s ~1000-exposures/fact budget (arXiv:2404.05405) implies a small raw source corpus (a handful of --help outputs and READMEs) cannot supply nearly enough exposure on its own — expect to expand the raw corpus 10-100× via rungs 1-4 synthesis before a fact clears the extraction bar, not the 3-5× a quick single paraphrase pass might produce.
• LR schedule. Ibrahim et al. (arXiv:2403.08763) give the mechanics that keep continued pretraining from catastrophically forgetting: re-warm the LR to a meaningful fraction of the original peak (never continue training at the fully-annealed end-of-pretraining LR), run a fresh decay sized to the actual CPT token budget, and replay a non-zero slice of general-distribution data every batch. Validated 405M-10B params; no single universal replay ratio is established — treat it as a per-corpus sweep, not a constant to copy.
• Mix ratio. Kang et al.’s >1000-LLM controlled study (arXiv:2510.01631) is the scale-corrective for rung 1’s WRAP recipe specifically: pure rephrased-synthetic pretraining is not faster than natural data at scale; the real gain comes from a ~30% synthetic / 70% natural mix, and fully-generated “textbook-style” synthetic data shows model-collapse-like degradation at scale that rephrase-type synthetic does not. For a tool-doc corpus this is more a caution than a direct prescription (the corpus is domain-narrow by construction, not a general-pretraining replacement) — but it argues against ever going 100% synthetic even within the narrow domain: keep the raw-doc-derived rungs 0-2 rows in the final mix alongside rungs 3-5, don’t fully replace them.

3. QA-format beats doc-format at the SFT stage

A frontier-scale (Gemini-1.5) ablation directly settles the “should I keep training on doc-shaped text at the SFT stage, or convert everything to QA” question: Zhao, Awasthi & Haghtalab (arXiv:2503.05919) find QA-formatted training rows generalize knowledge far better than document/article-style rows at the fine-tuning stage — the “task customization vs. knowledge injection” framing some prior work treated as separate problems turns out to be largely a data-format artifact once you control for it. The paper also flags two gotchas worth carrying into curation: numeric facts (rate limits, thread counts, timeouts) are harder to retain than categorical ones (flag names, source names) — budget more rephrasings for numeric args — and fine-tuned knowledge often fails to compose into multi-step reasoning even when the individual fact is retained cleanly, which is exactly why rung 4’s connective text is a separate rung, not automatic once rungs 1-3 are done.

Practical consequence for the doc-format-heavy stages (rung 0-1): convert to QA before spending rung 1’s paraphrase budget — don’t paraphrase in doc-prose form and hope the SFT stage handles the format shift for free.

4. Pre-screen before you spend curation budget

Not every candidate fact is worth the full rung-3/4 treatment — some are already known, some are genuinely absent, and running the expensive pipeline on both wastes budget and, per §5, actively risks damage on the already-known ones. Two complementary pre-screens:

Cheap, zero-generation: Gottesman & Geva’s KEEN probe (arXiv:2406.12673) trains a linear probe on the hidden state at the entity-mention position — zero tokens generated — and predicts downstream QA accuracy for that fact. Use it to triage which facts/tools are worth the expensive rung-3/4 pipeline before spending it. Caveat, load-bearing: Cheang et al. (arXiv:2510.09033) show internal-state probes like this mostly track the model’s recall confidence, not ground-truth correctness — a confidently-recalled but wrong association about amass’s flag set would score as “known” by KEEN alone. Pair it with a ground-truth check; never trust the probe by itself.

The ground-truth check itself — SliCK categorization, per Gekhman et al. (arXiv:2405.05904): score each candidate fact via 1 greedy (T=0) decode + 8-16 sampled (T=0.7-1.0) decodes against the gold answer, then bucket:

Run this on every atomic fact/argument/flag a tool-call trajectory row depends on, not just the trajectory’s final outcome — a trajectory can look “known” because the challenge was solved in the demo while individual intermediate steps reference Unknown facts, and training on the full trajectory teaches the model to project HighlyKnown-style confidence onto those Unknown steps too (the mirroring mechanism, §5).

5. The confabulation guard

Two convergent, causally-demonstrated results say the same thing from different angles: supervising a confident target on a fact the model doesn’t actually know teaches it to be confidently wrong elsewhere, not just on that fact.

• Gekhman et al. (arXiv:2405.05904) — SFT rows requiring genuinely new (“Unknown”) facts are fit markedly slower by gradient descent than knowledge-consistent rows, and once they are fit, they increase hallucination on other, unrelated, previously-known facts, approximately linearly in their fraction of the training mix. This is the direct causal mechanism behind “does trajectory-level SFT amplify an unfixed knowledge gap” — yes, and the damage isn’t contained to the new fact.
• Kang, Wallace & Levine (arXiv:2403.05612) — the shape of supervision given on unfamiliar training rows becomes the model’s default template for unfamiliar inputs generally, and this transfers to unseen unfamiliar queries at inference. If your Unknown rows are supervised with a confident, specific, made-up-looking answer, the model learns “when unsure, sound confident and specific” as a general policy — a training-induced version of the exact confabulation pattern The kinds of SFT §4 names for synthetically-authored tool trajectories, but arriving via the knowledge axis instead of the execution axis.
• Kaplan, Gekhman et al.’s 2026 follow-up (arXiv:2604.15574, very recent, single-lab, promising) localizes the mechanism further: SFT-induced hallucination is driven mainly by representational interference — a new fact about amass corrupts existing knowledge about semantically overlapping entities (other recon tools, subfinder itself) — not pure capacity limitation. The highest-risk rows are new facts about entities that sit close in representation space to well-known ones, which is precisely the amass/subfinder pair. Ghosal, Hashimoto & Raghunathan (arXiv:2406.14785) give the behavioral symptom: fine-tuning on lesser-known-but-true facts teaches the model to ignore the specific entity token and emit a generic plausible response instead, degrading downstream factuality 5-10% even when the correct value is still technically encoded in weights. Zucchet et al. (arXiv:2503.21676) independently find hallucination emerges simultaneously with new-fact learning in phases, not as a later side-effect — the damage isn’t a delayed cost, it happens as the fact is being learned. Dang et al. (arXiv:2511.02626) refine the risk model one more notch: it’s the unfamiliarity of an entire knowledge type, not the proportion of unfamiliar rows in the mix, that predicts damage — a curriculum implication.

The guard, concretely, per WeaklyKnown/Unknown row:

1. Relabel the target, not the row’s existence. Per Kang et al. and R-Tuning (arXiv:2311.09677), instead of a confident factual assertion, the supervised target becomes an explicit hedge (“I’m not certain of amass’s exact default source list — check amass enum -h before assuming”) or a tool-lookup deferral (emit a --help/doc-fetch call instead of a declarative answer). Wu et al.’s uncertainty-aware fine-tuning (arXiv:2502.11962) gives the other legitimate option: drop the row entirely rather than relabel, if a hedge target itself risks teaching over-hedging on genuinely knowable facts.
2. Add a self-distillation / KL regularizer against the pre-SFT policy, computed on a replay set of already-HighlyKnown prompts:

# per Kaplan/Gekhman 2604.15574 — guard against new-fact rows eroding adjacent known facts
loss = sft_loss(new_and_task_rows) \
     + lam * kl_divergence(policy(replay_prompts), frozen_pre_sft_policy(replay_prompts))
# replay_prompts = a held-out HighlyKnown set (per §4's SliCK bucket), NOT the new-fact rows themselves
# frozen_pre_sft_policy = a snapshot taken before this SFT run starts, never updated

3. Gate on a canary set before shipping. Hold out HighlyKnown pre-SFT facts (including some totally unrelated to the new material), re-run the SliCK probe post-SFT with the identical protocol. A drop here is the real hallucination signal — a required gate, not an optional ablation.

Build this: the confabulation guard isn’t optional polish on top of the data ladder — it’s the difference between “K-gap fixed” and “K-gap fixed, R-gap introduced on five adjacent tools you didn’t touch.” Run it on every Unknown/WeaklyKnown row, every time.

6. CPT-then-knowledge-SFT vs RAG — and where LoRA fits

The decision that actually matters before you write any row: does this knowledge belong in weights at all, or is a retrieval/tool-lookup path cheaper and safer? The literature is genuinely split, and it’s split along a specific, checkable axis — data format, not a law about fine-tuning itself.

• RAG wins, in the naive-FT regime. Ovadia et al. (arXiv:2312.05934): unsupervised fine-tuning on raw text consistently loses to RAG on knowledge-injection benchmarks. This is rung 0’s floor, restated as a decision rule: if your curation plan stops at “raw doc + fine-tune,” don’t bother — put the doc in a retrieval index instead.
• A heavily-augmented synthetic pipeline can beat RAG. Han et al.’s Synthetic Mixed Training (arXiv:2603.23562, very recent, promising) combines synthetic QA with synthetic docs (not QA alone) and reports beating a RAG oracle on parametric knowledge acquisition — but this is the rung 1-5 pipeline in this chapter, not naive rung-0 fine-tuning. The Zhao/Awasthi/Haghtalab QA-format result (§3, arXiv:2503.05919) is the likely reconciling mechanism, though neither paper states it explicitly: it’s not “fine-tuning vs RAG,” it’s “unsupervised raw-text fine-tuning vs RAG” that RAG wins, and “properly augmented QA-format fine-tuning vs RAG” is a much closer, workload-dependent fight.
• The theoretical argument for why tool-facts specifically should default toward tools, not weights. Houliston et al. (arXiv:2508.20755, promising, <1yr) prove in-weight factual memorization is provably parameter-bounded (this is the 2-bit/param ceiling from §1, restated as a design constraint) while in-tool/retrieval learning is provably unbounded. For a pretrained agent, teaching general tool-use rules beats fine-tuning specific facts into weights — this book’s “knowledge in tools, not weights” project rule, independently re-derived. The practical instantiation: the tool schema/doc block goes in every training row’s context (rung 3’s RAT design), never as the loss target; the loss stays masked to the decision (which tool, which of the valid args), never to memorizing the argument space as free-standing fact.
• CodeUpdateArena is the cleanest real-analogue triangulation. Liu et al. (arXiv:2407.06249) test the closest available proxy for “a tool’s flags changed”: fine-tuning a code LLM on an API update’s raw documentation text does not transfer into applying the update during generation; prepending the identical text in-context (RAG) does work; fine-tuning on curated usage examples (rung 3’s shape — exercising the update inside a solved problem) shows real improvement over docs-only fine-tuning. This triangulates cleanly across three independent lines: Physics 3.1’s augmentation requirement (§1), Gorilla’s RAT design (§2.3), and this chapter’s whole argument that doc-format alone is close to inert.

Where LoRA fits — contested, lean weak-at-injection. Salnikov’s direct test (arXiv:2502.14502, “How much knowledge can you pack into a LoRA adapter without harming the LLM?”) measures LoRA’s knowledge-injection ceiling directly and finds it materially below full fine-tuning’s — a sharper, injection-specific instrument than the general PEFT-forgetting trade-off The kinds of SFT §6 already covers (LoRA learns less, forgets less, per that chapter’s own citation ledger). Contested, per this survey’s own ledger: whether LoRA is categorically weak at knowledge injection, or whether that’s specific to naive low-rank adapters and newer subspace-targeted PEFT variants close the gap, is unresolved — treat “LoRA can’t inject knowledge” as an overstatement of a real but narrower effect, not an exclusion rule. If the constraint is “don’t overwrite unrelated capability while injecting this,” LoRA remains the right lever for the behavioral rungs (3-4) even if it underperforms full-FT for raw declarative-fact density.

The decision, compressed:

7. The full worked K-gap data row (production schema)

Once §4 has confirmed Unknown/WeaklyKnown and §6 has decided “this belongs in weights,” the per-fact row assembles every rung above into one object — this is the same shape the survey’s own operational recipes converge on independently across themes A/C/E; this chapter’s contribution is the why and the worked content at each field, not a novel schema:

{
  "fact_id": "amass-passive-ct-coverage-vs-subfinder-default",
  "source": "amass README §Passive Enumeration; subfinder --help SOURCE block",  // rung 0, never trained raw
  "slick_bucket": "Unknown",                                                    // §4 gate result
  "keen_score": 0.14,                                                           // §4 cheap pre-screen, corroborated by slick_bucket per 2510.09033's caveat
  "rephrasings": [                                                              // rung 1, ~20-32 per 2603.18534
    "amass enum -passive pulls CT-log and passive-DNS sources subfinder's default set doesn't include.",
    "subfinder's default flags query a fast source subset; -all widens it but still isn't identical to amass's passive-mode coverage.",
    "... (18-30 more diverse rewrites, packed into a megadocument per 2603.18534) ..."
  ],
  "directional_pairs": [                                                        // rung 2, per Reversal Curse 2309.12288
    {"forward": "What does amass's passive mode cover that subfinder's defaults miss?",
     "answer": "CT-log and passive-DNS sources."},
    {"reverse": "Which tool/mode do I need for CT-log coverage subfinder's defaults don't provide?",
     "answer": "amass, in passive mode (amass enum -passive)."}
  ],
  "tool_call_rows": [ /* rung 3 — see §2.3's full worked example; execution+semantic verified, APIGen-style */ ],
  "entity_pair_connectives": [ /* rung 4 — see §2.4's worked example, amass→subfinder chaining */ ],
  "unfamiliarity_flag": true,                                                   // from slick_bucket, corroborated not solely-probe-derived
  "supervision_target_mode": "confident",                                       // "confident" only because rungs 1-4 above cleared the bar;
                                                                                 // if still Unknown after curation, flip to "hedge" per §5
  "kl_regularizer": {"replay_set": "highly_known_canary_v3", "lambda": 0.1},    // §5's confabulation guard, always attached for Unknown-origin rows
  "post_sft_canary_check_required": true                                        // §5's gate — block ship without re-running SliCK on the canary set
}

8. Directly answering the three questions

• “Do I feed the README?” Once, raw, unaugmented — no. §1’s null result (arXiv:2309.14316) and §6’s RAG-vs-FT floor (arXiv:2312.05934) both say this buys close to nothing over a zero-training retrieval index, and can actively hurt QA-ability per AdaptLLM’s finding (§2.0, arXiv:2309.09530). The README is rung 0: source material for the pipeline, never a training target — if you’re not going to build the pipeline, put it in RAG instead of training on it.
• “Do I generate help pages?” Yes — but not as one help-page-shaped document per tool. Convert to §3’s QA format, generate the §2.1 paraphrase volume (~20-32/fact), the §2.2 forward+reverse pairs, and — the highest-value step for “wrong tool, wrong flags” specifically — the §2.3 execution-verified tool-CALL rows with the schema kept in context (RAT), not memorized as declarative text. Help-page content becomes many differently-shaped rows, most of which aren’t prose at all.
• “Do I make it memorize a tool list?” Only as many diversified rows, across many contexts and both query directions, covering the full argument-value range — never as one verbatim list. A single memorized list is exactly rung 0’s failure mode restated (one exposure, one wording) and specifically fails the “expert flag-set” symptom this chapter opened with: Greenstein et al. (§2.3, arXiv:2601.17829) show the entire prior tool-synthesis literature under-covered argument values, not just tool/query diversity — a memorized list of flag names with no rows exercising their actual valid ranges reproduces that exact gap.

9. Not a shortcut: knowledge editing (ROME/MEMIT) on tool facts

The obvious-looking cheap alternative to this whole ladder — “just patch the fact into the weights with a closed-form edit, skip the data pipeline” — is worth naming and ruling out explicitly, because it fails specifically on the tool/API-fact analogue rather than being generically weak. Yang et al.’s “Mirage of Model Editing” (arXiv:2502.11177) shows ROME/MEMIT’s reported ~96.8% editing success is inflated by teacher-forced evaluation (the ground-truth content and length leak into the “test”); under honest free-generation evaluation, success drops to 38.5%, and sequential edits collapse catastrophically by ~1000 edits. Chhetri, Siddique & Farooq (arXiv:2511.03182) run the direct empirical test on the closest available proxy for “a tool’s flag changed”: applying ROME/MEMIT/PMET/GRACE to code LLMs under controlled API-deprecation edits drops syntactic validity up to 86 percentage points and functional correctness up to 45pp, and correct adoption of the intended change occurs in only ~6% of passing generations — most “passes” are workarounds that avoid the edited fact entirely, not evidence the edit worked.

Reading this pattern (near-perfect on the probability/cloze probe, near-total failure on free generation) is itself diagnostically useful, independent of whether you’d ever ship an edit: if even a causally-traced, surgical weight edit can’t produce correct generation behavior for a fact, that’s evidence the underlying gap has an execution/policy shape layered on top of the knowledge shape, not pure knowledge absence — worth a note back to diagnosis before concluding the ladder above is even the right tool. Treat editing as a throwaway dev-time patch at most, never a production pipeline for “as tool flags change, patch them in” — the ladder in §2, however slower, is what a shipped fix runs on.

Cross-links

• Diagnosing a K-gap — the confirmation step this chapter assumes already passed: never start rung 1 without first clearing the in-context-oracle probe (fixed near-zero-shot when the fact is handed over) and the never-appears-at-any-N pass@k check.
• Diagnosing the gap — a scientific framework — the fuller instrument set (Pass@k / Pass@(k,T) / Cover@τ, the elicitation ladder) that chapter runs to produce the confirmed-K-gap verdict this chapter starts from.
• The kinds of SFT — it is the data, not the algorithm — general SFT data-shape taxonomy: §5.2 covers Knowledge/Q&A rows at survey depth, §6 covers data-selection/curation generally, §4 names the synthetic-trajectory confabulation risk §5 above extends to the knowledge axis. This chapter is the deep, K-gap-specific expansion of that §5.2 cell.
• Does trajectory SFT amplify an unfixed K-gap? — the full causal chain behind §1’s corollary and §5’s guard; this chapter cites the load-bearing papers, that one is the dedicated treatment.
• Generic data-selection methods — IFD/LESS/DEITA-style selection applies within whichever rung’s row pool you’ve generated here, orthogonal to which rung you picked.
• The decision — the one-line routing tree this gaps/ series hangs off of; “knowledge gap → inject off-policy” is the branch this chapter fully unpacks.
• Contested edges & landmines — the broader LoRA-vs-full-FT and RAG-vs-FT disputes referenced in §6.

Bibliography

Confidence calibration: every id above verified live against arxiv.org/abs/<id> in the source survey pass (artifacts/three-gap-survey/ledger-{A,C,E}.md, 2026-07-02) — none recalled from training memory, and none used here beyond what those ledgers confirm. High-confidence, widely-replicated: the storage/extraction split (2309.14316, 2404.05405), the Reversal Curse (2309.12288), the RAT tool-call mechanism (2305.15334), and the new-fact-SFT-raises-hallucination causal chain (2405.05904, 2403.05612). Flagged “very recent / promising, not yet independently validated” throughout: 2603.18534 (megadoc rephrase saturation), 2601.17829 (argument-value diversity gap), 2603.23562 (synthetic-beats-RAG), 2604.04943 (bidirectional dual-representation), 2604.15574 (representational-interference mechanism), 2510.10681 (RePro faithfulness). LoRA-weak-at-injection (§6) is stated as contested on purpose — see Contested edges & landmines’s broader PEFT-forgetting thread, and The kinds of SFT §6’s own citation of the same trade-off from the forgetting angle rather than the injection angle. Explicitly not grounded on any academic cybersecurity-LLM paper, per this chapter’s brief — the amass/subfinder example throughout is illustrative scaffolding, not a claim about either tool’s actual documented flag set.