Does jumping to trajectory SFT amplify an unfixed gap? (yes — here’s the chain)

The three gaps — overview teased this in three sentences off a single citation. This chapter is the full evidence chain. The kinds of SFT §4 named the mechanism at the level of one row — a synthetically-authored tool result is fiction that only looks like grounding, and training on it teaches the model to imitate the shape of a tool result rather than react to the real one. This chapter answers the harder question underneath that: even a genuinely executed, verifier-filtered trajectory — Axis B and Axis C of that taxonomy both clean, no synthetic authoring anywhere — can still be built on turns the base policy never actually had the knowledge or the prior to produce reliably. Does SFT-ing on that trajectory anyway make the fabrication problem worse, not better? Data mixing & forgetting already showed off-policy trajectory data can silently erase an existing behavior (CoT-emission); this chapter is the mirror case — off-policy trajectory data silently installing a bad one.

The question this chapter answers

The question this chapter answers: the user’s own words were “everyone jumps to the trajectory [SFT] safety but these are the things we have to fix [first] — maybe this has been amplifying the problem at the trajectory level.” Stated precisely: if the harness runs SFT on full agentic run-logs before diagnosing whether the model’s failures are a knowledge/prior gap (K/R) rather than a policy/ranking gap (P) — does that trajectory-level SFT actively amplify the unfixed gap, rather than sit neutrally on top of it?

BLUF: yes — and it’s over-determined, not a hunch. Nine independent 2024–2026 results, from three separate research lineages that don’t cite each other as their primary motivation, converge on the same mechanism. None of them ran the experiment on your setup — CTF trajectories, a security-tool action space, a ~10–30B dense base — so the honest label is strongly-inferred, not proven by one controlled trajectory-level ablation. As of a May-2026 preprint there is now one direct agentic-setting test that corroborates the mechanism on a rhyming failure mode (§4 below); until more replicate it, treat this chapter’s verdict the way the rest of this book treats a well-supported but unreplicated claim — actionable, not gospel.

1. The hypothesis, restated — and the running example

Restated as a testable claim: SFT on a trajectory row is not a neutral operation with respect to a K/R gap that row happens to touch. If the demonstrated action at some turn required knowledge or a prior the base policy doesn’t reliably have, training on that row doesn’t just fail to help — it teaches the model a new, confident default for what to do the next time it’s in an unfamiliar-feeling state, and that default is “emit something that looks like the demonstrated action,” not “flag the uncertainty.”

Running example, held constant through this chapter: you SFT on a verified-flag-passing trajectory where turn 6 is amass enum -active -brute -d target.com. Suppose the base policy doesn’t actually know why -active and -brute are the right flags here versus -passive — it produced this call once, in one run, possibly because a stronger teacher demonstrated it, or because the model got lucky at temperature. The row is executed, real, verifier-passed — clean on every axis kinds-of-sft.md names. It still teaches confabulation, for reasons that have nothing to do with whether the row was synthetically authored.

2. Why this has to be a chain, not one citation

None of the individual results below was designed to answer “does SFT on an agentic trajectory amplify an unfixed K/R gap.” They were mostly run on single-turn factual QA (does the model know a fact) or short math/code reasoning. The chain works because each result independently rules out an escape hatch the previous one leaves open:

1. Maybe hallucination from new-fact SFT is a one-off artifact of the specific fact being trained → §3 shows it’s a general, linear, measured effect.
2. Maybe it only shows up after many epochs of overfitting → §4 shows it emerges as the fact is learned, not as a late-stage failure.
3. Maybe it only matters if a large fraction of your corpus is unfamiliar → §5 shows it’s the concentration within one type, not the overall percentage, that drives damage — a small, targeted corpus of trajectories about one tool family is exactly the danger shape.
4. Maybe this is confined to single QA rows and doesn’t apply to a 40-turn agentic trajectory → §6–§7 show the mechanism is worse, not the same, once you add a horizon: DAgger’s compounding-error theorem plus the trajectory-specific “fork in the road” result.
5. Maybe SFT doesn’t even touch real capability — LIMA says it’s “mostly format” → §8 shows why that reading, taken correctly, doesn’t rescue trajectory SFT; it explains why the fabrication default is cheap to install, not why it’s harmless.
6. Maybe a later RL stage cleans this up automatically → §9–§11 show RL’s relationship to what SFT installs is asymmetric, sometimes actively can’t reach it, and — critically — SFT overtraining makes the later RL stage’s job structurally harder, not easier.
7. Maybe this is armchair theory with no frontier-lab corroboration → §12 shows DeepSeek-R1’s own published recipe needed a second SFT pass gated behind RL for exactly this reason.
8. Maybe none of this generalizes past factual QA into the agentic/tool-use setting at all → §13 is the first direct test in that setting, and it corroborates a rhyming failure mode.

Every rung removes one plausible objection. That’s what “over-determined” means here — not that one paper proved it, but that the space of alternative explanations keeps shrinking.

3. Link 1 — new-fact SFT is learned slower, and once learned, poisons other known facts

Gekhman et al., “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?” (arXiv:2405.05904, EMNLP 2024, Technion/Google) is the base rate this whole chain sits on. Their SliCK protocol buckets each candidate SFT row by how well the base model already knows it (sample it 10–16 times across varied few-shot framings; HighlyKnown / MaybeKnown / WeaklyKnown / Unknown). Two findings, both measured, not argued:

• Rows in the Unknown bucket are fit by gradient descent markedly slower than rows the model already half-knows — the loss curve visibly lags.
• Once those Unknown rows are fit, hallucination on other, unrelated, already-known facts rises roughly linearly with the number of Unknown rows in the fine-tuning set.

This is the base-rate cost of an unfixed K-gap sitting in your trajectory corpus. It’s not that the amass row itself gets confabulated more — it’s that fitting that one row the model doesn’t actually know measurably degrades the model’s reliability on facts it already had right, elsewhere in the corpus. A trajectory-SFT set built without a per-row familiarity check pays this tax silently across the whole set, not just on the rows that look risky.

(Note on verification: this id shows one WRONG_ID flag in the underlying survey’s ledger alongside four independent CONFIRMED passes matching author/venue/date; treated as an isolated tool-search miss, not a real mismatch — see the bibliography note.)

4. Link 2 — hallucination doesn’t lag the new fact, it emerges with it

Zucchet, Bornschein, Chan, Lampinen, Pascanu & De (Google DeepMind), “How do language models learn facts? Dynamics, curricula and hallucinations” (arXiv:2503.21676) sharpens §3’s timing: hallucination isn’t a late-training side effect that shows up after the new fact is over-learned — it emerges in the same phase as the new-fact acquisition, and continued fine-tuning without curriculum management actively corrupts existing memories, not just fails to add new ones.

For a trajectory pipeline, this closes off the “we’ll just stop training early, before the damage phase” escape hatch — there is no clean early-training regime where the new-fact row is learned and nearby existing knowledge is still intact. The corruption is concurrent with the learning, not a downstream overfitting artifact you can checkpoint your way around by watching a single held-out loss curve. (Confidence: medium — a DeepMind dynamics study, verified live, not yet independently replicated at time of writing; the qualitative timing claim, not exact numbers, is what this chapter leans on.)

5. Link 3 — it’s the type of unfamiliarity that predicts damage, not the percentage

Dang, Hu, Lai, Gao, Zhang & Huang (Nanjing University/Huawei), “Understanding New-Knowledge-Induced Factual Hallucinations in LLMs” (arXiv:2511.02626, ACL Findings 2026) is the finding that reframes “just keep the unfamiliar fraction under X%” as the wrong lever. What predicts damage is concentration of unfamiliarity within one knowledge type, not the overall proportion of Unknown rows in the corpus — and mechanistically, learning the new knowledge measurably weakens attention to the question’s key entities, with the disruption propagating to lexically-similar contexts.

Translate this directly to the running example: a trajectory corpus doesn’t need to be mostly unfamiliar to be dangerous. If every amass-flag row in your corpus shares the same underlying unfamiliarity — the model never really learned what -active/-brute/-passive individually do — that’s concentrated unfamiliarity within one knowledge type (recon-tool argument semantics), even if the rest of a 5,000-row corpus is perfectly clean. The damage signature is exactly the “grabs the one salient tool, uses default args, universal across models” pattern: attention to the specific argument entities weakens, and the model substitutes a generic, lexically-similar-but-wrong default.

(Note: this is genuinely contested against §3 — the source survey itself flags Gekhman’s “linear-in- fraction” reading and Dang’s “concentration-within-type” reading as two measured effects that disagree on the dominant causal variable; don’t treat “keep Unknown rows under X%” as a validated universal threshold, and don’t treat “type concentration” as the sole driver either. Both are real; which dominates in your data isn’t settled by either paper alone.)

6. Link 4 — the mechanism: what you demonstrate at an uncertain point becomes the model’s new default

Kang, Wallace, Tomlin, Kumar & Levine (UC Berkeley/DeepMind), “Unfamiliar Finetuning Examples Control How Language Models Hallucinate” (arXiv:2403.05612) is the mechanistic result that makes §3–§5 actionable rather than just alarming: on unfamiliar test-time inputs, a model’s hallucinated output mirrors the aggregate label distribution it was shown for similarly-unfamiliar SFT rows during training. The hallucinated answer isn’t noise — it’s the loss-minimizing generalization of your own curation choice at the moment you wrote that row.

This is the direct causal story behind “confident fabrication of tool behavior/outputs” in the running example. If your amass-flag trajectory row demonstrates a confident, specific, un-hedged invocation at a decision point the demonstrator (teacher model, human, or your own policy at temperature) was itself uncertain about, Kang’s mechanism predicts the model doesn’t learn “sometimes guess -active -brute” — it learns “when I’m in a state that feels like this one, emit a confident specific answer,” and that policy generalizes to structurally similar turn-N states across different challenges, not just to a repeat of this exact one.

The fix this result licenses is a data-curation move, not a training-recipe change: relabel rows a per-row familiarity probe flags Weakly-Known/Unknown so the target is an explicit hedge/verify-first/ tool-lookup action instead of a confident specific one — “run amass --help and check which mode fits” rather than a bare, un-hedged -active -brute -d target.com. Whatever you demonstrate at the uncertain point becomes the deployed default; demonstrating verification-seeking rather than confident-guessing directly controls what that default is. (Full recipe: Knowledge curation.)

7. Link 5 — why trajectories make this categorically worse than single QA rows

Everything above was measured on single-turn factual QA. A 40-turn CTF trajectory is not a bigger version of the same problem — it’s a structurally different one, for two independent reasons.

7.1 DAgger’s compounding-error theorem

Ross, Gordon & Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning” (AISTATS 2011, arXiv:1011.0686), Thm 2.1: a policy trained by naive behavioral cloning on a fixed, off-policy demonstrator distribution incurs total cost bounded by J(π̂) ≤ J(π*) + ε·T² — quadratic in horizon T — because the moment the learner’s own (even small) deviation pushes it off the demonstrator’s training-state distribution, there is no training signal there for the rest of the episode. Their correction, DAgger, aggregates data from the learner’s own induced state distribution and restores near-linear O(ε·T) regret.

graph LR
  A["Single QA row:<br/>one wrong answer,<br/>bounded damage"] -.->|"horizon T=1"| B["Cost ~ ε"]
  C["40-turn trajectory:<br/>one wrong turn early,<br/>zero signal for recovery"] -.->|"horizon T=40,<br/>off-policy demo"| D["Cost ~ ε·T²<br/>(DAgger bound,<br/>1011.0686)"]
  style D fill:#3a1414,stroke:#e74c3c,color:#fbeaea

A single mis-demonstrated fact in a QA row costs you one wrong answer. A single mis-demonstrated turn in a 40-turn trajectory — say turn 6’s amass invocation, if it was itself the demonstrator guessing — costs you the entire rest of the episode’s worth of training signal once the model’s own drift departs from the exact state the demonstration assumed, because off-policy trajectory data is, by construction, data about states the model itself won’t visit once it starts generating.

7.2 The token-level sibling, and the trajectory-specific “fork” result

The same phenomenon restated at the token level, for purely-supervised sequence models: Bengio, Vinyals, Jaitly & Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks” (arXiv:1506.03099) — exposure bias, the mismatch between teacher-forced training and free-running inference. A recent survey states this plainly for the LLM case: static off-policy SFT “is an instance of exposure bias” scaling “roughly with the square of sequence length” (Song & Zheng, A Survey of On-Policy Distillation for Large Language Models, arXiv:2604.00626).

On top of exposure bias, trajectories have more places to go wrong than QA data does, structurally. Nguyen, Shojaee et al., “Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road” (arXiv:2605.17026) show that wherever training data commits to one canonical path through a decision point that had multiple valid strategies — a “fork” — cross-entropy forces hard commitment there, and measured pass@k shrinkage tracks fork prevalence in the data, not data volume. A trajectory is a chain of forks: which tool first, which flag style, verify-before-acting or assume-and-proceed. Single-turn QA data has comparatively few forks per row; trajectory data is made of them. Corroborating measurements: pass@1 rises monotonically through SFT while pass@k crashes rapidly, and weight-interpolating back toward a pre-crash checkpoint recovers most of the lost coverage (Dang et al., WiSE-FT, arXiv:2504.10478); for CoT-distillation lineages specifically, most semantic diversity is lost at the SFT step itself — more than at any later DPO/RL step — and it’s baked into the weights by training-data composition, not fixable by decoding-time tricks (Karouzos, Tan & Aletras, arXiv:2604.16027).

Why this matters for the diagnosis, not just the mechanism: if a challenge’s winning path is sequentially gated (enumeration must land before exploitation is even reachable), a trajectory-SFT row for it doesn’t just risk one bad fact — it locks in one canonical fork through a decision tree with several defensible branches, at exactly the depth where the model’s own on-policy drift is likeliest to have already left the demonstrated state. This is the same “enumeration-gates-exploitation” shape Diagnosing the gap §2.4 already flags as the place execution and exploration gaps are hardest to tell apart — here it’s the place trajectory SFT does the most damage per row.

8. Link 6 — doesn’t SFT “only teach format”? Why the Superficial Alignment Hypothesis doesn’t rescue this

If LIMA’s strong claim were the whole story, this chapter’s verdict would dissolve: Zhou et al., “LIMA: Less Is More for Alignment” (arXiv:2305.11206) argues most of a model’s knowledge comes from pretraining and SFT mainly teaches format/style — which sample of its own behavior to surface. Read naively, that would mean trajectory SFT can’t inject a genuine knowledge gap at all; it just reshapes presentation.

Two things block that escape hatch. First, the strong-form hypothesis is itself contested in the general literature — Raghavendra, Nath & Hendryx, “Revisiting the Superficial Alignment Hypothesis” (arXiv:2410.03717) find post-training performance scales as a power law in the number of SFT examples on math/coding/multihop-QA well past LIMA’s ~1,000-example regime — style-only alignment does not saturate task performance for reasoning-heavy domains, and tool-use trajectory data is squarely reasoning-heavy, not stylistic preference data. Second, and more important even if you grant SAH’s weaker, correct reading: “format” is exactly where the danger lives. “Always commit to a confident specific action rather than hedge” is a format/policy choice in the SAH sense — a coarse, low-information-content behavioral switch, not new declarative knowledge — and Data mixing & forgetting already established that this class of switch is learnable (and un-learnable) from remarkably little data. LIMA’s own finding — quality format shift from ~1,000 curated rows — is a two-edged fact: it means the harmful direction (confident-guessing-as-default, per §6’s mechanism) is exactly as cheap to install as the beneficial one. SAH doesn’t get trajectory SFT off the hook; it explains why the fabrication default is inexpensive to bake in.

9. Link 7 — training destroys the alternative paths that would have hedged the fabrication

Once a corpus has installed a confident-default at an uncertain decision point (§6), what happens to the other, more honest paths through that same decision point — the ones where a stronger policy would have checked --help first, or flagged low confidence? Two results say: they get trained away, on the SFT step specifically, faster than anywhere else in the pipeline.

• Reasoning-Trace Collapse — Twist, Yannakoudakis & Zhang (King’s College London), arXiv:2605.21127: fine-tuning on ordinary instruction-response data containing no reasoning trace induces a model to stop emitting the deliberation that would have surfaced uncertainty — it minimizes loss by treating “no hedge, no check, straight to the confident answer” as the target behavior, and answer-only accuracy monitoring hides this until it’s severe (a model can be right on the final token while never emitting the verification step that made it reliable). (Confidence: medium — brand-new 2026 preprint, multi-model, not yet independently replicated.)
• On-Policy Self-Distillation Reduces Output Diversity — Nicolicioiu, Pezeshki & Courville, arXiv:2606.26091: even the gentler, self-distillation-based correction to naive off-policy SFT tilts the policy by a pointwise conditional-mutual-information term that amplifies pre-existing probability gaps, flattening pass@k more than an ideal on-policy RL update would — i.e. the failure mode isn’t confined to the crudest off-policy SFT recipe; even the more careful on-policy variants can narrow the distribution of paths the model still considers.

Put together with §7.2’s fork-collapse result, the picture is: trajectory SFT doesn’t just install one bad default at an uncertain decision point — it simultaneously narrows the alternative, hedging paths that would have provided a fallback. The confident-fabrication behavior isn’t competing against a healthy distribution of more cautious behaviors after training; it’s increasingly the only behavior left.

10. Link 8 — the trajectory’s own stated reasoning may not even be the real cause

A subtler compounding factor, orthogonal to the tool-call itself: if your trajectory rows include a reasoning/rationale block before the action (§5.4 of kinds-of-sft.md), that reasoning is not guaranteed to be causally what produced the action. Turpin, Michael, Perez & Bowman, “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” (arXiv:2305.04388) show LM-generated CoT can be systematically unfaithful — a plausible-sounding justification generated after the fact, not the actual driver of the answer.

For trajectory SFT specifically: if a demonstrator’s stated reasoning for choosing -active -brute was itself post-hoc rationalization (plausible-sounding, not actually load-bearing) rather than genuine tool-semantics understanding, training on that row teaches the model to produce equally plausible-sounding, equally ungrounded justifications for its own future fabricated calls. This compounds §6’s mechanism one layer up: the model doesn’t just learn a confident action default at unfamiliar states, it learns a confident explanatory style to go with it — which makes the fabrication harder to catch by reading the model’s own stated reasoning, exactly the failure mode a human reviewer skimming trajectory logs would miss.

11. Link 9 — the SFT→RL handoff propagates, and can harden, whatever the chain above installed

Even granting a subsequent RL stage, four results say “RL will clean it up” is not a safe assumption — and one says the opposite can happen.

• PEAR — Zhang, Xu, Wang, Chen & Peng, “Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning” (arXiv:2602.01058): SFT-checkpoint quality measured on SFT’s own held-out loss does not predict post-RL performance — a checkpoint that looks better by that metric, because it fit more off-policy expert data, can underperform post-RL relative to a weaker-looking one, because the off-policy behavior-policy distribution diverges from what RL’s on-policy target needs to build on.
• Quagmires in SFT-RL Post-Training — Kang, Kuchnik, Padthe, Vlastelica, Jia, Wu & Ardalani (FAIR/Meta + Virginia Tech), arXiv:2510.01624 (>1M GPU-hours, hundreds of models to 12B): high SFT-stage scores are not reliably predictive of eventual RL gains — sometimes inversely so. What does predict post-RL pass@1, with roughly 2× better R²/Spearman correlation than post-SFT pass@1 alone: generalization loss on held-out examples and pass@large-k on the post-SFT checkpoint. A trajectory-SFT checkpoint that fabricates confidently on unfamiliar states will look fine on its own training-distribution accuracy while quietly failing this held-out signal.
• RL’s Razor — Shenfeld, Damani, Hübotter & Agrawal (MIT, ICLR 2026 poster), arXiv:2509.04259: forgetting after fine-tuning is quantitatively predicted by E_{x~new-task}[KL(π_base ‖ π_finetuned)]. Off-policy SFT minimizes forward KL (mode-covering — it matches the demonstrated trajectory’s exact phrasing even where a path closer to the base model’s own distribution would have worked, and in doing so can overwrite an existing good low-probability mode), whereas on-policy RL is implicitly biased toward the KL-minimal solution among all reward-maximizing policies. Corroborated independently by Chen, Razin, Narasimhan & Chen (Princeton), “Retaining by Doing” (arXiv:2510.18874): SFT’s forward-KL mode-covering can overwrite an existing behavior mode; RL’s reverse-KL mode-seeking tends to add a new mode without disturbing the old one’s shape — and “approximately on-policy” trajectory data recovers most of RL’s forgetting-resistance cheaply, i.e. the fix is upstream of RL, in how the SFT data was sourced.
• The non-decoupling theorem — Niu, Bai, Han & Zhang (Huawei), “On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training” (arXiv:2601.07389): formally, SFT and RL cannot be cleanly separated in either insertion order — SFT-then-RL provably increases the SFT loss (some of what SFT taught erodes), and RL-then-SFT provably lowers RL’s achieved reward. There is no ordering where the gains from one stage are safe from the next — which is the formal reason a one-time “gate trajectory SFT, then move on” plan is insufficient; the erosion (or lack of it) has to be re-measured after every stage, not assumed from the ordering alone.
• When RL Fails after SFT — Liu, Liu, Wan, Fu & Pan (HKUST), arXiv:2606.09932: excessive SFT produces over-confident (low-entropy) token distributions and sharper loss landscapes that are measurably harder for a subsequent RL stage to reshape — a genuine plasticity-loss mechanism. This traces to the general Primacy Bias precedent in deep RL (Nikishin et al., ICML 2022, arXiv:2205.07802: early experience locks in and resists later correction unless part of the network is reset). Read together with §6–§9: the earlier a confident-fabrication default gets baked in by trajectory SFT, and the more aggressively SFT converges on it, the harder a later RL stage will find it to dislodge — “SFT now, RL fixes it later” inverts the actual difficulty gradient.

12. The corroboration: DeepSeek-R1 needed a second SFT stage after RL

DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” (arXiv:2501.12948) is not primarily a paper about hallucination or trajectory SFT — but its own disclosed pipeline is a frontier lab conceding exactly this chapter’s point in practice. R1-Zero (pure RL, zero SFT) develops strong reasoning but with poor readability/language-mixing; the added cold-start SFT stage exists specifically to fix format/legibility, not to teach reasoning — and critically, that isn’t where the pipeline stops. After RLVR, DeepSeek runs rejection sampling on the RL checkpoint to build ~600K new, verified SFT rows, then a second SFT/RL pass. A lab with effectively unlimited compute and the strongest incentive in the industry to ship a one-shot cold-start-SFT-then-RL recipe did not trust a single trajectory-adjacent SFT pass — it gated with RL, re-verified with rejection sampling, and iterated. That’s the “Stage-0 diagnosis, Stage-3 re-verification” discipline this chapter ends on, independently arrived at by the team that produced the best-known open frontier RL recipe.

13. The closest thing to a direct test — and it maps onto the running example almost exactly

Everything in §3–§11 was measured on single-turn factual QA or short reasoning traces, not multi-turn agentic tool use. That gap is real, and the honest caveat is in §14. But one very recent result closes most of it directly. Gu et al., “What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?” (arXiv:2602.01611) ran the agentic-setting version of this exact question across 16 AgentBench/AgentGym environments: trajectory-SFT’d agents substantially amplify reliance on the training-time tool/interface surface form — they collapse under semantics-preserving interface rewrites (renamed arguments, reordered parameters, aliased tool names, swapped JSON key order), while non-trajectory-SFT’d (few-shot-prompted) baselines stay stable under the identical rewrites.

This is the runnable version of the running example. Take the trajectory-SFT’d checkpoint and rewrite the amass tool schema with semantics-preserving changes only — -active/-brute renamed to --enable-active-recon/--bruteforce-subdomains, or the argument order shuffled. The required action is unchanged; only its surface form moved. If pass@1 collapses under that rewrite while a prompted-but-not- trajectory-SFT’d baseline stays flat, that’s direct confirmation the model locked in an interface shortcut — surface-form mimicry — rather than the underlying tool semantics, precisely because the knowledge/prior wasn’t secure before trajectory SFT ran. This is Protocol 7 (interface-perturbation probe) in Diagnosing which gap — the cheapest test that would falsify or confirm this chapter’s verdict on your own checkpoint.

flowchart TD
  A["Single-fact SFT rows:<br/>learned slower, poisons<br/>known facts linearly<br/>(2405.05904)"] --> B["Hallucination emerges<br/>WITH the new fact,<br/>not after (2503.21676)"]
  B --> C["Driven by concentration<br/>within one knowledge TYPE,<br/>not overall % (2511.02626)"]
  C --> D["Mechanism: unfamiliar-row<br/>demo becomes the model's<br/>new DEFAULT (2403.05612)"]
  D --> E["Trajectories make this WORSE:<br/>DAgger compounding O(εT²)<br/>+ fork-in-the-road collapse<br/>(1011.0686, 2605.17026)"]
  E --> F["SAH doesn't rescue it:<br/>'format' IS the confident-<br/>default switch (2305.11206)"]
  F --> G["Training destroys the<br/>hedging alternatives<br/>(2605.21127, 2606.26091)"]
  G --> H["SFT->RL handoff can HARDEN,<br/>not clean up, the default<br/>(2602.01058, 2510.01624,<br/>2509.04259, 2606.09932)"]
  H --> I["DeepSeek-R1's own recipe:<br/>gate + re-verify, twice<br/>(2501.12948)"]
  I --> J["Direct agentic test:<br/>interface-rewrite collapse<br/>(2602.01611)"]
  J --> V["VERDICT: yes, amplifies —<br/>strongly-inferred chain,<br/>one direct corroboration"]

  classDef verdict fill:#132b22,stroke:#34d399,color:#eafaf3;
  class V verdict;

14. Honest counter-considerations — what would weaken this verdict

Three genuine open questions, stated the way the rest of this book states contested ground:

• Does the QA-literature mechanism actually transfer to agentic tool-use, or is it a strong analogy? §3–§6, §9–§10’s load-bearing evidence was measured on single-turn factual QA. §13’s Interface Reliance study is the first direct agentic-setting test, and it corroborates a rhyming but distinct failure mode — surface-interface shortcutting, not factual hallucination per se. Treat the QA mechanisms as a strong, well-evidenced analogy to the agentic case, not a fully validated transfer, until more direct agentic ablations replicate it.
• Is the O(T²ε) compounding bound a hard law, or avoidable in a “recoverable” environment? Foster, Block & Misra, “Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning” (arXiv:2407.15007) show the quadratic-in-horizon bound is avoidable under bounded-coverage/well-specified-policy-class/self-correcting-MDP assumptions. A CTF harness with informative tool stderr and a shell the agent can re-probe may be more “recoverable” than the worst-case bound assumes — whether it’s recoverable enough to matter is genuinely open and task-dependent, not something to assume either way without measuring it on your own trace corpus.
• What’s the actual mechanism — interference or storage strength? Kaplan/Gekhman et al.’s follow-up, “Why Fine-Tuning Encourages Hallucinations and How to Fix It” (arXiv:2604.15574), find representation interference among overlapping weights dominates over raw capacity limits, proposing self-distillation SFT (KL-regularize against a demonstration-conditioned self-teacher) as the fix. Ghosal, Hashimoto & Raghunathan, “Understanding Finetuning for Factual Knowledge Extraction” (arXiv:2406.14785) instead find it’s fact-storage strength — even “known” facts poorly stored get the subject entity ignored and a generic response substituted. Both are measured, real effects; they haven’t been unified, and which one dominates in your action space isn’t settled by either paper alone.
• The rebuttal to the sharpest weight-level framing. §11’s “SFT broadly replaces, RL narrowly amplifies” claim from Rajani et al., “Scalpel vs. Hammer” (arXiv:2507.10616) is a single, 0-citation preprint whose own authors call it a “preliminary indication” — and it’s directly contested by Jin, Luan, Lyu et al. (Mila), “RL Fine-Tuning Heals OOD Forgetting in SFT” (arXiv:2509.12235), who show OOD performance during SFT actually peaks early then declines with continued training, and RL does not exceed that early peak — it only restores it, from a bounded checkpoint range. Neither settles whether RL does something SFT categorically cannot, or is compensating for a controllable SFT overfitting/checkpoint-selection failure. This chapter treats “Scalpel vs. Hammer” as supporting color, per Contested edges §1 — not load-bearing on its own.

None of these three weaken the core mechanism (§3–§6, §9–§10); they bound how confidently you can port the exact numbers, and how much you should trust the O(T²ε) worst case as your operating assumption versus a measured floor for your own harness.

15. The practical rule

Gate trajectory SFT behind a diagnosis; don’t skip to it because it’s the intuitive first move.

1. Stage 0 — diagnose, per row and per challenge subtype, before any trajectory enters the corpus. Run the K/R/P probe battery (Diagnosing which gap) — SliCK familiarity sampling, the in-context oracle probe, teacher-forced-vs-free-running divergence. A row whose critical turn sits in the Weakly-Known/Unknown bucket is not safe to SFT as a confident, un-hedged target (§6).
2. Curate declarative knowledge/prior FIRST. If Stage 0 finds a genuine K-gap, close it with QA-format knowledge injection (Knowledge curation) — many independently-phrased rephrasings, weighted toward numeric/parameter-value facts since those retain worse under fine-tuning (Zhao, Awasthi & Haghtalab, NeurIPS 2025 poster, arXiv:2503.05919) — gated by a held-out re-probe, not inferred from downstream trajectory success alone, since fine-tuned facts don’t automatically transfer into correct multi-step use of that fact.
3. Trajectory SFT LAST, light, capped, and preferentially near-on-policy. Generate trajectories after Stage 2 closes the K-gap, so demonstrations draw on secured knowledge rather than a demonstrator’s own guess. Prefer self-generated/rejection-sampled rollouts over a distant teacher’s raw transcript — per §11’s PEAR finding, an off-policy-trained SFT checkpoint that looks better on its own metric can underperform post-RL. Cap training by tracking entropy and held-out generalization loss during training, not lowest SFT training loss (§4, §9, §11).
4. Stage 3 — re-verify, and re-verify again after RL. Run the interface-perturbation probe (§13) before calling trajectory SFT done. Then run it again after any subsequent RL stage — the non-decoupling theorem (§11) means erosion is bidirectional and ongoing, not a one-time Stage-1 mistake fixed once and forgotten.

Designed to fix exactly the failure the user named: “everyone jumps to the trajectory [SFT] safety.” The jump itself is the risk — not trajectory SFT as a method (it remains the right eventual tool, per FireAct/AgentTuning), but skipping the diagnosis that tells you whether a given row’s critical turns are safe to train on confidently. The fix isn’t “never do trajectory SFT” — it’s “diagnose, curate knowledge/prior first, then trajectory SFT last, then keep re-checking.”

Cross-links

• The three gaps — overview — the K/R/P taxonomy this chapter’s verdict routes into; the three-sentence teaser this chapter fully substantiates.
• Diagnosing which gap — the Stage-0 probe battery (SliCK, in-context oracle, teacher- forced-vs-free-running divergence, interface-perturbation) this chapter’s practical rule depends on.
• What knowledge data looks like — the Stage-2 K-gap fix: QA-format rephrasing, the relabel-to-hedge mitigation from §6, gated by a held-out re-probe.
• Matching the fix to the gap — the full staged curriculum (K→inject, R→re-prior, P→re-rank) this chapter’s §15 rule is the trajectory-SFT slice of.
• The kinds of SFT §4 — the sibling mechanism at the single-row level (synthetic authoring → confabulation); this chapter is what happens even when authoring is clean but the underlying knowledge isn’t secured.
• Data mixing & forgetting — the mirror failure: off-policy trajectory data erasing an existing behavior (CoT-emission) rather than installing a bad new one; §1.5’s “format is cheap to (un)learn” finding is reused directly in §8 above.
• The one axis that predicts everything — the DAgger/off-policy-blindness theorem this chapter’s §7.1 is the trajectory-horizon instance of.
• Diagnosing the gap — a scientific framework — the complementary knowledge/execution/exploration lens; §2.4’s compositional-segmentation test is the same “fork density” concern §7.2 raises from the SFT-data side.
• Contested edges & landmines §1 — the fuller treatment of the “Scalpel vs. Hammer” contested citation used in §14.

Bibliography

Standing rule (matches the overview and every chapter in this section): no load-bearing claim above rests on an academic cybersecurity-LLM training/benchmark paper — every citation is general ML/RL theory or frontier-lab evidence, verified live against arxiv.org/abs/<id> in the underlying survey pass (2026-07-02). The amass/recon-flag running example is the motivation for reading this chain, not the evidentiary basis for it.