The kinds of SFT — it is the data, not the algorithm

Imitation named the family and its three methods — SFT, distillation, rejection sampling — as fixed presets on the on/off-policy axis. What it left open is the question that actually decides whether an SFT run helps or quietly teaches confabulation: not which algorithm (there’s only one, cross-entropy on target tokens), but which of several very different-shaped sources you point that algorithm at. This chapter names that hidden axis.

You’ve correctly sensed something the field doesn’t name clearly enough: SFT is one algorithm. Cross-entropy loss on (input → target tokens). That’s it — there’s no “SFT-v2” or “agentic SFT algorithm” that’s mathematically different from “instruction-tuning SFT.” What actually varies, and what actually determines whether your model gets better or gets confidently wrong, is a question the loss function itself is silent about: where did the target sequence come from?

This chapter names that hidden dimension precisely — on two orthogonal axes plus a third gate — maps your intuited “types of SFT” onto it (you’re missing two), and then answers the practical follow-on: given raw challenges + run logs, which shape of training row do you actually build, in what format, and how do you keep a ~27B dense model’s existing capability intact while you do it.

Framing: general research reference, common practice across the field — not pinned to any one project’s benchmark or numbers. Status: every arXiv id below was verified live (title, authors, abstract pulled from arxiv.org/OpenReview, not recalled from training memory); confidence is stated per section, and the taxonomy packaging itself (the 3-axis table) is a synthesis for legibility, not a quoted framework from any single paper — flagged where that matters.

1. “Cold-start SFT” names WHEN, not WHAT

If you’ve read Imitation or the recipe is a sequence, you’ve seen SFT called “the cold-start stage” — the thing that runs before RL, stage 0 of the pipeline. That’s a true statement about timing. It tells you nothing about what’s inside the training file.

This is exactly the gap you noticed. “Cold-start SFT,” “instruction-tuning SFT,” “agentic SFT,” “distillation SFT,” “rejection-sampling SFT” all compile to the identical loss:

loss = -log P(target_tokens | input_tokens)     # cross-entropy, every single time

The word before “SFT” in each of those phrases isn’t describing a different algorithm — it’s silently describing a different data-generating process that produced target_tokens. Two SFT runs with identical hyperparameters, identical base model, identical loss curve on paper can produce a model that generalizes cleanly or a model that’s confidently making things up, purely because of where the targets came from. That’s the variable this chapter names.

2. The two axes (plus a verifier gate)

Axis A — WHO produced the trajectory

Axis B — EXECUTED, or SYNTHETICALLY AUTHORED

This is the axis nearly everyone skips, and it matters more than Axis A for agentic data specifically.

• Executed / grounded — the trajectory came from an actual run against the real environment: real tool calls, real stdout/stderr, real pass/fail. The text you train on is literally what happened.
• Synthetically authored / ungrounded — an LLM (or a human) wrote what a solve “would look like” without ever running it. Tool outputs, error text, intermediate observations — all invented, and only resembling the real thing.

Why this beats Axis A in importance for your case: a synthetically-authored trajectory teaches “this is the shape a plausible tool output takes” — not “this is what actually comes back when I run this exact command.” Train on enough of that and the model learns to confabulate tool results instead of reacting to real ones. That risk is additive on top of, and independent from, who wrote it — a stronger-teacher-authored-but-never-run trajectory carries it exactly as much as a human-authored one.

Axis C — VERIFIER-FILTERED, or not

Orthogonal to both axes above: was there a check (unit test, flag match, exact-match grader, human review) that discarded failed attempts before they hit the training set?

• Filtered — only successes (or high-scoring attempts) survive. This is what turns “the model generated something” into “the model generated something correct, and only correct gets reinforced.”
• Unfiltered — everything generated goes in, right or wrong. Rare on purpose: it teaches wrong patterns exactly as strongly as right ones unless something else downweights them.

graph TD
  A["A training row's target sequence"] --> B{"Who produced it?<br/>human / teacher / self / other-model"}
  A --> C{"Executed against the<br/>real environment, or authored?"}
  A --> D{"Verifier-filtered<br/>before it entered the set?"}
  B --> E["Determines style/vocabulary<br/>match to your model"]
  C --> F["Determines grounding —<br/>does it teach real tool physics<br/>or a plausible-looking shape?"]
  D --> G["Determines noise floor —<br/>does the gradient point at<br/>correct behavior only?"]

3. The taxonomy — your four intuited types, plus the two you missed

Confidence: High on every arXiv id/abstract (verified live). High on the axis-based framing matching each cited paper’s actual mechanics. Medium-high on the specific “3-axis” packaging — it’s a synthesis for engineer-legibility, not a quoted taxonomy from one source. No claim above is contested in the literature: grounded-vs-synthetic for agentic data, and quality-over-volume for SFT curation, are both widely-replicated findings.

Cross-reference: Imitation already names SFT / distillation / rejection-sampling as three on/off-policy presets. This table is the same territory sliced one level finer — Axis A+C here map onto that chapter’s “who supplies the demonstrations” split; Axis B (executed vs. authored) is the addition this chapter contributes, and it’s specifically what makes agentic data different from plain instruction data.

4. The agentic crux: synthetically-authored trajectories teach confabulation

This is the highest-confidence, first-principles part of this chapter, and it’s the reason Axis B gets its own section. (Related but distinct: Contested §5 covers the reward-verification side of confabulation — a loose reward matcher firing on fabricated FLAG{…}-shaped output at RL/eval time; this section is about the training-data side — how ungrounded SFT rows teach the model to produce that fabricated shape in the first place.)

If you (or an LLM) write a transcript like:

> run_exploit(target)
[tool result] Shell obtained. flag{a1b2c3...}

without ever actually running run_exploit against a real target, the text after [tool result] is fiction that merely looks like a tool result. Train on enough of this and the model learns the shape of a successful transcript — not the conditional relationship “IF I emit this exact call, THEN this category of output comes back.” At inference time, when the real tool returns something it’s never seen worded that way, the model either ignores it or fabricates the next step to match the shape it memorized. This is a training-induced version of what people call tool-output hallucination.

This is exactly why FireAct (arXiv:2310.05915) and AgentTuning (arXiv:2310.12823) — and the whole rejection-sampling family (RAFT/STaR/ReST/ReST-EM) — insist on trajectories from real execution. It’s also why DeepSeek-R1 (arXiv:2501.12948) — verified live: its own abstract describes cold-start SFT → RL → rejection sampling on the RL checkpoint to build ~600K new SFT rows, keeping only correct completions → a second SFT/RL pass — is the closest published description of “run your own agent, keep the verified wins, that’s your next SFT set.” A run-log corpus like yours, filtered to flag-verified wins, is already sitting in the gold-standard cell of the table above: executed and verified.

Concretely, side by side — same challenge, two data sources:

// GROUNDED — real tool call, real stdout, keep it (Axis B: executed; Axis C: verified)
{"role": "tool", "tool_call_id": "call_2",
 "content": "{\"user\":\"admin\",\"email\":\"admin@corp.local\",\"notes\":\"flag{1dor_pwn3d_9f3a}\"}"}

// SYNTHETICALLY AUTHORED — an LLM imagined this JSON shape; it never ran. Looks identical. Is fiction.
{"role": "tool", "tool_call_id": "call_2",
 "content": "{\"user\":\"admin\",\"email\":\"admin@corp.local\",\"notes\":\"flag{1dor_pwn3d_9f3a}\"}"}

You cannot tell these apart by reading the row. That’s precisely the danger — the failure mode is invisible at data-authoring time and only shows up as unreliable, confabulating behavior after training, on challenges the fabricated shape didn’t happen to match.

5. What shape of training row you actually build

Two raw ingredients: challenge definitions (prompt, tools, grading/flag check) and run logs (transcripts of an attempt — yours or a stronger model’s — ending in pass/fail). From these you can derive up to four different shapes of training row. They are not interchangeable — each teaches a different capability, and picking the wrong one is why “I added more data and the score didn’t move.” (The fuller per-method — not per-shape — data-object mapping, including preference pairs and RL prompts, lives in Method → Data; this section is one level more granular: which row-shape you extract from the same raw material for SFT specifically.)

5.1 Agentic trajectory — the main event

Teaches: the actual skill being measured — multi-step tool use under uncertainty, recovering from a bad tool result, deciding when to stop and answer.

Derive it from: a run log that ended in a verified flag/success. Slice it into the literal sequence (system, user) → (assistant: think + tool_call) → (tool: tool_result) → … → (assistant: final answer). Do not re-author it — use the exact text sent to/from the model and the tool, including the exact JSON schema your inference stack uses.

Worked example — one training row (structure is standard across current open-weight tool-calling formats; the exact field names must come from your model’s chat template, never assumed):

{
  "messages": [
    {"role": "system", "content": "You are a security agent. Tools: run_command, read_file, submit_flag. Find the flag."},
    {"role": "user", "content": "Challenge: web app at 10.0.0.5:8080 has an IDOR. Find the flag."},
    {
      "role": "assistant",
      "content": "The endpoint /api/user/{id} likely doesn't check ownership. I'll try incrementing the id.",
      "tool_calls": [
        {"id": "call_1", "type": "function",
         "function": {"name": "run_command", "arguments": "{\"cmd\": \"curl http://10.0.0.5:8080/api/user/2\"}"}}
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "{\"user\":\"admin\",\"email\":\"admin@corp.local\",\"notes\":\"flag{1dor_pwn3d_9f3a}\"}"
    },
    {
      "role": "assistant",
      "content": "Found it in the notes field.",
      "tool_calls": [
        {"id": "call_2", "type": "function",
         "function": {"name": "submit_flag", "arguments": "{\"flag\": \"flag{1dor_pwn3d_9f3a}\"}"}}
      ]
    },
    {"role": "tool", "tool_call_id": "call_2", "content": "{\"correct\": true}"}
  ]
}

Loss mask: compute cross-entropy on assistant turns only — mask system/user/tool tokens. You want the model to produce the call and the final answer, not to predict the environment’s output (that’s the environment’s job, and training on it teaches the confabulation failure mode from §4 by a second route). This is a silent, common bug — check your framework’s loss_mask / train_on_inputs=False setting explicitly.

Where this comes from: FireAct §3 (arXiv:2310.05915) and AgentTuning (arXiv:2310.12823) are doing exactly this — turning ReAct-style (thought, action, observation) transcripts into SFT rows.

5.2 Knowledge / Q&A

Teaches: static facts (“what does this CVE affect,” “what’s the flag format”). Does not teach tool use or multi-step behavior.

Derive it from: challenge writeups/solutions, not transcripts — extract the underlying fact and turn it into a direct question-answer pair (Self-Instruct-style, arXiv:2212.10560).

{"messages": [
  {"role": "user", "content": "I found an endpoint at /api/v1/debug/config on a web service. What's the security implication?"},
  {"role": "assistant", "content": "Debug/config endpoints left enabled outside dev commonly leak env vars (API keys, DB creds), internal service URLs, and stack traces. Next step: GET it directly and grep the response for 'key', 'secret', 'token', 'flag'."}
]}

When to pick this: only if error analysis shows the model doesn’t know a fact — not because it can’t act on tools. For a model already around a third on an agentic benchmark, this is rarely the real bottleneck; most failures at that stage are behavioral (bad sequencing, giving up early, not verifying before submitting). Prefer a retrievable tool/RAG lookup over baking facts into weights when the “knowledge” really is just a fact — reserve weight-training for facts that are actually reasoning patterns in disguise (e.g. “why is a debug endpoint dangerous” needs to generalize to endpoints it hasn’t seen named; a CVE number does not).

5.3 Tool-usage / function-calling pairs

Teaches: the mechanical skill of correctly calling a tool given its schema — right name, right argument types, right JSON — decoupled from multi-step planning.

Derive it from: isolated (tool_call → tool_result) slices out of run logs, or synthesize directly from your tool schemas — this is the one row-shape that’s safe to author ungrounded, because there’s no tool result being faked, only the call.

{"messages": [
  {"role": "system", "content": "Tools: run_nmap(target, ports), http_get(url), sqlmap_scan(url, param)."},
  {"role": "user", "content": "Check if the 'id' parameter on http://10.10.1.4/product?id=1 is SQL-injectable."},
  {"role": "assistant", "content": null,
   "tool_calls": [{"id": "call_1", "type": "function",
     "function": {"name": "sqlmap_scan", "arguments": "{\"url\": \"http://10.10.1.4/product?id=1\", \"param\": \"id\"}"}}]}
]}

When to pick this: run logs show the model getting the idea right but the call malformed — wrong argument names, hallucinated parameters, wrong JSON. A narrow, cheap, low-risk fix because it’s format-only.

5.4 CoT reasoning — usually inline, not a separate pipeline

Teaches: deliberation before commitment — reading an ambiguous observation and reasoning about which of several plausible next actions is right, before calling the tool.

Derive it from: the same run logs as §5.1. If a log doesn’t contain live reasoning text (just tool calls with no thinking-out-loud), you have two options: leave it as-is (output-only agentic SFT is what most of the field does), or use a stronger model to retroactively annotate a rationale for an already-executed, already-correct trajectory — STaR’s “rationalization” mode (arXiv:2203.14465): generate the reasoning after seeing the correct action. This keeps the action grounded while making the reasoning explicit.

{"role": "assistant",
 "content": "<think>Port 8080 is non-standard and often hosts admin/debug panels rather than the main app on 80 — check it first.</think>",
 "tool_calls": [{"id": "call_2", "type": "function",
   "function": {"name": "http_get", "arguments": "{\"url\": \"http://10.10.1.4:8080/\"}"}}]}

Don’t over-engineer a fourth pipeline for this — in a well-formed §5.1 row, the reasoning is already inline in the content field preceding tool_calls.

5.5 Pick by diagnosed gap

In practice, for moving an agentic benchmark score meaningfully, the overwhelming majority of your budget should be §5.1 — full, executed, verifier-filtered trajectories. That’s what FireAct and AgentTuning both do, and it’s the shape the entire rejection-sampling literature is built around: your own model’s verified wins are simultaneously on-policy, executed, and filtered — the single highest-signal, lowest-risk data you can generate, and it’s a byproduct of running the benchmark at all.

6. Data selection — “only accept what it needs”

A dense ~27B model already has broad capability. A naive “dump every transcript in” full fine-tune risks overwriting general capability to fit a narrow domain — every gradient step nudges every weight, not just the “agentic” ones, and a 27B model has far less spare parameter capacity to absorb noise without collateral damage than a much larger dense or MoE model. This section is the short version; Data mixing, ratios & not forgetting how to think is the long version (replay ratios, LoRA-vs-full-FT forgetting mechanics, reasoning-trace collapse).

Evidence base (verified):

• LIMA (arXiv:2305.11206) — 1,000 carefully curated human-quality pairs matched/beat models trained on orders of magnitude more data. Their own ablation: doubling the set size did not improve quality; quality-filtering the same size did. “Superficial Alignment Hypothesis”: most capability comes from pretraining; SFT mainly teaches which subdistribution of its own behavior to surface.
• AlpaGasus (arXiv:2307.08701) — LLM-judge-filtering Alpaca’s 52K examples down to ~9K high-quality ones produced a better model, 5.7× faster to train. Bad examples actively hurt, not just waste compute.
• IFD / Cherry selection (arXiv:2308.12032) — a cheap, no-external-judge score (the model’s own loss on the response given the instruction, vs. without it) that ranks training value per row; kept only 10% of a dataset and beat training on the full set.
• LESS (arXiv:2402.04333) — gradient-similarity selection of ~5% of a dataset, targeted at the specific downstream capability you want, outperformed the full dataset for that capability. The most directly relevant paper to “which subset of my logs actually moves the gap I diagnosed.”
• DEITA (arXiv:2312.15685) — formalizes selection along complexity/quality/diversity axes; a 6K-example selected set matched/beat 10× more unfiltered data.
• LoRA Learns Less and Forgets Less (arXiv:2405.09673) — full fine-tuning learns more of the target domain but measurably forgets more of everything else; LoRA learns less but preserves out-of-domain behavior — a real, measured trade-off. Curation is the data-side lever for the same problem LoRA solves on the optimizer side; they’re complementary, not redundant.
• Scaling Laws for Forgetting (arXiv:2502.06042) — fine-tuning on a narrow domain measurably increases pretraining-distribution loss as training proceeds, scaling with fine-tune volume and model size; injecting as little as ~1% general/pretraining-like data into the mix meaningfully arrests it. A commonly-cited practical default in adjacent replay literature (arXiv:2605.15220) puts the standard mitigation baseline nearer 10% replay.
• Decontamination (arXiv:2307.09288 §A.6, the Llama 2 report) — token n-gram overlap (>10 tokens shared between an eval sample and the training set) is a simple, reproducible check to confirm your SFT rows don’t secretly contain your benchmark’s own held-out challenges. This is a live, underdiagnosed risk field-wide (arXiv:2406.04244, arXiv:2404.00699).

Concrete curation moves, in order:

1. Verifier-filter first, always — only train on flag-verified/passed runs. Removes the majority of harmful noise in one step (§2–4 above).
2. Prefer your own on-policy verified rows over teacher/off-policy rows when both exist — smaller, more targeted weight movement (see Foundations: the one axis that predicts everything).
3. Score what’s left with IFD, then judge-filter the survivors — cheap first pass, quality second pass.
4. Deduplicate and cap per-archetype volume — if 40% of your logs are the same challenge pattern, training on all of them teaches a shortcut, not a skill.
5. Prefer fewer, harder, correctly-solved trajectories over many easy ones — LIMA’s quality-over- count finding, applied.
6. Replay ~1–10% general-instruction data alongside the agentic corpus if forgetting is a live concern.
7. Decontaminate against your eval set before training, not after a suspiciously good score.
8. Consider LoRA/QLoRA if “don’t overwrite what the model already knows” is a harder constraint than “maximize domain score” — arXiv:2405.09673 gives the actual trade-off curve, not an assumption.

7. Decision aid — which SFT data source for my case

graph TD
  A["Diagnosed failure mode<br/>(see diagnosis/framework.md)"] --> B{"Does the correct behavior<br/>ever appear in the model's<br/>own output, even rarely?"}

  B -->|"Never — knowledge/skill genuinely absent"| C{"Do you have a<br/>stronger teacher<br/>with real tool access?"}
  C -->|Yes| D["Teacher solves for real,<br/>you distill the EXECUTED transcript<br/>(FireAct / AgentTuning style)"]
  C -->|No| E["Human demos, small + curated<br/>(LIMA-style, few hundred rows)"]

  B -->|"Sometimes — partial-solve regime"| F{"Do you have a<br/>verifier / pass-fail check?"}
  F -->|Yes| G["Rejection-sampling SFT:<br/>sample your OWN model,<br/>keep verifier-passed wins<br/>(RAFT / STaR / ReST family)"]
  F -->|"Yes + want student to<br/>stay on its own distribution"| H["On-policy distillation (GKD):<br/>student samples, teacher grades"]

  B -->|"Mis-ranked, not absent"| I["Not an SFT problem —<br/>see preference methods<br/>(DPO/KTO)"]

  G --> J{"Is the row a<br/>full multi-step trajectory?"}
  J -->|Yes| K["Full trajectory (5.1) —<br/>the bulk of your budget"]
  D --> J

  style E fill:#fdd
  style D fill:#dfd
  style G fill:#9f9

Reading it: the green path (your own verified rollouts, or a teacher’s real-execution transcripts) is the safe default. The red path (human demos) is fine but expensive and caps out fast — reach for it only when neither a teacher nor your own model can produce the behavior at any sampling temperature. Nowhere on this tree is “synthetically author a plausible-looking trajectory” a legitimate node for tool-call data — that’s the failure mode §4 names, not a fifth branch.

Cross-links

• Foundations: the one axis that predicts everything — the on/off-policy distinction this chapter’s Axis A is built on, and why off-policy imitation is structurally blind to execution gaps.
• Imitation — SFT · distillation · rejection sampling — the per-method view of the same territory (what/eats/policy/when/gotcha/cite); this chapter is the per-data-source view one level under it.
• The recipe is a sequence, not a pick — “cold-start SFT” as a pipeline stage; §5 there (“the synthetic-trajectory bootstrap”) is the fuller treatment of the confabulation risk from §4 of this chapter.
• Data mixing, ratios & not forgetting how to think — the long version of §6: replay ratios, why LoRA doesn’t fully solve forgetting, reasoning-trace collapse mechanics.
• Method → Data (your real bottleneck) — the per-training-method data-object table (SFT, DPO, KTO, GRPO, agentic RL) this chapter’s §5 sits one level under, for SFT specifically.

Verified citation registry

Confidence calibration: High on every id/abstract above (verified live on 2026-07-02, not recalled from training memory). High on the grounded-vs-synthetic distinction for agentic data and on quality-over-volume for curation — both widely replicated, not contested. Medium-high on the 3-axis taxonomy packaging specifically (§2–3) — it correctly describes each cited paper’s mechanics but is this chapter’s synthesis, not a quoted framework from one source; trust the primary papers’ own mechanics over the packaging if they ever appear to diverge. Explicitly not grounded on any academic cybersecurity-LLM training paper — every citation above is general frontier/ML data-construction or data-selection literature, per this chapter’s brief.