Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Method → Data (your real bottleneck)

Your words: “it’s not that we don’t have data; it’s that we don’t know what data we want and what fine-tune we want.” This chapter is the fix, and it’s a single causal claim:

You do not pick data and then a method. You pick the method — by failure type — and the method dictates the data object you must produce.

This assumes the method itself gets picked the right way first: by evidence tier (Start here: a proven-first ranking of the methods), then by gap type (The decision, Diagnosing the gap — a scientific framework). Once the method is chosen, “what data do we want” is answered mechanically. Here’s the mapping:

MethodData object it consumesWhere it comes from
SFT / off-policy distillationfull trajectories from a sourcecurate, or run a stronger model on your challenges and keep its solves
On-policy distillationyour model’s own rollouts, graded per-token by a teacheryour rollouts + a stronger teacher model
Rejection-sampling FTyour model’s own verifier-passed trajectoriesyou already generate these — filter the runs that pass your verifier
DPO(chosen, rejected) trajectory pairs at a decision pointpair a solved run vs a failed run on the same challenge
KTOunpaired trajectories tagged good/badyour solved pile + your failed pile, as-is (no pairing)
GRPO / RLVRprompts + a verify() fn — no fixed datasetyour challenge set + a deterministic verifier
Agentic RLa live environment emitting rollouts + end-of-episode rewardyour harness itself, as a rollout service

Two consequences you can act on immediately:

  • RLVR needs almost no dataset — just challenges + a verifier. You have both. The “data problem” nearly vanishes; the work moves to the reward fn and rollout infra.
  • Rejection-sampling FT needs only your own solves, which you’re already producing. It’s the lowest-friction first move because the data object is a byproduct of running the benchmark — this is a commonly-reported low-friction entry point, not something specific to your setup.

A caution before you take this table as the full picture: it answers “what does one round of this method eat,” not “in what order do rounds run” or “what happens when a later round’s data mixture is wrong.” Those are separate, larger questions answered elsewhere in this book — The recipe is a sequence, not a pick and Is the recipe a loop? establish that these stage-types get revisited across rounds, not produced once; Ordering rules: interleaving stages & fixing N problems is the table for which stage is safe to run after which (off-policy SFT run after on-policy RL can erode the RL gains — check that chapter before scheduling); and Data mixing, ratios & not forgetting how to think is what to read before combining two of these data objects in the same round, so the mix doesn’t erode capability the base model already had.

So the real question isn’t “what data” — it’s “which gap”

The data object is downstream of the gap diagnosis. Do that first (The decision), and the data spec falls out. The diagnostic that routes everything:

Does the correct action ever appear in the model’s own outputs, even rarely, at high sampling N?

  • Never → knowledge gap → you need external trajectories (SFT / teacher / a tool). Data = curated or teacher-generated.
  • Sometimes (a partial-solve regime — the correct action fires occasionally, not never and not reliably; this is a commonly-observed middle state, not an edge case) → execution gap → data = your own rollouts (rejection-sampling) or a verifier (RLVR). You already have both.
  • Mis-ranked → data = good/bad pairs or tagged logs (DPO/KTO). You already have both piles.

In all three of the last cases, you already possess or can trivially generate the data — which is why the instinct that “data isn’t the bottleneck” is usually correct once you’re past the knowledge-gap case. The bottleneck was the method, and the method is chosen by the gap.

The rest of this chapter makes the above concrete: exactly how many “kinds” of SFT-shaped data exist and why they behave so differently (Q1), and exactly what a training row literally looks like for each of the four data shapes you’d actually build from a corpus of challenges + run logs, and how you pick among them (Q2). No jargon left unexplained; every claim is a live-verified arXiv id.


Q1 — “How many types of SFT are there?” (it’s not the algorithm, it’s the data source)

You correctly sensed there’s one algorithm underneath SFT — cross-entropy on “given this input, produce this exact output, token by token” — and that something else is doing all the governing work. That something is where the target text came from. Same loss function, wildly different outcomes, because the training signal is only as good as its source. This is the axis nobody names for you when they say “just SFT it.”

The two axes that actually matter (plus a third, independent, filter)

Axis A — WHO produced the trajectory:

WhoConcretely
HumanA person did the task, or hand-wrote the “correct” trace.
Stronger teacherA more capable model generated it — this is distillation.
The model itself, on-policyThe model you’re training generated it via sampling, and you kept the good ones.
A different model, off-policySome other model (not the teacher, not the student) generated it — a static dataset, or old logs from a stale checkpoint.

Axis B — was it EXECUTED for real, or SYNTHETICALLY AUTHORED? The axis people skip, and the one that matters most for agentic data — full derivation, the who×executed×filtered table, and the dedicated confabulation-mechanism deep-dive live in The kinds of SFT — Axis B / §4. One-line version for this chapter’s purposes: executed/grounded trajectories teach the real conditional relationship between an action and what comes back; synthetically-authored ones teach only the shape of a plausible transcript, and training on enough of the latter makes the model confabulate tool results.

Axis C (independent of A and B) — was it VERIFIER-FILTERED? Was there a check (unit test, flag match, exact-match grader) that threw out the wrong attempts before they hit the training set?

  • Filtered — only successes go in. This turns “the model generated something” into “the model generated something correct.”
  • Unfiltered — everything goes in, right or wrong. Rare for a reason: it reinforces wrong patterns as strongly as right ones.

Mapping your four intuited types, plus the two you’re missing

The full cell-by-cell table (who × executed? × filtered?, six named methods with citations, grounding risk, when to use each) plus the boxed synthetic-transcript warning live in The kinds of SFT — §3 the taxonomy. Same conclusion applies here: your existing corpus of run logs, filtered to verified wins, already sits in that table’s lowest-grounding-risk cell.

For the full method-level treatment of SFT/distillation/rejection-sampling — including the “positives-only causes entropy collapse, graduate to GRPO” gotcha and the production evidence from Llama 2 and DeepSeek-R1 — see Imitation — SFT · distillation · rejection sampling. This chapter stays focused on the data object, not the algorithm.


Q2 — What do I actually make from raw challenges + run logs, in what format, and which do I pick?

You have two raw ingredients:

  1. Challenge definitions — the prompt, the environment, the tools available, the grading/flag check.
  2. Run logs — transcripts of an agent (yours, or a stronger model) actually attempting a challenge, tool call by tool call, ending in pass or fail.

From these two things you can derive four different shapes of training row. They are not interchangeable. Each teaches a genuinely different capability, and reaching for the wrong shape is the concrete mechanism behind “I added more data and the score didn’t move.”

The four forms, at a glance

FormTeachesDerive fromRow shapePick when you observe
1. Agentic trajectory (obs → think → tool_call → tool_result → … → flag)The actual procedure: which tool, in what order, how to read a bad result and recover, when to stopA verified, executed run log — filtered to ones that actually passed the challenge’s own checkMulti-turn chat transcript; one assistant turn per step, tool turns carry the literal real outputModel “solves unreliably” — has the moves, doesn’t chain them consistently, or gives up/loops
2. Knowledge / Q&AStatic facts: what a term/technique/format means, decoupled from any procedureChallenge writeups/hints/solution docs, turned into standalone question→answer pairsSingle-turn chat pair, no tool callsModel never even attempts a whole class of challenge — genuinely doesn’t know the concept exists
3. Tool-usage / function-callingThe interface, in isolation: correct schema, correct argument names/types, which tool to reach forSingle (call → result) slices sliced out of run logs, or synthesized directly from tool schemastools schema + one assistant turn with a tool_calls arrayModel picks the right idea but the call itself is malformed/hallucinated-args
4. CoT reasoningDeliberation: why this action given this observation, before committingAlmost always already inline in form 1’s assistant turns; if missing, retroactively annotatedA short reasoning block immediately before the tool call it justifies — inside the trajectory row, not a separate fileModel reasons about the right concept but picks the wrong action among several plausible ones

Load-bearing subtlety: these four are not four separate files sitting side by side. Form 1 is the natural home for form 4 (interleaved) and incidentally contains many form-3 examples as a byproduct (every tool call in a trajectory is also a valid isolated tool-usage example). Form 2 is the true outlier — a genuinely separate, single-turn shape — and per the curation section below, it’s also the one most likely to be the wrong choice for a small dense model.

This is the expanded version of the “training object” table in What “data” actually means for an agent — that chapter establishes “a demonstration is a trajectory, not an answer” and the loss-masking rule; this section shows you the literal bytes.


Form 1 — Agentic trajectory (the main event)

What it teaches: given the current tool output, pick the next action; given a failed action, recover instead of giving up; recognize when you’ve actually solved it (verified) vs when you merely think you have.

How to derive it:

  1. Take a run log that ended in a verified pass (the challenge’s own flag/test check said yes — not “looks plausible”).
  2. Slice it into the literal turn sequence: systemuser (the task) → repeating (assistant: reasoning + tool_call) → (tool: the real result) → … → assistant final answer.
  3. Do not clean it up or re-author it. A slightly meandering, real path — including a dead end that self-corrected — is more valuable than a hand-polished shortest path, because at inference time the model will also hit dead ends and needs to have seen what recovery looks like, not just success.
  4. If you have several different successful runs of the same challenge (e.g. from sampling at temperature), keep more than one — this is a measured diversity effect (FireAct, arXiv:2310.05915).

Worked example — one literal training row (OpenAI/Llama-3-style tool-calling chat JSON — this is the shape; swap in your model’s exact chat template before you trust it, see the format-must-match-inference rule below):

{
  "messages": [
    {"role": "system", "content": "You are a security agent. Tools: run_nmap, http_get, submit_flag. Find the flag."},
    {"role": "user", "content": "Target: 10.10.1.4. Find the flag."},

    {"role": "assistant", "content": null,
     "tool_calls": [{"id": "call_1", "type": "function",
       "function": {"name": "run_nmap", "arguments": "{\"target\": \"10.10.1.4\", \"ports\": \"1-1000\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "name": "run_nmap",
     "content": "PORT   STATE SERVICE\n22/tcp open  ssh\n80/tcp open  http\n8080/tcp open http-proxy"},

    {"role": "assistant", "content": null,
     "tool_calls": [{"id": "call_2", "type": "function",
       "function": {"name": "http_get", "arguments": "{\"url\": \"http://10.10.1.4:8080/\"}"}}]},
    {"role": "tool", "tool_call_id": "call_2", "name": "http_get",
     "content": "<html>...<!-- debug endpoint: /api/v1/debug/config --></html>"},

    {"role": "assistant", "content": null,
     "tool_calls": [{"id": "call_3", "type": "function",
       "function": {"name": "http_get", "arguments": "{\"url\": \"http://10.10.1.4:8080/api/v1/debug/config\"}"}}]},
    {"role": "tool", "tool_call_id": "call_3", "name": "http_get",
     "content": "{\"flag\": \"FLAG{exposed_debug_endpoint}\", \"env\": \"staging\"}"},

    {"role": "assistant",
     "content": "Found it in an exposed debug config endpoint at /api/v1/debug/config on port 8080.",
     "tool_calls": [{"id": "call_4", "type": "function",
       "function": {"name": "submit_flag", "arguments": "{\"flag\": \"FLAG{exposed_debug_endpoint}\"}"}}]},
    {"role": "tool", "tool_call_id": "call_4", "name": "submit_flag", "content": "{\"correct\": true}"}
  ]
}

Notice the tool role content on call_2/call_3 — that’s the literal output the real tool produced in that run log, not a rewritten summary and not a plausible-sounding invention. Losing that grounding is the number-one way agentic SFT quietly teaches confabulation (Q1’s boxed warning).

Loss mask: grade only the assistant turns. Mask (ignore) the system, user, and tool turns — you don’t want to train the model to predict tool output, that’s the environment’s job, and you don’t want to train it to predict the user’s own prompt. Standard practice across the agentic-SFT literature; check your trainer’s loss_mask / train_on_inputs=False setting explicitly — silently leaving it on the whole sequence is a common, quiet bug.

Where this is done at real scale: FireAct, arXiv:2310.05915 and AgentTuning, arXiv:2310.12823 build exactly this shape from ReAct-style transcripts. DeepSeek-R1, arXiv:2501.12948 does the rejection-sampling version of this at scale: cold-start SFT → RL → rejection-sample the RL checkpoint to build ~600K new SFT rows, keeping only correct completions, then a second SFT/RL pass — the closest published description of “run your own agent, keep the verified wins, train on those.”


Form 2 — Knowledge / Q&A

What it teaches: facts and concepts — “what does an exposed debug endpoint typically leak,” “what does this error mean” — the kind of thing a lookup answers, with no multi-step procedure involved.

How to derive it: pull the natural-language writeup/hint/solution text that already exists per challenge (or per challenge class — you want concept coverage, not one row per challenge instance). Turn it into an explicit question a model would need answered before it would even attempt the right move, paired with a concise, correct answer. De-duplicate across challenges sharing a root concept.

Worked example — one training row:

{
  "messages": [
    {"role": "user", "content": "I found an endpoint at /api/v1/debug/config on a web service. What's the risk and what should I check for?"},
    {"role": "assistant", "content": "Debug/config endpoints left enabled outside dev commonly leak environment variables (API keys, DB credentials), internal service URLs, and stack traces. Standard next step: GET it directly (often unauthenticated by mistake) and grep the response for 'key', 'secret', 'token', 'password', 'flag'."}
  ]
}

When to pick this — and when NOT to: only if error analysis on your run logs actually shows the model failing because it doesn’t know a fact exists, not because it can’t act on tools it already understands. This is closer to classic instruction-tuning (Self-Instruct, arXiv:2212.10560) than to agentic training, and for a dense model already scoring meaningfully above zero on an agentic benchmark, most failures at that stage are behavioral (bad sequencing, giving up early, not verifying before submitting) — not knowledge gaps. Don’t reach for form 2 reflexively; see the curation section below for why baking facts into weights is often the wrong fix for a small model in the first place.


Form 3 — Tool-usage / function-calling (isolated, single-turn)

What it teaches: the mechanical skill of calling a tool correctly — right function name, right argument schema, right JSON — decoupled from the multi-step plan.

How to derive it: slice single (context → one tool_call) pairs out of run logs, or synthesize them straight from your tool schemas. This one is safe to synthesize even without real execution, because there’s no tool result being faked — you’re only teaching the shape of the call.

Worked example — one training row:

{
  "messages": [
    {"role": "system", "content": "Tools: run_nmap(target, ports), http_get(url), sqlmap_scan(url, param)."},
    {"role": "user", "content": "Check if the 'id' parameter on http://10.10.1.4/product?id=1 is SQL-injectable."},
    {"role": "assistant", "content": null,
     "tool_calls": [{"id": "call_1", "type": "function",
       "function": {"name": "sqlmap_scan", "arguments": "{\"url\": \"http://10.10.1.4/product?id=1\", \"param\": \"id\"}"}}]}
  ]
}

When to pick this: when the diagnosis is narrow — the model has the right idea but botches the mechanical call (wrong argument names, hallucinated parameters, malformed JSON). Grounded in Gorilla, arXiv:2305.15334 (fine-tuned specifically to cut hallucinated API calls) and ToolLLM, arXiv:2307.16789 (constructs multi-tool call chains by searching for a valid call path, then trains on the validated sequence, and shows the pattern generalizes to unseen APIs — relevant if you expect new tools later). This is a cheap, low-risk fix because it’s format-only, not reasoning-only — don’t reach for a full trajectory rebuild when this narrow slice fixes it.


Form 4 — CoT reasoning (interleaved, not a separate file)

What it teaches: deliberation before commitment — reading an ambiguous observation and reasoning about which of several plausible next moves is right, before calling the tool. The difference between a model that has the right moves but applies them inconsistently, vs one that’s genuinely missing a move.

How to derive it: for each assistant turn in a form-1 trajectory row, prepend a short reasoning block explaining why this action given this observation — either the real reasoning the agent produced live (if your logs capture it) or a short authored rationale bridging observation → action for logs that didn’t. This is STaR’s “rationalization” mode: generate the rationale after seeing the correct answer, keeping the actions/tool-calls grounded while making the reasoning explicit.

Worked example — same trajectory as form 1, with reasoning added:

{
  "messages": [
    {"role": "system", "content": "You are a security agent. Find the flag."},
    {"role": "user", "content": "Target: 10.10.1.4. Find the flag."},

    {"role": "assistant",
     "content": "Start broad: scan for open ports before probing any single service.",
     "tool_calls": [{"id": "call_1", "type": "function",
       "function": {"name": "run_nmap", "arguments": "{\"target\": \"10.10.1.4\", \"ports\": \"1-1000\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "name": "run_nmap",
     "content": "PORT   STATE SERVICE\n22/tcp open  ssh\n80/tcp open  http\n8080/tcp open http-proxy"},

    {"role": "assistant",
     "content": "Port 8080 (http-proxy) is non-standard and often hosts admin/debug panels rather than the main app on 80 — check it first.",
     "tool_calls": [{"id": "call_2", "type": "function",
       "function": {"name": "http_get", "arguments": "{\"url\": \"http://10.10.1.4:8080/\"}"}}]}
  ]
}

Do not build a separate, standalone “reasoning dataset” disconnected from tool calls — that tends to produce a model that “thinks well” in the abstract but doesn’t connect the thought to the actual next action. Keep it interleaved inside form-1 rows. Grounded in DeepSeek-R1’s own practice of generating the reasoning trace as part of the same rejection-sampled trajectory, and STaR’s rationalization mode for backfilling rows whose live reasoning was messy or absent.


Pick by diagnosed gap — one table, per challenge cluster

Don’t build all four forms uniformly across the whole corpus. Diagnose each failing challenge (or cluster) first — using The decision / Diagnosing the gap — then pick the form that matches:

You observe in your run logs…Diagnosed gapMake this form
Right idea, tool-call JSON malformed / wrong argsMechanical interfaceForm 3 — small, targeted dose
Calls tools fine, but gives up / loops / doesn’t recover from a bad resultMulti-step procedureForm 1 — the bulk of your training budget
Doesn’t even know the vulnerability class or technique appliesMissing knowledgeForm 2 — small, curated dose, and only after verifying it’s really the gap (see curation §)
Reasons about the right concept but picks the wrong action among several plausible onesMissing deliberation at the decision pointForm 4, layered onto Form 1

In practice, for moving an agentic benchmark meaningfully, the overwhelming majority of your training budget should be Form 1 (full, executed, verified trajectories) — this is what FireAct and AgentTuning both do, and it’s the whole point of the rejection-sampling literature (RAFT/STaR/ReST/ReST-EM): your own model’s verified wins are simultaneously on-policy, executed, and filtered — the single highest-signal, lowest-risk training data you can generate, and you’re already producing it as a byproduct of running the benchmark.


Format must match inference — this breaks silently

This is the single most consequential, least forgiving rule across all four forms, and it deserves its own section because it fails silently, not loudly.

Every row above must be rendered through your model’s actual chat_template — the exact same role markers, tool-call wrapper tokens, and <think>-tag conventions (if your base model uses them) that your serving stack applies at inference time. A mismatch does not throw an error. It quietly degrades output.

  • Hugging Face’s own post on this states it plainly: “Using a format different from the format a model was trained with will usually cause severe, silent performance degradation… this is an especially dangerous issue because using the wrong chat format is a silent error — you won’t get a loud failure or a Python exception.” (huggingface.co/blog/chat-templates)
  • Concrete, repeatedly-reported failure modes: garbled/run-on output because the stop-token position was never learned; format collapse on the first generated token because the training prompt didn’t end exactly where the reply begins; “great eval loss, bad real output” because eval was measured with the training template while production applies a different one; double-templating (manually wrapping messages in role tags AND letting the trainer’s tokenizer re-apply apply_chat_template) silently nests special tokens and corrupts every row.
  • Practical check, before every training run: render one example with tokenizer.apply_chat_template(...), decode it back with skip_special_tokens=False, and eyeball that every role marker / tool-call wrapper appears exactly once per turn and matches what the serving stack will actually feed the model.
  • The loss-mask rule from Form 1 follows the same logic: mask everything except assistant-authored tokens, and keep the end-of-turn/stop token inside the graded region — masking it out is a documented cause of the model never learning to stop generating.

Takeaway: whatever schema you pick for tool calls (tool_calls array, inline <tool_call>{...}</tool_call> tags, whatever your model family uses), it must be byte-for-byte the schema your serving stack presents at inference. Copy it FROM the serving config; don’t invent one independently in a data-prep script.


“Only accept what it needs” — data selection so SFT doesn’t overwrite the model

This is the part that’s cheap to skip and expensive to have skipped. A dense open-weight model already carries broad general capability. Dumping every raw transcript you have at it — full fine-tune, no filter — risks overwriting general capability to fit a narrow domain pattern. Two distinct, separately-cited risks, both real:

1. More data doesn’t reliably mean a better model

LIMA, arXiv:2305.11206 — a 65B model fine-tuned on just 1,000 carefully curated (human-quality) examples matched or beat models trained on orders of magnitude more data. Their own ablation is the load-bearing part: doubling the training set, holding quality fixed, does not improve response quality — but filtering for quality does give a real, measured jump. Their “Superficial Alignment Hypothesis”: most of a model’s knowledge comes from pretraining; SFT mostly teaches it which of its own latent behaviors to surface, in what format. Translated to your setup: quality (did this trajectory actually solve the challenge cleanly, with a sound tool-use trace) and diversity (coverage across challenge types, not raw count of runs) beat volume. Corroborated on filtering specifically by AlpaGasus, arXiv:2307.08701: filtering Alpaca’s 52K examples down to ~9K LLM-judge-scored high-quality ones produced a better model than training on all 52K — noisy examples actively hurt, they don’t just waste compute.

2. Fine-tuning on a narrow domain measurably erodes capability outside it

LoRA Learns Less and Forgets Less, arXiv:2405.09673 — full fine-tuning learns more of a target domain but also forgets more of what the base model could already do outside that domain; low-rank (LoRA/QLoRA) fine-tuning learns less but preserves general capability better, a real measured trade-off, not folklore. If “don’t overwrite what the model already knows” is a hard constraint, this is your citation for choosing LoRA/QLoRA over full fine-tune, or for capping how many full-fine-tune epochs you run. This mechanism is explored in much more depth — including why it fails specifically as reasoning-trace collapse, not generic weight destruction — in Data mixing, ratios & not forgetting how to think; read that chapter before your first real training run, not after a collapse.

The concrete curation recipe

  1. Verifier-filter first, always. Only train on trajectories whose pass/flag check actually succeeded — this alone removes the majority of harmful noise, and it’s the line between “grounded, verified data” and the confabulation risk from Q1’s boxed warning.
  2. Prefer your own model’s verified successes over a teacher’s or synthetic data, when you have them. This is the on-policy-distillation principle generalized: GKD, arXiv:2306.13649 shows training a student on its own sampled outputs (with a teacher only grading, not authoring) beats training on a fixed external corpus, because it avoids the train/inference distribution mismatch off-policy data introduces — the update stays small and targeted instead of force-fitting the model to a different model’s style. A 2026 follow-up sharpens this further: on-policy distillation only helps when the teacher signal offers genuinely new capability beyond what the student already produces — otherwise the update is close to a no-op.
  3. Score what’s left with a cheap filter before accepting it. Two independent, complementary options: an intrinsic, model-own-loss-based filter (Instruction-Following Difficulty, “Cherry” selection — kept just 10% of a dataset and beat training on all of it), or an extrinsic LLM-judge filter (the AlpaGasus method above) that catches quality issues the intrinsic filter can’t, like a correct flag reached via sloppy or lucky reasoning.
  4. Deduplicate and cap per-challenge-archetype volume. If a large fraction of your logs are the same technique (e.g. one vulnerability class solved the same way over and over), training on all of them teaches a surface-pattern shortcut, not general skill — cap examples per archetype.
  5. Reserve a small replay slice of general-purpose data. Injecting even a small fraction of general instruction/knowledge data (unrelated to your challenge domain) into the mix meaningfully arrests forgetting — the mechanism is measured directly in continual-fine-tuning literature; treat a small floor as the minimum and a larger fraction as the commonly-used practical default, and re-verify the exact ratio on your own eval rather than copying a number from a different domain/scale. Full treatment of replay ratios: Data mixing, ratios & not forgetting how to think.
  6. Decontaminate against your eval set before you train. If your raw challenge corpus and your benchmark’s held-out scored set overlap — same vulnerability class, same generator/template family — any training row derived from a benchmark-adjacent challenge risks the model memorizing the answer pattern instead of the skill, inflating the training-set score without inflating actual capability. A simple, reproducible check: n-gram overlap between training rows and eval challenges (the methodology Llama 2, arXiv:2307.09288 §A.6 uses — a token counts as contaminated if it sits in a shared run of more than ~10 tokens between an eval sample and the training set). This is a known, actively-studied reliability problem in the field, not overkill for a small corpus.
  7. Prefer a retrievable tool/RAG lookup over baking pure facts into weights, when the diagnosed gap really is Form 2 (knowledge). A small dense model has limited spare capacity; a lookup retrieves perfectly every time where weight-memorization degrades as the fact-set grows and risks displacing other knowledge. Reserve actual Form-2 training rows for cases where the “knowledge” is really a reasoning pattern wearing a fact’s clothes (e.g. “why is an exposed debug endpoint dangerous” generalizes to endpoints the model has never seen named) — that’s worth baking in because it needs to generalize, not just be looked up. Confidence on this specific framing: medium — it’s standard retrieval-vs-parametric-knowledge reasoning, not pinned to one paper in the notes behind this chapter.

Why this bites harder for a small dense model specifically: every parameter is shared across every skill in a dense model — there’s no unused capacity to overwrite consequence-free, and a smaller model has less raw slack than a much larger one to absorb noise without measurable collateral damage. This is the reason curation isn’t optional polish here — it’s the data-side lever for the exact same problem LoRA addresses on the optimizer side, and the two are complementary, not redundant: LoRA constrains how much the weights move, curation constrains in what direction they’re pushed.



Confidence & citation registry

All arXiv ids below were verified live (title, authors, abstract pulled directly from arxiv.org, not from training-data memory) on 2026-07-02, during the research pass behind this chapter. High confidence on every id’s existence and core claim. Medium confidence on the framing choices that are this chapter’s own synthesis (the 3-axis taxonomy in Q1; the RAG-over-weights framing at the end of the curation section) — flagged inline above, not presented as a quoted claim from a single source. No academic cybersecurity-LLM paper is used as grounding anywhere in this chapter — every citation below is general frontier-lab or ML data-construction/data-selection literature.

idPaperRole in this chapter
2203.02155InstructGPTHuman-demo SFT origin
2212.10560Self-InstructSynthetic-authoring instruction generation
2304.12244WizardLM / Evol-InstructSynthetic-authoring, complexity escalation
2306.11644Textbooks Are All You Need (phi-1)Synthetic “textbook quality” data
1503.02531Distilling the Knowledge in a Neural Network (Hinton et al.)Conceptual root of distillation
1606.07947Sequence-Level Knowledge Distillation (Kim & Rush)Teacher-executed transcript distillation
2306.13649GKD — On-Policy Distillation of LMsMissed type #1; on-policy-vs-off-policy curation principle
2304.06767RAFTRejection-sampling SFT
2203.14465STaRRejection-sampling + rationalization (Form 4 derivation)
2308.08998ReSTIterative rejection-sampling SFT
2312.06585ReST-EM (Beyond Human Data)Iterative rejection-sampling, scaling
2310.05915FireActAgentic-trajectory SFT (missed type #2; Form 1 derivation)
2310.12823AgentTuningAgentic-trajectory SFT (missed type #2; Form 1 derivation)
2501.12948DeepSeek-R1Rejection-sampling at scale; reasoning-trace-in-trajectory practice
2305.15334GorillaForm 3 grounding — reducing hallucinated API calls
2307.16789ToolLLMForm 3 grounding — validated multi-tool call chains
2305.11206LIMAQuality/diversity > volume; Superficial Alignment Hypothesis
2307.08701AlpaGasusLLM-judge filtering beats full-dataset training
2402.04333LESSTargeted, capability-specific data selection
2312.15685DEITA (What Makes Good Data for Alignment?)Complexity/quality/diversity selection axes
2308.12032Cherry_LLM (IFD selection)Cheap intrinsic quality filter
2405.09673LoRA Learns Less and Forgets LessPEFT vs full-FT forgetting trade-off
2502.06042Scaling Laws for Forgetting during Finetuning with Pretraining Data InjectionReplay-slice mitigation, forgetting mechanism
2605.15220Always Learning, Always MixingReplay-ratio practical baseline context
2604.13016Rethinking On-Policy Distillation of LLMsOn-policy distillation only helps with genuinely new teacher signal
2307.09288Llama 2n-gram decontamination methodology (§A.6)
2406.04244Benchmark Data Contamination of LLMs: A SurveyWhy decontamination matters, field-wide
2404.00699LLMSanitizeContamination detection tooling context

Not independently re-verified for this specific pass (mentioned only where they surfaced in the source research, not load-bearing here): none — every id above was verified in the notes this chapter draws from.