What “data” actually means for an agent

You asked the right question earlier: “demonstration = the answer — are you talking about trajectories?” Yes. Being concrete about the data object dissolves most of the confusion, because each method eats a different-shaped object, and for an agent those shapes are not what the chatbot literature implies.

This chapter stays deliberately narrow: it’s about the shape of the training object, not about which method to pick. That routing question belongs to The one axis that predicts everything — on-policy vs off-policy is the real axis, and each method below is a fixed preset on it, not one of several independent knobs you mix and match freely (see Contested edges on why a “combinatorial grid” framing over-reaches).

A “demonstration” is a trajectory, not an answer

• Chatbot SFT example = (prompt → ideal response text).

• Agent SFT example = the whole trajectory:

  system prompt
  → [assistant: reasoning] → [tool_call: shell "nmap …"] → [tool_result: <output>]
  → [assistant: reasoning] → [tool_call: http_request …] → [tool_result: <output>]
  → …
  → [assistant: submit_flag("FLAG{…}")]
  

The training target is the path, not the flag. That is what SFT and rejection-sampling FT imitate token-by-token.

Loss-masking: don’t train the model to predict the world

Tool outputs / observations are part of the sequence but are not the policy’s own tokens — they come from the environment. Standard practice is to mask the loss on prompt and observation spans and only compute loss on the model-generated reasoning + tool_call + submit tokens. Otherwise you teach the model to hallucinate stdout. (This is the same principle as instruction-tuning masking the prompt; for agent traces it’s applied to every observation span.) The project’s own trajectory-synthesis note treats observation-masking as a first-class filter step — see lessons/post-training/verified-trajectory-synthesis-recipe.md in the shared memory pool. (Caveat: that recipe’s specific numbers derive from an unverified third-party report — trust the procedure, not the figures.)

The data shapes, per method preset (forward reference)

These rows are not independent toggles you compose freely — each is a named method with a fixed data object, and which preset applies is determined by where the failure sits on the on/off-policy axis (Foundations, The decision):

This table is the hinge of the whole book. It’s expanded in Method → Data, which is the chapter that actually addresses your stated bottleneck (“we don’t know what data we want”). The short version: you don’t choose data and then a method — you choose the method, and the method dictates the data object.

A caution worth stating before you take that table as license to freely order/mix methods: the object shape here answers “what does one round of this method eat,” not “in what order do rounds run” or “what happens when you feed a later stage’s data mixture badly.” Those are separate, larger questions this book answers elsewhere — see The recipe is a sequence, not a pick, Is the recipe a loop?, Ordering rules: interleaving stages & fixing N problems, and Data mixing, ratios & not forgetting how to think (the last one is the direct continuation of this chapter’s “trajectory as training object” framing — it’s what happens when an off-policy trajectory mixture is wrong even though the shape per row above is right). When you’re ready to rank candidate methods/recipes by how proven they are rather than how novel, that’s Proven-first ranking.