Introduction

This is a living field manual for post-training an agent — written for an engineer who wants to know it, then do it, not experiment blindly. It grows: each session adds or sharpens a chapter.

📄 Offline? Grab the whole book as a single PDF: download.pdf — every chapter, diagrams included, for reading with no network.

The problem it exists to answer

Run a CTF-playing agent against a real benchmark and the pattern is a familiar one: most challenges fail, only a minority solve. You have the harness, the workflow, the budget, and the backing to do anything. The bottleneck is usually not data — it’s knowing which fine-tuning method you want, and therefore what data to build, and why.

Everything here builds toward answering that from first principles.

The shape of this book

Four parts, front to back — Problem → Learnings → Understanding → Conclusion:

1. Problem — this chapter, the 5-minute journey, and the diagnostic chapters (behavioral audit → training signal, diagnosing the gap) establish what’s actually broken and how to know it from evidence, not vibes.
2. Learnings — how models are actually built — the general theory, unattached to your case: the on/off-policy axis, what “data” means, the method family (imitation / preference / reinforcement / agentic RL / PEFT), and how frontier labs sequence, loop, order and batch, and mix data without forgetting these into real recipes.
3. Understanding — applying it to your case — the same theory turned toward your bottleneck: ranking methods by proven-ness, deciding whether you have one problem or many, mapping method → data, and what instrumentation has to exist before a training run.
4. Conclusion — the actual decision, the roadmap and forks still open, and the edges that are genuinely contested (so you don’t mistake open questions for settled ones).

How to read it

• Want the feel first? → The 5-minute journey — the interactive version, embedded.
• Want the theory? → start at Learnings and go in order.
• Want the answer for your case? → jump to Method → Data and The decision.
• Want to know if you actually have an execution gap, and which RL technique fixes it? → Diagnosing the gap, then the RL sweep at RL that creates value.

What’s canonical vs. what’s a teaching scaffold (read this once)

Being honest about provenance, because you’re becoming a researcher and the distinction matters:

• Canonical, universal, you’ll find it in any RL/post-training text: the on-policy vs. off-policy axis, and the three learning paradigms (imitation / preference / reinforcement). These are load-bearing and not up for debate.
• My teaching scaffold: any packaging that presents these as “N knobs you freely toggle.” The axes describe methods; they are not independent dials you combine — each named method is a fixed preset. An earlier interactive matrix implied free combination and produced nonsense for some products. That was the scaffold over-reaching. Corrected here: learn the one axis (on/off-policy) + the fixed method presets, not a combinatorial grid.

The one line to anchor on

Every method is the same move — push probability mass toward good behavior — differing only on whose distribution the data comes from (off- vs on-policy) and whether you also learn from failures.

Keep that; the rest is detail.