Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

How this book grows

This is a living document maintained by the researcher seat of the llmresearch project. It grows one verified topic at a time; each chapter cites its sources so claims are checkable, not assertions.

Conventions

  • Engineer-level. Assumes you know logprobs, KL, advantage, rollouts, MoE, PPO clip. No 101 filler.
  • Cite or don’t claim. Every substantive statement carries an arXiv id or a named lab report/blog. Where something is contested, it’s marked contested with both sides (Contested edges).
  • Honesty about status. Methods are tagged mainstream / niche / promising-not-proven / experimental based on whether a frontier flagship’s report actually uses them.
  • Verified live. arXiv ids are crawl-checked; lab-recipe claims come from 2025–2026 tech reports and blogs, not training recall. Re-verify before betting a run — this field ships weekly.

Build & run locally

# one-time: install the toolchain (macOS)
brew install mdbook mdbook-mermaid
# from the book root:
mdbook-mermaid install .   # vendors mermaid assets + wires the preprocessor
mdbook serve --open        # live-reload server at http://localhost:3000

Mermaid flowcharts and raw HTML/iframes (e.g. the embedded journey) render offline — no CDN required.

Log

  • 2026-07-02 — v1.0. Added Teaching a tool, teaching recon — the data behind a skill (wired into “Learnings,” right after Hint-guided bootstrapping — put the walkthrough in the prompt, then train it away — the data-object-first answer to “Q&A or trajectories or executed traces, for teaching a tool vs. teaching recon,” drawn from seven sources (Toolformer, ToolLLM/ToolBench, APIGen/xLAM, Gorilla/DocPrompting, terminal-CLI-agent literature, search/recon-RL literature, and a procedural-vs-declarative synthesis): teaching a fixed-signature TOOL is a strict quality ladder — doc-grounding (Gorilla, DocPrompting) < unverified trajectories < executed-and-verified traces (APIGen/xLAM/APIGen-MT’s format→execution→semantic 3-gate filter) as the non-negotiable ceiling, with Toolformer’s loss-reduction filter as the self-supervised executed-trace variant; teaching RECON (search/information-seeking) is a structurally different shape — a thin, format-only SFT seed followed by outcome-only RL that lets the branching strategy emerge (Search-R1’s retrieved-token masking, R1-Searcher’s two-stage reward, DeepResearcher’s emergent self-reflection/cross-validation, TIER’s negative case against trajectory-supervised reward collapsing past search depth 4–6, and Nemotron-Research-Tool-N1’s contested pure-RL-beats-SFT-then-RL finding vs. WebGPT’s imitation-then-rejection-sampling counter-case for large action spaces); unanimous verdict across all seven sources — Q&A caps out at declarative recall, trajectories are necessary for procedural transfer. Ships a skill-type → data-object → how-made → executed? → technique consolidated table, worked training rows for both tool-teaching and recon-teaching, and a docs → executed → verify → SFT → optional-RL mermaid pipeline. Cross-linked into kinds-of-sft.md, methods/imitation.md, methods/rl-long-horizon-exploration.md, methods/agentic-rl.md, hint-guided-bootstrapping.md, and foundations/on-off-policy.md. Merged 5 new citations into References via a new “Teaching a tool, teaching recon” section (Toolformer, DocPrompting, xLAM/APIGen-MT multi-turn extension, Nemotron-Research-Tool-N1, TIER), deduped against the existing corpus — Gorilla, ToolBench/ToolLLaMA, APIGen/xLAM-function-calling-60k, APIGen-MT, Search-R1, ToolRL, and WebGPT were already present from earlier passes and are not repeated.

  • 2026-07-02 — v0.9. Recursive deepening pass on Case study: how coding engineered its data (and what transfers to cyber) — the v0.7 chapter grew five new sections (§5 execution sandbox infrastructure at scale — isolation-strategy cost table incl. SWE-MiniSandbox kernel-level isolation and the SWE-World Docker-free learned surrogate, plus a per-rung $/sample cost curve; §6 synthetic test quality and the weak-verifier failure mode — VeriScale’s 83× adversarial mutation-testing suite expansion and the SpecBench reward-hacking-gap-vs-code-size finding; §8 decontamination and the memorization-vs-reasoning gap — the “SWE-Bench Illusion” study’s 23-point memorized-vs-fresh localization gap and metamorphic-transform diagnosis, extended into a cyber-specific three-timestamp CVE filtering rule via BountyBench’s exploitation-as-retrieval evidence; §9 environment-generation mechanics — SWE-Factory and million-scale SWE-Universe automating the remaining manual Docker-environment stages; §12 the vulnerable-target-reconstruction open problem, a 2026-07 update — CVE-Factory’s three-stage CVE→Docker pipeline (LiveCVEBench), ARVO, the DARPA AIxCC SoK, and CVE-Bench, closing with the honest verdict that patch/exploit semantic correctness remains substantially unsolved), plus a deepened §11 (Seed-Coder, and three Rung-4 attempts — RepoZero, ProgramBench, CodeAlchemy — that each converge on the same real-oracle circular dependency, plus the independent output-diversity-collapse-under-recursive-training finding). Added a Source registry section tabulating every §5–§12 id by subsection with an explicit confidence note (which ids were independently re-verified this pass vs. inherited from the underlying deep-research artifacts). Merged 18 new citations into References’s existing “Case study: how coding engineered its data” section — SWE-MiniSandbox, SWE-World, VeriScale, SpecBench-style analysis, SWE-Bench Illusion, metamorphic-testing memorization diagnosis, BountyBench, SWE-Factory, SWE-Universe, Seed-Coder, RepoZero, ProgramBench, CodeAlchemy, output-diversity-collapse, CVE-Factory, ARVO, SoK: DARPA AIxCC, All You Need Is A Fuzzing Brain, CVE-Bench — deduped against the chapter’s own already-covered-elsewhere note (StarCoder, StarCoder2/Stack v2, DeepSeek-Coder, Phi-1, Self-Instruct, CodeRL, PPOCoder, RLTF, StepCoder, SWE-Gym, R2E-Gym/SYNGEN, SWE-RL) and against the rest of the corpus (AlphaCode, KodCode, CodeT, Sol-Ver, STaR, Kimi K2, DeepSeek-Coder-V2 already present). Two ids (SpecBench 2605.21384, metamorphic-testing 2604.21579) are recorded as inherited-not-independently-re-crawled this pass, matching the chapter’s own confidence note.

  • 2026-07-02 — v0.8. Added Hint-guided bootstrapping — put the walkthrough in the prompt, then train it away (wired into “Learnings,” right after The kinds of SFT — it is the data, not the algorithm — the cheat-sheet-training survey: names the family behind “put the solution in the system prompt, execute for real, mask it out, then SFT on the unhinted task” across four independent lineages (STaR-rationalization, context/prompt distillation, hint-guided RLVR exploration, and Learning Using Privileged Information/asymmetric actor-critic, 2009–2026); formalizes the hint-then-mask loss (Snell et al.) and its latent-variable/biased-stochastic-EM reading (TRICE); tables 8 named techniques with what each contributes and where each breaks; separates three distinct failure-mode classes (shortcut learning/hint-copying, distribution mismatch/exposure bias, elicit-not-expand) rather than conflating them; and lands the honest verdict — the mechanism is real and well-precedented, but NuRL’s own ablation is the load-bearing caveat against the user’s specific design choice (a full walkthrough, not an abstract hint, hurts relative to an abstract cue) — with a concrete mitigated recipe (mask + hint-reliance filter + fade-the-hint curriculum + verify-unaided-on-never-hinted tasks) as a mermaid diagram. Cross-linked into kinds-of-sft.md, method-to-data.md, rl-long-horizon-exploration.md, is-the-recipe-a-loop.md, and contested.md. Merged 24 new citations into References via a new “Hint-guided bootstrapping” section (TRICE, V-STaR, Askell et al. context distillation, PING, Snell/Klein/Zhong, Kujanpää et al., OPCD, on-policy self-distillation diversity-collapse, OPSA, HiLL, ReGFT, Multi-level Stepwise Hints, E2H Reasoner, AdaRFT, Nair et al. demo-loss annealing, Go-Explore Nature version, Asymmetric Actor-Critic, Shortcut Learning survey, HANS, Clever Hans on COPA, RAWR, Turpin et al. unfaithful CoT, Ross/Peters/Marasović, and Countdown-Code), deduped against the existing corpus — STaR, HER, Go-Explore (original), NuRL, the Reward Hacking survey, and Peng et al./RCG/SSR were already present from earlier passes and are not repeated.

  • 2026-07-02 — v0.7. Added Case study: how coding engineered its data (and what transfers to cyber) (wired into “Learnings,” right after Cybersecurity is one of a family — what cracked the others — a general-purpose case study on the DATA trajectory this book had under-covered: how raw public material actually becomes training rows, not which loss function to use. Coding is the case study because it’s the closest twin to cybersecurity — both are executable domains with a deterministic oracle and a noisy public commons — and coding already completed the autocomplete→single-function-correctness→self-debugging→whole-repo-agentic arc cyber is now starting. A 4-rung ladder ordered by how much execution is in the data pipeline (Rung 0 pretraining/autocomplete, execution ≈ absent; Rung 1 grounded synthetic instructions, execution gates the synthesis recipe; Rung 2 execution-verified rejection-sampling/RL, execution is the primary per-sample filter/reward; Rung 3 repo-to-environment agentic trajectories, execution is the entire training substrate), each rung’s centerpiece table mapped 1:1 onto a cybersecurity analog (CTF sandbox + flag verifier = compiler + tests, cleanest at Rung 2), plus two explicit honest caveats: the raw-corpus scale gap between GitHub and CTF writeups/CVE databases is real and unresolved, and SWE-RL’s rule-based difflib reward beating execution-per-rollout on cost grounds is a possible cost-saving lever for cyber RL rewards, not a proven one. Cross-linked into kinds-of-sft.md, method-to-data.md, adjacent-domains-transfer.md, frontier-recipe-is-a-sequence.md, and data-mixing-and-forgetting.md). Merged 12 new citations into References via a new “Case study: how coding engineered its data” section (The Stack, Code Llama, FIM, WizardCoder/Evol-Instruct, Magicoder/OSS-Instruct, CodeT, LEVER, GenX, KodCode, SOL-VER, Self-Debugging, SWE-bench), deduped against the existing corpus — StarCoder, StarCoder2/Stack v2, DeepSeek-Coder, Phi-1, Self-Instruct, CodeRL, PPOCoder, RLTF, StepCoder, SWE-Gym, R2E-Gym/SYNGEN, and SWE-RL were already present from earlier passes and are not repeated.

  • 2026-07-02 — v0.6. Added The kinds of SFT — it is the data, not the algorithm (wired into “Learnings,” right after Imitation — the per-data-source companion to that chapter’s per-method view: SFT is one loss function, cross-entropy on (input → target tokens); what actually varies is the data — human-authored, synthetic-authored, distilled off-/on-policy, and rejection-sampled/self-generated — with the two most commonly missed types called out explicitly: formalized agentic-trajectory SFT (FireAct/AgentTuning) and on-policy distillation (GKD)) and rewrote Method → Data (your real bottleneck) in place into a concrete, worked data guide — per-method (SFT/DPO/KTO/GRPO/agentic-RL) data-object tables, curation/selection axes (LIMA, AlpaGasus, Cherry_LLM/IFD, LESS, DEITA), and a decontamination pass (Llama 2 n-gram methodology, LLMSanitize). Also de-pinned the BSides/baseline framing book-wide (no longer anchored to one conference-talk-shaped example) and refreshed the foundational chapters (What “data” actually means for an agent, The one axis that predicts everything, Introduction) for consistency with the new data-first framing. Merged 11 new citations into References via a new “The kinds of SFT & Method → Data” section (Sequence-Level KD, FireAct, AlpaGasus, Cherry_LLM/IFD, LESS, DEITA, Scaling Laws for Forgetting, Always Learning Always Mixing, the contamination-survey + LLMSanitize pair, and Rethinking On-Policy Distillation), deduped against the existing corpus — the large majority of both new/rewritten chapters’ citations (Self-Instruct, WizardLM, phi-1, Hinton distillation, GKD, RAFT, STaR, ReST/ReST-EM, AgentInstruct/AgentLM, DeepSeek-R1, LIMA, LoRA-learns-less, Llama 2, Gorilla, ToolLLaMA, InstructGPT) were already present from earlier passes and are not repeated.

  • 2026-07-02 — v0.5. Wired four new chapters, completing the “loop-shape / ordering / data-mixing” arc this book’s sequencing story was still missing, plus a fast-start entry point: Is the recipe a loop? (the macro finding — post-training is a bounded/asymmetric loop, not a one-shot pipeline: the pretrain/anneal wall + base-choice + reward contract are the one-shot boundary; SFT/preference/RLVR is the iterated tail; continue-vs-restart is a per-stage not global rule; non-commutativity now has a formal proof, not just observation; a loop-exit criterion so an iterative plan isn’t an open-ended compute sink), Ordering rules: interleaving stages & fixing N problems (the micro companion — an eleven-transition safe/conditional/erosive ordering table keyed on data provenance not stage-name, the RL’s-Razor/entropy-collapse mechanism for why foreign data after RL is dangerous, resolving the DeepSeek-R1 “SFT after RL” paradox by data provenance not stage label, and the mix-don’t-sequence verdict for batching N pass@k-identified problems), Data mixing, ratios & not forgetting how to think (diagnoses a real field anecdote — LoRA SFT on off-policy terse trajectories made a reasoning model stop emitting CoT entirely — as named, measured “reasoning-trace collapse,” not weight destruction; why LoRA’s “learns less, forgets less” is a magnitude-bounded aggregate-benchmark claim, not a behavioral-direction guarantee, via the “intruder dimensions” mechanism; the 1–10% replay-ratio band converged across six papers; a merge-vs-mix decision aid; a concrete anti-forgetting recipe with a mandatory per-checkpoint “does it still think?” probe), and Start here: a proven-first ranking of the methods (placed at the top of “In practice” as the fast-start entry point — ranks every method in the book by adoption breadth × flagship usage × measured impact with novelty penalized, T1–T4 tiers across the RL/preference/SFT families, landing on the one proven end-to-end starting sequence: SFT → rejection-sampling SFT → DPO → GRPO/RLVR → iterate). All four chapters already carry their own Cross-links section back into frontier-recipe-is-a-sequence.md, decomposition-vs-monolithic.md, diagnosis/framework.md, foundations/on-off-policy.md, instrumentation-and-data-readiness.md, and methods/peft.md — verified resolvable, not re-authored here (this pass is nav/reference wiring only, not chapter content). Merged ~50 new citations into References via three new sections (“Is the recipe a loop?,” “Ordering rules: interleaving stages & fixing N problems,” and “Data mixing, replay ratios & capability forgetting”) plus one small “Start here: a proven-first ranking” addendum, deduped against the existing ~340-citation corpus (includes two non-arXiv sources — the DMT OpenReview paper and Thinking Machines Lab’s “LoRA Without Regret” post). Verified during integration, not re-litigated: the 2507.10616 (“Scalpel vs. Hammer”) citation in the new ordering-rules chapter is used as one of three converging, independent mechanisms (alongside CHORD and RL’s Razor) rather than as a standalone load-bearing claim — consistent with its existing LOW-confidence/contested framing in decision.md, methods/reinforcement.md, and diagnosis/framework.md; the reward-tampering-vs-proxy-gaming distinction already established in contested.md/decomposition-vs-monolithic.md/roadmap-inputs.md (Denison et al., arXiv:2406.10162) is unaffected by, and not contradicted by, any claim in the four new chapters.

  • 2026-07-02 — v0.4. Wired three new chapters into the “Toward a frontier cybersecurity model” section, re-ordered to read as a coherent arc — organizing reframe first, then the two chapters it’s a prerequisite for, then the existing family→path→forks arc: The recipe is a sequence, not a pick (retires the “which technique” framing at the root — two explicit stage sequences, Sequence A from-scratch-foundation-model and Sequence B fine-tune-an-open-weight-dense-model [this project’s actual path], why order matters and stages compound rather than add, the synthetic-trajectory bootstrap and its off-policy execution-gap caveat, and a stage-wise evaluation protocol for Sequence B), Continued pretraining on an instruction-tuned model (can you run raw CPT directly on an already-instruct/RLHF’d checkpoint without destroying it — yes, but naive CPT-on-instruct reliably causes format/alignment collapse, not fact erasure; a six-technique preservation decision table; recommends CPT-on-base→re-instruct as the default with chat-vector reattachment as a cheap fallback; an IFEval+MMLU+domain-QA stage-boundary gate to verify it didn’t break), and Proven post-training datasets — a usage-cited registry (a ~60-dataset registry across instruction/chat SFT, tool/function-calling, preference, reasoning/CoT, willingness/refusal-calibration, and Chinese-labs/multilingual — every row proven-by-usage in a named shipped model/recipe, never a single-paper-only academic artifact, mapped onto Sequence B’s actual stage order). Cross-linked frontier-cyber-model-path.md and roadmap-inputs.md to both the recipe-sequence and dataset-registry chapters where their existing arguments (Stage 1 SFT/data synthesis, fork (b)’s SFT-now-vs-measure-first) are specific instances of the general point. Merged ~90 new citations into References via two new sections (“Post-training recipe as a sequence — order, compounding, synthetic-trajectory bootstrap” and “Continued pretraining on an instruction-tuned model — preservation techniques”) plus a new “Datasets (proven-by-usage)” subsection for the dataset-card/model-card links, deduped against the existing corpus.

  • 2026-07-02 — v0.3. Wired three new chapters into the book: Before you train — instrumentation & data readiness (what the harness already emits vs. the minimal per-stage-verifier gap to instrument, grounded in a direct source read of go/libs/agent/events + the flag-verification pipeline), One problem, or many? — monolithic vs decomposed (the eval-decomposition-vs-training-decomposition split, the potential-based-shaping safety net, the verdict for this project), and Where you are & the forks ahead (the capstone — five forks, a dependency DAG, seven falsifiable hypotheses, placed last before References). Applied the project’s standing no-academic-cybersecurity-LLM-as-research-basis stance across all three: every CTF-Dojo/Cyber-Zero/Pentest-R1/HackSynth/AutoPenBench/DRLRM-PT-style citation is labelled “academic, cited for context — not a basis for our decisions,” with load-bearing claims re-anchored on frontier-lab disclosures, general RL/ML theory, or this project’s own measured data. Merged ~35 new citations into References (new Hierarchical RL/decomposition/reward-shaping section; additions to Exploration & entropy collapse, CTF/pentest RL environments, Agent benchmarks & failure taxonomies, and Capability boundary sections) and added the CTF/pentest-RL section’s context-only header note.

    Later same day — added the frontier north-star section. Wired the two standing capstone chapters into a new top-level section, “Toward a frontier cybersecurity model,” placed after “In practice” and before References: Cybersecurity is one of a family — what cracked the others (the cross-domain structural-analogy survey — six adjacent long-horizon/sparse-reward/verifiable domains and what actually cracked each), The path to a frontier cybersecurity model (the capstone recipe + gap analysis — what “frontier” costs beyond this project’s own portfolio), and moved Where you are & the forks ahead into this new section as its final chapter (out of “In practice”), completing the arc family → path → your forks. Cross-linked roadmap-inputs.md at top and in its Cross-links section to both new chapters. Merged the new chapters’ citations into References via two new sections — “Domain-specialization lineages (code/math/medical)” and “Adjacent-domain structural transfer” — ~65 new arXiv/DOI/PMC ids, deduped against the existing ~250-citation corpus, academic-security entries kept labelled context-only. Salvaged three of the higher-value ideas the standing academic-cybersecurity-LLM stance would otherwise have excluded, by re-grounding each on independent frontier-lab or general-RL-theory evidence instead: (1) staged/kill-chain reward shaping — salvaged as the theorem-backed potential-based form only (Ng-Harada-Russell, ICML 1999), never the flat per-stage bonus academic pentest-RL papers use; (2) subgoal/curriculum decomposition of a long episode — salvaged via DeepSeek-Prover-V2 and AlphaGeometry (cold-start SFT data generation only, never densifying the RL reward itself); (3) failure-corpus-to-curriculum conversion — salvaged via WebRL’s self-evolving curriculum and HER’s relabeling principle (mining flag=0 trajectories for sub-skill SFT data), not any academic CTF-RL paper’s claim.

  • 2026-07-02 — v0.2. New Diagnosis section: Diagnosing the gap — a scientific framework (the pass@k crossover protocol, Cover@τ, sandbagging/elicitation tests — is a low k=1 solve rate an execution gap or a knowledge gap, before betting a GRPO run on the answer) and From behavioral audit to training signal (maps a commonly-observed agentic-pentest failure profile — tool avoidance, no methodology, brittle single-guess, uneven PTES phases, benchmarks-measure-speed-not-thoroughness — onto the specific post-training techniques designed to fix each). New method chapter RL that creates value — long-horizon · exploration · reasoning · novelty, a ~50-paper sweep tagged [L]/[E]/[R]/[N] against this project’s own diagnosis (GiGPO, DAPO/Clip-Cov, ProRL, ReTool/ToRL/Search-R1, CTF-Dojo/Pentest-R1/HackSynth, pass@k-is-diagnostic-not-objective). Extended reinforcement.md and agentic-rl.md with cross-links into the sweep. Rewrote What the frontier labs actually do with a full last-year (2025-07→2026-07) refresh across all 10 tracked labs, filling in previously-thin xAI/Grok, Mistral (Magistral/Ministral), Zhipu/GLM, Xiaomi/MiMo, and deepening Kimi K2→K2.5→K2.6, each now carrying the same [L]/[E]/[R]/[N] tags. Extended Contested edges and The decision with the new capability-boundary and sandbagging/elicitation literature. Merged ~130 newly-cited sources into References.

  • 2026-07-02 — v0.1. Initial build from a live session: the on/off-policy foundation, the method genealogy (imitation/preference/reinforcement + agentic RL + PEFT), verified 2026 frontier-lab recipes, the method→data reframe, the decision tree, and contested edges. Embeds the interactive decision journey.

Backlog (next sessions)

  • A worked example: filtering your verified solves into a rejection-sampling SFT set (the verified-trajectory pipeline).
  • The reward-function chapter: building an ungameable verify(state) for CTF flags (state, not transcript).
  • Pass@k methodology: per-challenge bucketing before choosing a branch.
  • The train↔inference precision-mismatch rabbit hole (TIS vs FP16), for when you reach GRPO.
  • Harness-shape coupling: ruling out “it’s the scaffold, not the model” before fine-tuning.