Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

References

Every id below was crawl-verified during the sessions that built this book (title/authors/date confirmed on the arXiv abstract page). Lab blogs/tech reports are linked to their source. Citation counts are unreliable for <18-month-old work — venue/lab-report presence is the stronger signal.

Foundations & imitation

Preference

Reinforcement

Long-horizon & multi-turn agentic RL — credit assignment across turns, not tokens

  • GiGPO (step-level advantage from state-hash-matched steps across rollouts, zero extra rollouts) — arXiv:2505.10978
  • ArCHer (two-timescale: off-policy turn-level critic + on-policy token-level PG) — arXiv:2402.19446
  • RAGEN / StarPO (multi-turn agentic RL framework, state-thinking-action loop) — arXiv:2504.20073
  • Turn-Level Reward Design (dense per-turn reward layered under a terminal reward) — arXiv:2505.11821
  • Turn-PPO (turn as the MDP unit, not token or trajectory) — arXiv:2512.17008
  • Demystifying RL for Long-Horizon Tool-Using Agents (5-axis systematic ablation: reward/scale/data/algorithm/environment) — arXiv:2603.21972
  • Verlog (dual-discount GAE, memory-windowing, validated to 400+ turn episodes) — no arXiv id, cite OpenReview:GmodkWwMV3
  • Kimi k1.5 (128k-context RL via partial-rollout checkpoint/resume, no MCTS/value-fn/PRM) — arXiv:2501.12599
  • AgentGym-RL / ScalingInter-RL (horizon curriculum: short turn cap expanding to full budget over training) — arXiv:2509.08755
  • MUA-RL (trains against a dynamic, LLM-simulated counterpart instead of a static script) — arXiv:2508.18669
  • HiPER (hierarchical credit assignment) — arXiv:2602.16165 · Hindsight Credit Assignment for Long-Horizon LLM Agents — arXiv:2603.08754
  • RL-PLUS (names “capability boundary collapse” — pass@k at large k dropping even as pass@1 rises under RLVR) — arXiv:2508.00222

Hierarchical RL, decomposition & potential-based reward shaping — “one problem, or many?”

  • Sutton — “The Bitter Lesson” (hand-built structure plateaus, general search+learning wins at scale; intellectual ancestor of the monolithic-outcome-RL case) — no arXiv id, incompleteideas.net (2019)
  • OpenAI Deep Research system card (long-horizon tool-using agent trained end-to-end on outcome/rubric reward; “end-to-end training beats manual orchestration”) — no arXiv id, OpenAI system card
  • Options / SMDP framework (Sutton, Precup, Singh — the seminal HRL / temporal-abstraction paper) — Artificial Intelligence 112 (1999), no arXiv id, DOI 10.1016/S0004-3702(99)00052-1
  • FeUdal Networks (Manager/Worker HRL, fixes option-collapse) — arXiv:1703.01161
  • HIRO (off-policy correction for HRL non-stationarity / subgoal ceiling-capping) — arXiv:1805.08296
  • Ng, Harada & Russell — potential-based reward shaping, F(s,a,s')=γΦ(s')−Φ(s) provably policy-invariant — ICML 1999, no arXiv id (predates arXiv’s routine ML use), ACM DL 10.5555/645528.657613
  • Müller & Kudenko (PBRS practical effectiveness still depends on potential scaling) — arXiv:2502.01307
  • RUDDER (learned, return-equivalent reward redistribution — a learned alternative to hand-specifying Φ) — arXiv:1806.07857
  • Go-Explore (pure outcome RL structurally fails on sparse/deceptive long-horizon tasks without explicit remember-and-return exploration) — arXiv:1901.10995 / Nature s41586-020-03157-9
  • OpenAI Five (long-horizon precedent needed huge scale + a per-frame shaped reward, not a single terminal bit) — arXiv:1912.06680
  • Credit Assignment survey (separates credit-assignment variance from exploration burden) — arXiv:2312.01072
  • MiRA (milestone-based dense reward; Gemma3-12B WebArena-Lite 6.4%→43.0%, beating WebRL/GPT-4-Turbo) — arXiv:2603.19685
  • Verifiable Process Rewards / VPR (safe ground-truth checklist process reward; own caveat that open/unstructured stages remain unsolved) — arXiv:2605.10325
  • CM2 (checklist-style verifiable sub-criteria reward) — arXiv:2602.12268
  • Curriculum Learning (Bengio et al. — foundational, order training by difficulty, touches no reward function) — ICML 2009, no arXiv id
  • h1 (curriculum + pure outcome-only reward yields an exponential sample-complexity gain) — arXiv:2510.07312
  • FastCuRL (context-length curriculum, entropy-collapse timing) — arXiv:2503.17287
  • BPO (curriculum + rejection-sampling refine; vanilla GRPO on sparse reward gains only marginally without curriculum) — arXiv:2508.03018
  • TIPS (turn-level potential shaping for search-augmented LLMs — shaping machinery directly on-point, domain is not) — arXiv:2603.22293
  • Randlov & Alstrom — the canonical non-potential-based “bicycle shaping” failure (agent farms a looks-like-progress bonus instead of reaching the goal) — ICML 1998, no arXiv id
  • Kill-chain-staged reward (cyber-defense red-teaming) (academic cybersecurity-LLM work — cited for context only, not a basis) — arXiv:2605.17075 (May 2026)
  • DRLRM-PT (reward machine over kill-chain phases, classical/non-LLM pentest RL) (academic — cited for context only, not a basis; explicitly named in the project’s standing rule) — arXiv:2405.15908 / DOI 10.1109/ijcnn60899.2024.10650368
  • Node-fragility reward shaping (classical dense-reward pentest, non-LLM regime) (academic — cited for context only, not a basis) — DOI 10.3390/electronics13214311

Tool-integrated / tool-use RL — fixes for a commonly-observed tool-avoidance failure mode

  • ReTool (trajectory-level tool-integrated RL) — arXiv:2504.11536
  • ToRL (tool-integrated RL, math) — arXiv:2503.23383
  • Search-R1 (RL for search-agent tool use) — arXiv:2503.09516
  • ToolRL (fine-grained, decomposed per-call tool-selection reward) — arXiv:2504.13958
  • Tool-Star (forced exposure to under-used tools via multi-tool synthesis pre-RL) — arXiv:2505.16410
  • Tool Preferences in Agentic LLMs are Unreliable (diagnosis of pattern-1-shaped tool avoidance) — arXiv:2505.18135

Exploration & entropy collapse

  • The Entropy Mechanism of RL for Reasoning LMs (R = -a·e^H + b; Clip-Cov/KL-Cov fixes) — arXiv:2505.22617
  • Beyond the 80/20 Rule (top-20%-entropy “forking tokens” carry nearly all exploration signal) — arXiv:2506.01939
  • Reasoning with Exploration: An Entropy Perspective — arXiv:2506.14758
  • Representation-Based Exploration for Language Models (hidden-state diversity bonus, usable at inference time) — arXiv:2510.11686
  • Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective — arXiv:2511.16231
  • Spurious Rewards (RLVR gains on Qwen2.5-Math nearly as large with completely wrong rewards; model-family-dependent) — arXiv:2506.10947
  • Absolute Zero (self-play task-proposal + solve, zero external labeled data) — arXiv:2505.03335
  • LIMO (817 curated SFT examples beat >100k loosely-curated ones — SFT as cognitive templates, not knowledge source) — arXiv:2502.03387
  • Test-time compute scaling (Snell et al., difficulty-adaptive allocation matches a 14x larger model) — arXiv:2408.03314 · o3-mini vs o1-mini (accuracy without longer CoT) — arXiv:2502.15631
  • OpenAI o1 System Card (methodology precedent, cited across the field) — arXiv:2412.16720
  • Reward-hacking-under-RL cluster: Specification Gaming in Reasoning Models — arXiv:2605.02269 · LLMs Gaming Verifiers (extensional vs intensional correctness) — arXiv:2604.15149 · Reward Hacking in the Era of Large Models (Proxy Compression Hypothesis) — arXiv:2604.13602
  • Per-step / process-reward hacking convergence (the case against naive per-stage reward): PURE / Stop Summation (summation-form credit assignment “easily induces LLMs to hack steps with high rewards”) — arXiv:2504.15275 · Reward Under Attack (SOTA PRMs as “fluency detectors rather than reasoning verifiers”) — arXiv:2603.06621 · Gao et al. (learned PRM/ORM + success reward can hurt vs success-only) — arXiv:2410.15115 · PRIME (authors’ own admission that process labels are “prohibitively expensive,” PRMs vulnerable to hacking) — arXiv:2502.01456 · MONA (multi-step reward hacking even when no single step looks bad to an overseer) — arXiv:2501.13011

Self-correction & tool-use self-correction RL

  • SCoRe — Training LMs to Self-Correct via RL (reward for improvement, not final correctness) — arXiv:2409.12917
  • From Correction to Mastery (earliest-error RL, distinct “SCoRe”) — arXiv:2509.14257

Post-training recipe as a sequence — order, compounding, synthetic-trajectory bootstrap

(new citations from The recipe is a sequence, not a pick; ids already covered elsewhere — Llama 3, Tülu 3, OLMo 2, DeepSeek-V3/R1, Qwen3, GRPO/DeepSeekMath, LoRA-learns-less, STaR, RAFT, ReST-EM, DAPO, the rejection-sampling→REINFORCE entropy-collapse paper, “Scalpel vs Hammer” — are not repeated here.)

  • Self-Instruct (off-policy synthetic instruction generation) — arXiv:2212.10560
  • WizardLM / Evol-Instruct (off-policy synthetic, complexity-evolved instructions) — arXiv:2304.12244
  • Llama 2 (RLHF report; iterative-round rejection-sampling non-monotonicity — “struggled more… to compose rhyming lines” when only the latest round was sampled) — arXiv:2307.09288
  • Persona-driven synthetic data generation — arXiv:2406.20094
  • Qwen2.5 (dense 0.5B–72B; explicit SFT→offline-DPO→online-GRPO staging) — arXiv:2412.15115
  • “SFT Memorizes, RL Generalizes” (GeneralPoints/V-IRL testbed; SFT stabilizes format, RL then generalizes) — arXiv:2501.17161 (ICML 2025 poster)
  • Llama-Nemotron (post-training recipe report) — arXiv:2505.00949
  • Distillation-vs-pattern-imitation ablation (“only the DeepSeek model shows a meaningful increase in capability” — new knowledge, not pattern transfer, is what expands pass@k) — arXiv:2505.14216
  • AdaSTaR (efficient iterative rejection-sampling; curriculum sampling, −58.6% training FLOPs at equal-or-better accuracy) — arXiv:2505.16322
  • SFT-vs-RFT forgetting comparison (SFT 52.1%→40.1% drop vs. RFT 54.2% improvement on the same setting) — arXiv:2507.05386
  • “RL Fine-Tuning Heals OOD Forgetting in SFT” (contested re-framing of “SFT memorizes/RL generalizes” as “SFT forgets, RL recovers”) — arXiv:2509.12235
  • Domain-continual pretraining forgetting / backward-transfer at scale (“moderate forgetting, low-to-moderate backward transfer”) — arXiv:2510.17776
  • Backward-synthesis answer-anchoring / confabulation risk (STaR-rationalization follow-up; the answer acts as a cognitive anchor) — arXiv:2602.14469
  • “Revisiting DAgger in the Era of LLM-Agents” (SFT’s off-policy covariate shift vs. RLVR’s on-policy but sparse feedback, stated precisely for multi-turn agents) — arXiv:2605.12913
  • Qwen3-4B SFT degrading TruthfulQA/HaluEval (Qwen-family-specific forgetting evidence) — arXiv:2605.20005

Is the recipe a loop? — round-trip revisits, non-commutativity, loop-exit criteria

(new citations from Is the recipe a loop?; ids already covered elsewhere — DeepSeek-R1, Llama 3, STaR, ReST-EM, the non-decoupling proof 2601.07389, Task Arithmetic 2212.04089, TIES 2306.01708 — are not repeated here.)

  • Self-Rewarding Language Models (iterative DPO, judge+generator both re-derived from the current policy each round) — arXiv:2401.10020
  • Apple AFM iTeC (committee of RS/DPO/IPO/online-RL variants, round-over-round evaluation picks the propagating optimizer) — arXiv:2407.21075
  • Havrilla et al. — Teaching LLMs to Reason with RL (EI/RCRL: generate with SFT checkpoint, reset training to base — model-resetting “crucial for best performance”; also the n=5-until-saturation loop-exit rule) — arXiv:2403.04642
  • “Iterative Finetuning is Mostly Idempotent” (U Chicago; trait amplification vanishes when models are reinitialized each cycle) — arXiv:2605.01130
  • WARM (souped reward models) — arXiv:2401.12187 · WARP (merges inside every RL iteration: EMA anchor + spherical interpolation + linear interpolation to pretrained init) — arXiv:2406.16768
  • Model Soups (weight-averaging independently fine-tuned models often beats picking the single best) — arXiv:2203.05482
  • Overoptimization dynamics across iterations of iterated RLHF (overoptimization itself decreases each round; performance gains still diminish) — arXiv:2505.18126
  • Self-rewarding iterative DPO diminishing-gains shape (“accumulated bias in the reward system”) — arXiv:2410.12735
  • Off-policy preference optimization’s distributional gap between the data-collection policy and the target policy — arXiv:2406.11827
  • Multi-round online DPO, resampling from the updated policy each round — a principled justification, not a heuristic — arXiv:2506.04272
  • 20+-model RLHF scaling study (marginal gains despite increasing training reward — the loop-exit diminishing-returns signal) — arXiv:2412.06000
  • Niu, Bai, Han, Zhang — On the Non-decoupling of SFT and RL in Post-training (formal proof: SFT-then-RL and RL-then-SFT each provably degrade what the prior stage converged to, two distinct theorems) — arXiv:2601.07389

Ordering rules: interleaving stages & fixing N problems — provenance-based safety, N-problem batching

(new citations from Ordering rules: interleaving stages & fixing N problems; ids already covered elsewhere — DAgger 1011.0686, R1 2501.12948, STaR 2203.14465, Llama 4 blog, Tülu 3 2411.15124, Llama 2 2307.09288, catastrophic-forgetting 2308.08747, the non-decoupling proof 2601.07389, “Scalpel vs Hammer” 2507.10616, Task Arithmetic 2212.04089, Model Soups 2203.05482 — are not repeated here.)

  • Shenfeld, Pari, Agrawal — RL’s Razor (RL implicitly stays KL-close to the policy that generated its reward signal) — arXiv:2509.04259
  • CHORD (Zhang et al., Alibaba; foreign expert data onto a drifted policy produces a “shift → readapt → overfit” curve) — arXiv:2508.11408
  • “RL Is Neither a Panacea Nor a Mirage” (RL-FT restores moderate SFT-induced OOD damage but not a checkpoint pushed into a markedly different representation regime) — arXiv:2508.16546
  • “Towards On-Policy SFT” (no improved SFT strategy fully eliminates forgetting once the data distribution deviates from the model’s own) — arXiv:2602.12222
  • “Mind the Gap” (have the current model rewrite a foreign demonstration into its own voice before training on it) — arXiv:2509.15157
  • PEAR (down-weight SFT tokens implausible under the current policy — loss-weighting version of the same data-alignment fix) — arXiv:2602.01058
  • Forgetting is biased, not uniform, across categories under sequential fine-tuning — arXiv:2412.16469
  • de Masson d’Autume et al. — Episodic Memory in Lifelong Language Learning (classical experience-replay mechanism) — arXiv:1906.01076
  • On-Policy Replay / OPR (10% on-policy replay budget lifts sequential-SFT backward-transfer from −13.93 to −0.65 on Qwen2.5-7B-Instruct) — arXiv:2605.29495
  • Excessive SFT measurably reduces subsequent RL plasticity (over-confident, sharper output distributions harder for RL to reshape) — arXiv:2606.09932
  • DMT (mixing specialized skills first, small general-ability replay slice added at the end — mix-within-a-stage-plus-final-replay recipe) — no arXiv id, OpenReview:6M5G5hNiAU

Data mixing, replay ratios & capability forgetting — reasoning-trace collapse, LoRA behavior-vs-magnitude

(new citations from Data mixing, ratios & not forgetting how to think; ids already covered elsewhere — DAgger 1011.0686, GKD 2306.13649, Biderman “LoRA Learns Less and Forgets Less” 2405.09673, “Scalpel vs Hammer” 2507.10616, Task Arithmetic 2212.04089, TIES-Merging 2306.01708, DARE 2311.03099, Scialom et al. 2205.12393, Ibrahim et al. 2403.08763, InstructGPT alignment-tax 2203.02155, R1 2501.12948 — are not repeated here.)

  • Twist, Yannakoudakis, Zhang (King’s College London) — “Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning” (names the phenomenon; answer-only accuracy hides the collapse; loss-masking fix) — arXiv:2605.21127
  • Lobo, Agarwal, Lakkaraju — “On the Impact of Fine-Tuning on Chain-of-Thought Reasoning” (NAACL 2025; SFT on non-reasoning data reduces CoT accuracy and faithfulness, worse in smaller models) — arXiv:2411.15382
  • Zhang, Lin, Rajmohan, Zhang (Microsoft) — “From Reasoning to Answer” (Reasoning-Focus Heads; activation patching on reasoning tokens causally alters the final answer) — arXiv:2509.23676
  • Zhang, Morris, Shmatikov (Cornell Tech, ICML 2026) — fine-tuning with vs. without reasoning traces present, large measured MATH500/JEEBench deltas — arXiv:2603.07267
  • Zhu, Zhang, Wang, Xu, Lyu, Wu — “To Think or Not to Think” (empty <think></think> block flips straight to the final answer; the same behavior is backdoorable during ordinary SFT/DPO) — arXiv:2502.12202
  • Bengio et al. — scheduled sampling / exposure bias (the sequence-level, purely-supervised restatement of the DAgger covariate-shift result) — arXiv:1506.03099
  • Zhou et al. (Meta AI) — LIMA (65B LLaMA SFT’d on 1,000 curated pairs; Superficial Alignment Hypothesis — format is cheap to (un)learn) — arXiv:2305.11206
  • Lermen, Rogers-Smith, Ladish — “LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B” (QLoRA takes refusal rate to ~1% while preserving general-capability benchmarks) — arXiv:2310.20624
  • Shuttleworth, Andreas, Torralba, Sharma (MIT) — “LoRA vs Full Fine-tuning: An Illusion of Equivalence” (LoRA develops “intruder dimensions” full fine-tuning does not; forgetting is concentrated there and worsens across sequential LoRA rounds) — arXiv:2410.21228
  • “Representation Collapse in Sequential Post-Training of Large Language Models” (LoRA updates from different stages occupy overlapping subspaces; long-CoT tuning carves a separable reasoning-format manifold) — arXiv:2605.30524
  • Kalajdzievski — rsLoRA, rank-stabilization scaling factor (γ = α/√r; now use_rslora in HF PEFT) — arXiv:2312.03732
  • PiSSA (learning-capacity accelerant; moves LoRA behavior closer to full fine-tuning’s) — arXiv:2404.02948
  • DoReMi (Xie et al., NeurIPS 2023; Group-DRO domain weights beat the default heuristic mixture by 6.5 points) — arXiv:2305.10429
  • RegMix (Liu et al., ICLR 2025 Spotlight; “domains interact in complex ways often contradicting common sense”) — arXiv:2407.01492
  • Scaling Laws for Optimal Data Mixtures (Shukor et al., Apple, NeurIPS 2025; analytically solves for optimal domain weights given a budget) — arXiv:2507.09404
  • Tülu 2 (explicit downsample of oversized FLAN, dropped Dolly entirely for hurting the mix average) — arXiv:2311.10702
  • GeRe (small fixed general-sample replay set resolves both general-capability and task-specific forgetting) — arXiv:2508.04676
  • Marek, Cho, Qiu, Chunara, Izmailov, Wilson (NYU) — self-generated replay (sampling the model’s own completions before fine-tuning nearly eliminates forgetting, only with spare capacity) — arXiv:2605.26097
  • Spiegelhalter, Franke, Hutter (NeurIPS 2025 workshop) — replay-ratio sweep, “more than 5–10% replay is not necessary for general knowledge retention” — arXiv:2510.11842
  • Kotha & Liang — generic-distribution replay can improve, not just preserve, target-task data efficiency — arXiv:2603.04964
  • Zheng et al. — spurious-forgetting result, old-task performance restored by briefly training on ~10 anchor/alignment instances (ICLR 2025) — arXiv:2501.13453
  • Maximal-Update Adaptation (formalizes why the optimal LoRA LR moves with rank) — arXiv:2602.06204
  • Thinking Machines Lab — “LoRA Without Regret” (frontier-lab operational report; optimal LoRA LR ≈10× full-FT LR, both SFT and RL) — no arXiv id, thinkingmachines.ai/blog/lora (2025)

The kinds of SFT & Method → Data — data selection, decontamination, on-policy distillation

(new citations from The kinds of SFT — it is the data, not the algorithm and Method → Data (your real bottleneck); ids already covered elsewhere — Self-Instruct 2212.10560, WizardLM/Evol-Instruct 2304.12244, phi-1 2306.11644, Hinton distillation 1503.02531, GKD 2306.13649, RAFT 2304.06767, STaR 2203.14465, ReST 2308.08998, ReST-EM 2312.06585, AgentInstruct/AgentLM 2310.12823, DeepSeek-R1 2501.12948, LIMA 2305.11206, LoRA Learns Less and Forgets Less 2405.09673, Llama 2 2307.09288, Gorilla 2305.15334, ToolBench/ToolLLaMA 2307.16789, InstructGPT 2203.02155 — are not repeated here.)

  • Kim & Rush — Sequence-Level Knowledge Distillation (teacher-executed transcript distillation) — arXiv:1606.07947
  • FireAct (formalized agentic-trajectory SFT) — arXiv:2310.05915
  • AlpaGasus (LLM-judge filtering beats full-dataset training) — arXiv:2307.08701
  • Cherry_LLM / IFD selection (cheap self-guided intrinsic quality filter) — arXiv:2308.12032
  • LESS (targeted, capability-specific data selection) — arXiv:2402.04333
  • DEITA — What Makes Good Data for Alignment? (complexity/quality/diversity selection axes) — arXiv:2312.15685
  • Béthune et al. — Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection (replay-slice mitigation, forgetting mechanism) — arXiv:2502.06042
  • Always Learning, Always Mixing (~10% replay as a practical mitigation baseline) — arXiv:2605.15220
  • Benchmark Data Contamination of LLMs: A Survey — arXiv:2406.04244
  • LLMSanitize (contamination detection tooling) — arXiv:2404.00699
  • Rethinking On-Policy Distillation of LLMs (on-policy distillation only helps with genuinely new teacher signal) — arXiv:2604.13016

Start here: a proven-first ranking of the methods — tiering metric, method-selection

(new citation from Start here: a proven-first ranking of the methods; this chapter is otherwise a recombination — under a proven-by-usage tiering metric — of methods already cited above: GRPO 2402.03300, RLVR/R1 2501.12948, PPO 1707.06347, DAPO 2503.14476, GSPO 2507.18071, GiGPO 2505.10978, PRM 2305.20050, DPO 2305.18290, KTO 2402.01306, InstructGPT 2203.02155, ORPO 2403.07691, SimPO 2405.14734, IPO 2310.12036, Constitutional AI/RLAIF 2212.08073, STaR 2203.14465, RAFT 2304.06767, ReST 2308.08998, Hinton distillation 1503.02531, GKD 2306.13649, WizardLM/Evol-Instruct 2304.12244, Llama 2 2307.09288, Tülu 3 2411.15124, Qwen3 2505.09388, Llama-Nemotron 2505.00949 — not repeated here.)

  • Gemma 2 (flagship-disclosed teacher-distillation production use, cross-references Gemini 1.5) — arXiv:2408.00118

Continued pretraining on an instruction-tuned model — preservation techniques

(new citations from Continued pretraining on an instruction-tuned model; LoRA-learns-less-forgets-less [2405.09673] already cited above, not repeated.)

  • DAPT — Gururangan et al., “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks” (ACL 2020, seminal domain-adaptive-pretraining concept) — arXiv:2004.10964
  • MMLU — Hendrycks et al. — arXiv:2009.03300
  • Scialom et al. — replay mitigates forgetting in continual instruction-tuning (EMNLP 2022) — arXiv:2205.12393
  • Ilharco et al. — “Editing Models with Task Arithmetic” (ICLR 2023, seminal task-vector/delta-arithmetic result) — arXiv:2212.04089
  • TIES-Merging (NeurIPS 2023; trim + elect-sign + merge) — arXiv:2306.01708
  • Gupta et al. — continual-pretraining LR re-warm/re-decay — arXiv:2308.04014
  • Luo et al. — empirical study of catastrophic forgetting in LLMs during continual fine-tuning (1B–14B; forgetting worsens with scale) — arXiv:2308.08747
  • AdaptLLM — auto-converts raw domain text into reading-comprehension/QA replay pairs — arXiv:2309.09530
  • Qi, Zeng et al. — “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” (ICLR 2024) — arXiv:2310.03693
  • Chat Vector (Huang et al., ACL 2024; language-shift instance of the task-arithmetic reattach recipe) — arXiv:2310.04799
  • DARE — “Super Mario” random delta-dropping + rescaling (ICML 2024) — arXiv:2311.03099
  • IFEval — “Instruction-Following Evaluation for Large Language Models” (Zhou et al., Google) — arXiv:2311.07911
  • LLaMA Pro — block-expansion CPT that never touches original weights, then a separate instruction-tuning pass (ACL 2024) — arXiv:2401.02415
  • Li & Lee — “Examining Forgetting in Continual Pre-training of Aligned Large Language Models” (direct CPT-on-Llama-2-7b-chat comparison) — arXiv:2401.03129
  • RESTA — DARE-sparsified delta subtraction/restoration (ACL 2024) — arXiv:2402.11746
  • Ibrahim et al. — simple/scalable continual-pretraining strategies (re-warm+re-decay+replay matches from-scratch retraining) — arXiv:2403.08763
  • Qi et al. — “Safety Alignment Should Be Made More Than Just a Few Tokens Deep” (ICLR 2025 Oral; shallow safety-alignment mechanism) — arXiv:2406.05946
  • Instruction Pre-Training (Microsoft, EMNLP 2024; 200M synthesized instruction-response pairs woven into raw CPT corpus) — arXiv:2406.14491
  • Jindal, Badrinath, Bharti, Vinay & Sharma (Samsung Research) — “Balancing Continuous Pre-Training and Instruction Fine-Tuning” (the direct S1-vs-S2 CPT-on-instruct-vs-base comparison, 4 model families) — arXiv:2410.10739
  • Mousavi, Alghisi & Riccardi (U. Trento) — “What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs” (loss curves don’t reveal instruct-layer damage in real time) — arXiv:2601.03858
  • Zheng, Cai, Qiu & Ma — “Spurious Forgetting in Continual Learning of Language Models” (ICLR 2025 poster; forgetting is often a task-alignment/metric artifact, not true knowledge loss) — no arXiv id, OpenReview:ScI7IlKGdI
  • Harmon, Hochlehnert, Bethge & Prabhu (Tübingen AI Center) — “Mapping Post-Training Forgetting in Language Models at Scale” (~30 model pairs; “model merging does not reliably mitigate forgetting”) — no arXiv id found, anonymous ICLR 2026 submission, OpenReview:qCIg2WGudx

Datasets (proven-by-usage) — general post-training data, Sequence-B rungs

Full registry, inclusion rule, and per-dataset detail: Proven post-training datasets — a usage-cited registry. Papers backing named training recipes for these datasets (Tülu 3, OLMo 2, Qwen2.5-Math, DPO, KTO, IPO, ORPO already cited above, not repeated):

Case study: how coding engineered its data — the 4-rung execution-in-the-loop ladder

(new citations from Case study: how coding engineered its data (and what transfers to cyber); ids already covered elsewhere — StarCoder 2305.06161, StarCoder2/Stack v2 2402.19173, DeepSeek-Coder 2401.14196, Phi-1 2306.11644, Self-Instruct 2212.10560, CodeRL 2207.01780, PPOCoder 2301.13816, RLTF 2307.04349, StepCoder 2402.01391, SWE-Gym 2412.21139, R2E-Gym/SYNGEN 2504.07164, SWE-RL 2502.18449 — are not repeated here.)

  • The Stack (permissively-licensed GitHub corpus, 3.1→6.4TB) — arXiv:2211.15533
  • Code Llama (near-deduped GitHub pretraining, ~500B–1T tokens) — arXiv:2308.12950
  • Bavarian et al. — Fill-in-the-Middle (FIM span-corruption training objective) — arXiv:2207.14255
  • WizardCoder / Evol-Instruct (iterative complexity-mutation instruction synthesis, benchmark-pass@1-gated) — arXiv:2306.08568
  • Magicoder / OSS-Instruct (real-code-snippet-grounded synthetic instruction generation) — arXiv:2312.02120
  • CodeT (dual solution+test execution consensus, RANSAC-style) — arXiv:2207.10397
  • LEVER (execution-result-verified program synthesis) — arXiv:2302.08468
  • GenX (execution-verified rejection-sampling data generation) — arXiv:2412.13464
  • KodCode (execution-verified synthetic coding dataset at scale) — arXiv:2503.02951
  • SOL-VER (execution-verified solution generation) — arXiv:2502.14948
  • Self-Debugging (execution-error-message as part of the training row) — arXiv:2304.05128
  • SWE-bench (fail-to-pass execution oracle, 3-stage cascade denoising) — arXiv:2310.06770
  • SWE-MiniSandbox (kernel-level isolation, no full container; ~25% of Docker’s setup time, ~5% disk) — arXiv:2602.11210
  • SWE-World (learned Docker-free execution surrogate; surrogate+best-of-8 exceeds real-Docker baseline, 5–10% accuracy regression vs. ground truth) — arXiv:2602.03419
  • VeriScale (adversarial mutation-testing at scale; 83× test-suite expansion drops scores from 70–85% to 20–40%, exposing weak-verifier reward hacking) — arXiv:2605.22368
  • SpecBench-style analysis (reward-hacking gap grows ~28pp per 10× code-size increase; long-horizon visible-vs-hidden-test divergence) — arXiv:2605.21384 (not independently re-crawled this pass)
  • SWE-Bench Illusion (memorization vs. reasoning; 76% vs. 53% file-localization gap, 15–30% inflated SWE-Bench-Verified gains) — arXiv:2506.12286
  • Metamorphic testing for memorization diagnosis (semantics-preserving transforms; NLL-correlated post-transform collapse) — arXiv:2604.21579 (not independently re-crawled this pass)
  • BountyBench (CVE exploitation jumps ~5%→57.5–90% once vulnerability location is given — exploitation-as-retrieval evidence) — arXiv:2505.15216
  • SWE-Factory (automates the remaining manual environment-setup stages) — arXiv:2506.10954
  • SWE-Universe (million-scale environment generation with a hacking detector) — arXiv:2602.02361
  • Seed-Coder (replaces 100+ hand-crafted pretraining-data heuristics; execution-as-reward confirmation at scale) — arXiv:2506.03524
  • RepoZero (agent regenerates an existing package as its own test-case oracle; 30–55% pass rate, still Rung-3-with-a-harder-task) — arXiv:2605.07122
  • ProgramBench (fuzzes a gold compiled reference to reconstruct it; zero tasks fully resolved across 200 instances) — arXiv:2605.03546
  • CodeAlchemy (most extensively execution-verified attempt at a bootstrapped/real-code-seed-free Rung-4 pipeline) — arXiv:2606.10087
  • Output-diversity collapse under recursive synthetic training (independent reason a bootstrapped-only data pipeline degrades) — arXiv:2603.12683 (not independently re-crawled this pass)
  • CVE-Factory (three-stage multi-agent CVE→Docker pipeline; 95% solution correctness, 96% environment fidelity; produces LiveCVEBench) — arXiv:2602.03012
  • ARVO (large-scale vulnerable-target reconstruction/reproduction dataset) — arXiv:2408.02153
  • SoK: DARPA AIxCC (systematizes the 2023–2025 AI Cyber Challenge; patch/exploit semantic-correctness still substantially unsolved) — arXiv:2602.07666
  • All You Need Is A Fuzzing Brain (AIxCC runner-up; pure fuzzing+LLM system, 28 vulnerabilities found incl. 6 novel zero-days, 14 validated patches) — arXiv:2509.07225
  • CVE-Bench (evaluates exploitation rather than patching directly) — arXiv:2503.17332

Hint-guided bootstrapping — hint-then-mask, context distillation, LUPI

(new citations from Hint-guided bootstrapping — put the walkthrough in the prompt, then train it away; ids already covered elsewhere — STaR 2203.14465, HER 1707.01495, Go-Explore original 1901.10995, NuRL 2509.25666, Reward Hacking survey 2604.13602, Peng et al./RCG/SSR 2602.14469 — are not repeated here.)

  • TRICE (Phan et al.) — formalizes STaR-rationalization as biased stochastic-EM; gradient shrinks toward zero on the hardest, most-hint-dependent examples — arXiv:2312.02179
  • V-STaR (Hosseini et al.) — discarded hinted-failures reused as negative signal, not pure waste — arXiv:2402.06457
  • Askell et al. — coins “context distillation” for LLM alignment — arXiv:2112.00861
  • Choi et al. — Prompt Injection / PING, independent formalization of teacher(w/ prompt)→student(no prompt) on synthetic pseudo-inputs — arXiv:2206.11349
  • Snell, Klein, Zhong — generalized context-distillation framework, the exact hint-then-mask loss — arXiv:2209.15189
  • Kujanpää, Valpola, Ilin — prompt distillation matches RAG-level knowledge injection — arXiv:2412.14964
  • OPCD (Ye et al.) — on-policy fix for exposure bias / mode-covering forward-KL; names “experiential knowledge distillation” — arXiv:2602.12275
  • Nicolicioiu, Pezeshki, Courville — on-policy self-distillation from a sampled demo still collapses output diversity / flattens pass@k — arXiv:2606.26091
  • OPSA (Fu et al.) — “teacher flip rate” diagnostic for whether privileged context converts failures or just elicits — arXiv:2605.15239
  • HiLL (Xia et al.) — names “advantage collapse” + a differentiable “hint reliance” metric; lower reliance ⇒ stronger transfer to no-hint policy — arXiv:2604.00698
  • ReGFT (Wu et al.) — self-generated, reference-guided trajectories beat raw-reference SFT — arXiv:2603.01223
  • Multi-level Stepwise Hints (Zhang et al.) — graded hint depths as a concrete fading mechanism — arXiv:2507.02841
  • E2H Reasoner (Parashar et al.) — easy-to-hard curricula prevent overfitting to the assistance signal — arXiv:2506.06632
  • AdaRFT (Shi et al.) — adaptive difficulty targeting, same frontier-tracking spirit as hint-fading — arXiv:2504.05520
  • Nair, McGrew, Andrychowicz, Zaremba, Abbeel — classical precedent for explicit demo-loss annealing (hint fading) — arXiv:1709.10089
  • Go-Explore (Nature version) — canonical citation for “exploit, then robustify” — arXiv:2004.12919
  • Asymmetric Actor-Critic (Pinto et al.) — LUPI applied to deep RL, critic sees privileged sim state, actor doesn’t — arXiv:1710.06542
  • Shortcut Learning in Deep Neural Networks (Geirhos et al.) — seminal Clever-Hans framing; shortcuts indistinguishable on i.i.d. data — arXiv:2004.07780
  • HANS (McCoy, Pavlick, Linzen) — canonical NLI shortcut-detection challenge set — arXiv:1902.01007
  • Clever Hans on COPA (Kavumba et al.) — same recipe, different architecture — genuine vs. shortcut learner — arXiv:1911.00225
  • RAWR (Feng et al.) — input-reduction detection technique for artifact exploitation — arXiv:1804.07781
  • Turpin, Michael, Perez, Bowman — biasing a prompt toward an answer produces unfaithful, non-load-bearing CoT — arXiv:2305.04388
  • Ross, Peters, Marasović — self-rationalization can itself become a new shortcut surface, especially ungrounded — arXiv:2210.13575
  • Countdown-Code (Khalifa et al.) — as little as 1% contaminated distillation data teaches reward-hacking behavior, amplified by later RL — arXiv:2603.07084

Teaching a tool, teaching recon — data objects for tool-use vs. search RL

(new citations from Teaching a tool, teaching recon — the data behind a skill; ids already covered elsewhere — Gorilla 2305.15334, ToolBench/ToolLLaMA 2307.16789, APIGen/xLAM-function-calling-60k 2406.18518, APIGen-MT 2504.03601, Search-R1 2503.09516, ToolRL 2504.13958, WebGPT 2112.09332 — are not repeated here.)

  • Toolformer (Schick et al.) — self-supervised single (call, result) insertion into raw text, kept only if loss drops by ≥ τ_f against the real API output — arXiv:2302.04761
  • DocPrompting (Zhou et al.) — joint retriever+generator conditioned on retrieved docs, doc trusted as ground truth, no execution — arXiv:2207.05987
  • xLAM / APIGen-MT multi-turn extension (Salesforce) — arXiv:2409.03215
  • Nemotron-Research-Tool-N1 (NVIDIA) — pure RL from a cold start outperformed SFT-then-RL; thick SFT seed on distilled reasoning traces overfits to style and hurts downstream RL — arXiv:2505.00024
  • TIER — trajectory-supervised reward (matching one authored search path) collapses past depth 4–6 by penalizing valid alternative orderings; the negative case for hand-authoring a recon decision tree — arXiv:2605.16790

Domain-specialization lineages — the frontier recipe (code / math / medical)

  • Kaplan et al. — Scaling Laws for Neural Language Models — arXiv:2001.08361
  • Hoffmann et al. — Chinchilla, compute-optimal scaling — arXiv:2203.15556
  • phi-1 (textbook-quality data substitutes for ~100x scale in a narrow domain) — arXiv:2306.11644
  • DeepSeek-Coder V2 (abandons from-scratch pretraining, continues from a strong base) — arXiv:2406.11931
  • Qwen2.5-Math (co-evolves RM + SFT data across rounds before RL, reuses RM for best-of-N at inference) — arXiv:2409.12122
  • DeepSeek-Prover-V2 (subgoal-decomposed cold-start data + kernel-verified RL) — arXiv:2504.21801
  • OLMo 2 (mid-training as a named 5-10% FLOPs bridge stage) — arXiv:2501.00656
  • Mid-training mechanism study (outperforms CPT-alone at matched budget, reduces catastrophic forgetting before SFT) — arXiv:2510.14865
  • Med-PaLM v1 (mentioned, not a basis for cyber claims — prompt-tuning-only ceiling, motivates v2) — arXiv:2212.13138
  • Med-PaLM 2 (domain instruction fine-tuning + ensemble refinement) — arXiv:2305.09617
  • MedGemma (domain VLM pretraining + task fine-tuning, explicitly not clinical-grade alone) — arXiv:2507.05201
  • Full fine-tuning vs. LoRA on code/math domain-skill acquisition (10-100x effective-rank gap) — arXiv:2405.09673
  • Benchmark contamination (GSM8K/MMLU scores inflated up to 22.9%/19.0%) — arXiv:2406.13990
  • LiveCodeBench (time-segmented, contamination-resistant-by-construction) — arXiv:2403.07974
  • GPQA (“Google-proof” QA) — arXiv:2311.12022
  • Tülu 3 (decontamination as a first-class deliverable; primary public naming of RLVR) — arXiv:2411.15124
  • Emerging RL-environment-scale bottleneck framing (moderate confidence, new/low-citation) — arXiv:2511.09586
  • SIMA (scalable instructable multiworld agent) — arXiv:2404.10179
  • SIMA 2 (self-generated tasks + rewards via Gemini) — arXiv:2512.04797

Adjacent-domain structural transfer — coding agents, competitive programming, theorem proving, web agents, games, robotics

  • SWE-agent / Agent-Computer Interface (fixed action set + concise feedback lifts pass@1 pre-RL) — arXiv:2405.15793
  • SWE-Gym (executable-environment SFT trajectories) — arXiv:2412.21139
  • R2E-Gym — arXiv:2504.07164
  • SWE-RL (execution-verified reward beats a difflib patch-similarity fallback) — arXiv:2502.18449
  • o1→o3 coding RL — arXiv:2502.06807
  • Progressive context/turn-budget curriculum for long-horizon RL — arXiv:2508.03501
  • SWE-Master (mask environment-feedback tokens out of the SFT loss, low-confidence/very recent) — arXiv:2602.03411
  • DeepSeek-Coder v1 (from-scratch pretraining, the lone exception in the lineage) — arXiv:2401.14196
  • StarCoder2 / The Stack v2 (curation quality substitutes for parameter count — data-pipeline lesson only) — arXiv:2402.19173
  • StarCoder — arXiv:2305.06161
  • CodeRL — arXiv:2207.01780
  • PPOCoder — arXiv:2301.13816
  • RLTF — arXiv:2307.04349
  • StepCoder — arXiv:2402.01391
  • RLEF (turn-level value function over a multi-turn POMDP; Meta FAIR, ICML 2025 spotlight) — arXiv:2410.02089
  • Sailor (SEA-language CPT) — arXiv:2404.03608
  • SEA-LION — arXiv:2504.05747
  • LLaMA Beyond English — arXiv:2401.01055
  • Tokenizer/vocabulary coverage as an architectural precondition — arXiv:2406.11477
  • BLOOM+1 (adding a new language to the SFT mixture beats continued pretraining) — arXiv:2212.09535
  • Aya — arXiv:2402.07827
  • AlphaCode (sampling breadth + cheap filter) — arXiv:2203.07814
  • GrandCode / Agentic GRPO (purpose-built GRPO variant for delayed reward + off-policy drift; single team, very recent) — arXiv:2604.02721
  • AlphaGeometry (Nature 2024; synthetic self-play manufactures its own training problems) — DOI 10.1038/s41586-023-06747-5
  • AlphaProof (Nature 2025; AlphaZero-style self-play/search on top) — DOI 10.1038/s41586-025-09833-y
  • WebGPT (learned human-preference reward; origin of the SFT-cold-start-then-RL recipe shape, flagged OOD-weak by its own authors) — arXiv:2112.09332
  • WebRL (self-evolving curriculum generated from the model’s own unsuccessful attempts; ICLR 2025) — arXiv:2411.02337
  • R1-Searcher (sequential, not summed, two-stage tool-use reward) — arXiv:2503.05592
  • DeepResearcher (training end-to-end in the real live environment is “a fundamental requirement”; EMNLP 2025) — arXiv:2504.03160
  • AlphaStar (Nature; league-based diverse self-play population fixes strategy collapse) — DOI 10.1038/s41586-019-1724-z
  • NetHack (honest “still unsolved” calibration point) — arXiv:2006.13760
  • HER — Hindsight Experience Replay (relabel a failed trajectory as the goal it accidentally satisfied; NeurIPS 2017) — arXiv:1707.01495
  • Firestone — competence vs. performance (formal/functional split; independently reconfirmed by the particular-language literature) — PMC7604508
  • KLong (progressive horizon curriculum, second converging source; low-confidence) — arXiv:2602.17547
  • BEACON (2026 long-horizon credit-assignment cluster; low-confidence individually) — arXiv:2605.06078
  • Ecpo (2026 long-horizon credit-assignment cluster; low-confidence individually) — arXiv:2606.05885

CTF / pentest RL environments

(academic, cited for context — not a basis for our decisions; see the stance in Contested edges)

  • CTF-Dojo (486 verified trajectories, 31.9% pass@1 credibility yardstick) — arXiv:2508.18370
  • Cyber-Zero (monolithic outcome-RL, simulated env, +13.1%) — arXiv:2508.00910
  • Pentest-R1 (two-stage RL for CTF methodology) — arXiv:2508.07382
  • HackSynth (crypto-CTF GRPO) — arXiv:2506.02048
  • InterCode-CTF (seminal monolithic-reward CTF environment) — arXiv:2306.14898
  • Cybench (subtask decomposition, eval-only) — arXiv:2408.08926
  • AutoPenBench (milestone taxonomy near-matching this book’s F1–F4 split, eval-only) — arXiv:2410.03225
  • NYU CTF Bench (CTF benchmark family) — arXiv:2406.05590
  • EnIGMA (“soliloquizing” fabrication failure mode, ICML 2025) — arXiv:2409.16165
  • Guided Reasoning via Structured Attack Trees (deterministic ATT&CK-derived task tree, +5x subtask completion on the same weights) — arXiv:2509.07939
  • From Capabilities to Performance (pentesting ablations) — arXiv:2509.14289
  • PentestAgent (RAG-fix framing of a knowledge gap; contested against the scaffolding/execution readings above) — arXiv:2411.05185
  • Capture the Flags: Family-Based Evaluation via Semantics-Preserving Transformations (CTF-specific robustness benchmark) — arXiv:2602.05523
  • What Makes a Good LLM Agent for Real-world Penetration Testing? (Task Difficulty Assessment + Evidence-Guided Attack Tree Search) — arXiv:2602.17622

Agent benchmarks & failure taxonomies

  • τ-bench (fault-assignment × fault-type taxonomy) — arXiv:2406.12045
  • AgentRx (localizes the single critical failure step in a long trajectory) — arXiv:2602.02475
  • AgentBoard (“progress rate” metric — general, non-security capability-decomposition principle, NeurIPS 2024 Oral) — arXiv:2401.13178
  • MAST (14-mode/3-category multi-agent failure taxonomy, κ=0.88) — arXiv:2503.13657
  • AgentErrorTaxonomy / AgentDebug (root-cause diagnosis alone, no reward change, buys +24% all-correct accuracy) — arXiv:2509.25370
  • Phase-aligned taxonomy for autonomous agents (independent-domain convergence on a phase-keyed failure split) — arXiv:2508.13143

Capability boundary, elicitation & sandbagging (contested)

  • Yue et al. — RL elicits, not expands — arXiv:2504.13837 (2025-04-18)
  • ProRL — prolonged RL expands — arXiv:2505.24864
  • Cohen-Inger et al. — “LLMs are Like a Chameleon” (benchmark scores mask overfitting; semantics-preserving perturbation robustness check) — arXiv:2502.07445 (2025-02-11)
  • Zhang et al. — “Memorize or Generalize?” (Memorization Risk Index via semantic-perturbation code rewriting; companion robustness-check citation) — arXiv:2503.02296 (2025-03-04)
  • PSN-RLVR — arXiv:2602.02555 · NuRL — arXiv:2509.25666 · CoT-Pass@K (Wen et al., RLVR implicitly incentivizes correct reasoning) — arXiv:2506.14245 (2025-06-17)
  • Scalpel vs Hammer (GRPO amplifies, SFT replaces) — arXiv:2507.10616
  • Zhai et al. — Does RL Expand the Capability Boundary of LLM Agents? Pass@(k,T) — arXiv:2604.14877 (2026-04-16)
  • Dragoi et al. — Beyond Pass@k: Breadth-Depth Metrics / Cover@τ — arXiv:2510.08325 (2025-10-09)
  • Kang et al. — Quagmires in SFT-RL Post-Training — arXiv:2510.01624 (2025-10-02)
  • Chen et al. — The Coverage Principle — arXiv:2510.15020 (2025-10-16)
  • Greenblatt et al. — Stress-Testing Capability Elicitation with Password-Locked Models — arXiv:2405.19550 (2024-05-29)
  • Hofstätter et al. — The Elicitation Game — arXiv:2502.02180 (2025-02-04)
  • van der Weij et al. — AI Sandbagging — arXiv:2406.07358 (2024-06-11)
  • Ryd et al. — Removing Sandbagging via Weak Supervision — arXiv:2604.22082 (2026)
  • Stroebl et al. — Inference Scaling fLaws — arXiv:2411.17501 (2024-11-26)
  • Dorner et al. — ROC-n-reroll — arXiv:2507.12399 (2025-07-16)
  • Huang et al. — Is Best-of-N the Best of Them? — arXiv:2503.21878 (2025-03-27, ICML 2025)
  • Mahowald et al. — Dissociating Language and Thought in LLMs — arXiv:2301.06627 (2023-01-16)
  • He et al. — LLMs as Neurolinguistic Subjects — arXiv:2411.07533 (2024-11-12)
  • Boháček et al. — Uncovering Competency Gaps (sparse autoencoders on internal representations) — arXiv:2512.20638 (2025-12-06)

PEFT

Frontier lab recipes (reports & blogs) — full-year refresh, 2025-07 → 2026-07, all 10 tracked labs

Llama (historical anchor): Llama 3 — arXiv:2407.21783 · Llama 4 — ai.meta.com/blog/llama-4-multimodal-intelligence

Anthropic (Claude) — Constitutional AI/RLAIF backbone arXiv:2212.08073; inoculation prompting arXiv:2510.04340, Anthropic’s own study arXiv:2511.18397; release posts/system cards — Opus 4.1 · Sonnet 4.5 card · Sonnet 4.5 · Sonnet 4.5 research · Haiku 4.5 card (PDF) · Haiku 4.5 · Opus 4.5 card · Opus 4.5 research · Opus 4.5 card walkthrough (secondary) · inoculation prompting post · emergent misalignment / reward hacking · “teaching Claude why” post · research page · Opus 4.6 · Opus 4.6 sabotage risk report (PDF) · Sonnet 4.6 · Opus 4.7 · Opus 4.8 · Fable 5 / Mythos 5 · Fable 5 & Mythos 5 card (PDF) · Mythos guardrails coverage (secondary) · Sonnet 5 · Sonnet 5 card · Sonnet 5 launch coverage (secondary) · Sonnet 5 launch guide (secondary) · AI organizations post

OpenAI — GPT-5 (2025-08-07) · GPT-5 system card (PDF) · GPT-5 for developers · safe-completions arXiv:2508.09224 · safe-completions post · GPT-5-Codex addendum · Codex system card (PDF) · GPT-5.1 · GPT-5.1 deployment safety · routing/model-choice post (secondary) · GPT-5.1-Codex-Max · Codex-Max system card · Codex-Max safety training · long-horizon Codex tasks · GPT-5.2 · GPT-5.2 for science/math · GPT-5.2 system-card update · GPT-5.2-Codex · GPT-5.3-Codex · 5.3-Codex system card · 5.3-Codex coverage (secondary) · GPT-5.4 · GPT-5.4 thinking system card · GPT-5.4 (secondary) · graders docs · RFT guide · RFT use-cases · RFT wind-down, 2026-05-08 · community thread (secondary)

Google DeepMind (Gemini) — Gemini 2.5 tech report arXiv:2507.06261 (HTML, §2.4/2.5 mirror, cross-checked (secondary)) · Deep Think launch · Gemini 3 Pro model card (PDF) · Gemini 3 launch · agent-building with Gemini 3 · Gemini 3 Deep Think · Deep Think update · Gemini 3 Flash · Flash for enterprise · Gemini 3.1 Pro · Gemini 3.5 · Vending-Bench/τ²-bench cross-check (secondary)

xAI (Grok) — Grok 4 · Grok 4 model card (PDF) · Grok 4 analysis (secondary) · Grok Code Fast 1 · Code Fast 1 model card (PDF) · Grok 4 Fast · Grok 4 Fast model card (PDF) · coverage (secondary) · coverage (secondary) · Grok 4.1 · Grok 4.1 model card (PDF) · sycophancy coverage (secondary) · Grok 4.1 Fast · news index

Mistral — Magistral arXiv:2506.10910 (HF paper page, benchmark tables) · Ministral 3 arXiv:2601.08584 · Mistral 3 blog · Magistral blog · Mistral-Large-3 card · Magistral-Small-2509 card · Magistral-Small-2507 card · Magistral Medium 1.2 docs

DeepSeek — V3 arXiv:2412.19437 · R1 arXiv:2501.12948 · V3.2 arXiv:2512.02556 (HTML) · V3.1 release · V3.1 card · V3.1-Terminus · V3.1-Terminus card · V3.2-Exp · V3.2-Exp repo · V3.2 / V3.2-Speciale

Qwen (Alibaba) — Qwen3 tech report arXiv:2505.09388 · GSPO arXiv:2507.18071 · Qwen3-Omni arXiv:2509.17765 · Qwen3-VL arXiv:2511.21631 · Qwen3-Coder-Next arXiv:2603.00729 · Qwen3.5-Omni arXiv:2604.15804 · GSPO blog · Qwen3-Coder blog · Qwen3 blog · Qwen3 README · Qwen3-Next efficiency · Qwen3-Max · Qwen3-Max-Thinking · Qwen3.5 blog · Qwen3.5-397B-A17B card · Qwen3.7-Max “agent frontier” · Qwen3.7 blog · The Batch coverage (secondary) · VentureBeat coverage (secondary) · TechSphere coverage (secondary) · Qwen3.6-35B-A3B agentic coding

Moonshot AI (Kimi) — K2 arXiv:2507.20534 (verified via arXiv abs + full HTML crawl) · K2.5 arXiv:2602.02276 (verified via arXiv abs + full HTML crawl) · Kimi-K2-Thinking card (no arXiv paper) · K2 Thinking intro post · Kimi-K2.6 card (no arXiv paper) · K2.6 tech blog · K2.6 benchmark deltas · K2.6 coverage (secondary) · K2.6 method non-disclosure (secondary) · Kimi-K2.5 model card/benchmarks

GLM / Z.ai (Zhipu) — GLM-4.5 arXiv:2508.06471 · GLM-5 arXiv:2602.15763 (HTML) · GLM-4.5 blog · GLM-4.6 blog · GLM-4.7 blog · GLM-5 blog · GLM-5.2 blog · GLM-4.5 repo · GLM-5 repo · slime RL infra repo · GLM-4.7-Flash card · MoE architecture deep-dive (secondary, unverified-primary) · agentic RL post citing GLM-5 report (secondary) · RL infra post citing GLM-5 report (secondary) · GLM-5.2 vs 5.1 (secondary) · GLM-5.2 open-source coverage (secondary) · GLM-4.7-Flash coverage (secondary)

Xiaomi (MiMo) — MiMo-V2-Flash arXiv:2601.02780 · MiMo-Embodied arXiv:2511.16518 · MiMo-VL-Miloco arXiv:2512.17436 · MiMo-Audio arXiv:2512.23808 · MiMo lineage (background only, outside the 12-month window): MiMo arXiv:2505.07608, MiMo-VL arXiv:2506.03569 · MiMo-V2.5-Pro card · MiMo-V2.5 card · MiMo-V2.5-Pro blog · MiMo model-update docs

The verified concept-map + genealogy notes in the shared memory pool (research/post-training-inference-concept-map.md, research/post-training-method-genealogy-onpolicy-offpolicy.md, research/frontier-lab-post-training-recipes-2026.md, research/rl-for-long-horizon-exploration-reasoning.md, research/diagnosing-capability-vs-execution-gap-framework.md) are the machine-readable companions to this book.