Is the recipe a loop?

The recipe is a sequence, not a pick established that post-training is an ordered stage skeleton (base → CPT → SFT → rejection-sampling → preference → RLVR), not a single technique choice. This chapter asks the structural question that skeleton leaves open: does the sequence run once, left to right, or does it loop back on itself?

Concretely: (1) what stays one-shot, outside any loop, vs what gets repeated, inside one? (2) Can you go SFT → RL → back to SFT → RL again? Is RL run once or N times — and can you go back to DPO after RL? (3) When you revisit a stage, do you continue from the current checkpoint or restart from the base model (a fresh fork)? (4) Non-commutativity — is a stage applied late the same operation as the same stage applied early, or does the model’s accumulated drift/forgetting make it a genuinely different intervention? (5) The payoff: given N known behaviors to fix, do you plan a unidirectional roadmap (one behavior per stage, run once, ship) or an iterative loop that re-targets whatever the current measurement says is the bottleneck?

Point (5) is not a new axis — it is decomposition-vs-monolithic (dissect the problem into independent sub-problems, or treat it as one) projected onto the sequence instead of onto architecture. Keep that identity in view throughout: “how many sub-policies” and “how many rounds” turn out to resolve the same way.

Stance held throughout, per project standing rule: no claim below is grounded in an academic cybersecurity-LLM project (CTF-Dojo, Cyber-Zero, Pentest-R1, HackSynth, AutoPenBench, DRLRM-PT) — none appear as evidence anywhere in this chapter. Grounding is frontier-lab technical reports and general RL/ML theory, cross-checked with this project’s own funnel/pass@k methodology. Every arXiv id below was verified live (arXiv API / ar5iv crawl) during the research pass this chapter draws from — confidence is stated per claim, and “contested” or “promising, not yet validated” is used honestly where the evidence doesn’t close the question.

1. IN the loop vs OUT of the loop

Not every decision is a loop candidate. Conflating a one-shot wall with an iterable stage is the most common planning error here.

flowchart TD
  subgraph OUT["OUT of the loop -- one-shot, never revisited"]
    direction TB
    Pre["Pretraining\n(the base-model generation you inherit)"] --> Mid["Mid-training / annealing\n(context extension, curated upsample)"]
    Mid --> Wall["THE WALL\nfrozen base/instruct checkpoint\n+ tokenizer/precision/serving stack\n+ reward contract (flag_verified)"]
  end

  Wall --> Cold["Cold-start SFT\n(small, format-only, once)"]

  subgraph IN["IN the loop -- the iterated tail"]
    direction TB
    Cold --> RS["Rejection-sampling /\non-policy SFT"]
    RS --> Pref["Preference opt\n(DPO/KTO)"]
    Pref --> RLVR["RLVR / GRPO"]
    RLVR -.->|"generate next round's\nSFT data from THIS\ncheckpoint's rollouts"| RS
    RLVR -.->|"DeepSeek-R1: retrain\nfrom clean base with\nRL-checkpoint-generated data"| SFT2["SFT (round 2)"]
    SFT2 -.-> RLVR2["RL (round 2)"]
  end

  classDef out fill:#2b1313,stroke:#f87171,color:#fde8e8;
  classDef wall fill:#3a2a10,stroke:#f5b942,color:#fff3d6;
  classDef in fill:#132b22,stroke:#34d399,color:#eafaf3;
  class Pre,Mid out;
  class Wall wall;
  class Cold,RS,Pref,RLVR,SFT2,RLVR2 in;

loop_takeaway: the pretrain/anneal wall is the real boundary — think of the base checkpoint (+ the reward contract, + the infra pinning) as an immutable artifact received once per experiment. Everything after that wall is fair game for revisiting; nothing before it is. Confidence: high (direct, repeated frontier-lab framing across independent recipes).

2. Yes, you can revisit stages — this is standard, not exotic

DeepSeek-R1’s own pipeline is SFT → RL → SFT → RL, quoted directly (arXiv:2501.12948):

“we begin by collecting thousands of cold-start data to fine-tune the DeepSeek-V3-Base model. Following this, we perform reasoning-oriented RL… Upon nearing convergence in the RL process, we create new SFT data through rejection sampling on the RL checkpoint… and then retrain the DeepSeek-V3-Base model… After fine-tuning with the new data, the checkpoint undergoes an additional RL process, taking into account prompts from all scenarios.”

Stage-by-stage: ColdStartSFT → RL₁ (reasoning-only) → rejection-sample RL₁’s rollouts → SFT₂ (~800k = 600k reasoning + 200k general, 2 epochs) → RL₂ (all-scenario, reward model added for helpfulness/harmlessness). SFT happens twice, RL happens twice — a fixed 4-stage round-trip, not an open-ended loop. This is the flagship reasoning-model recipe of 2025, not a hack.

Llama 3 runs 6 explicit iterative rounds, quoted directly (arXiv:2407.21783):

“Following Llama 2, we apply the above methods in six rounds. In each cycle, we collect new preference annotations and SFT data, sampling synthetic data from the latest models.”

Each of the 6 cycles: collect new human preference annotations against the current best model → update the reward model → rejection-sample new SFT data from the current policy → SFT → DPO → evaluate → feed forward into round i+1. Reference model, sampling policy, and SFT targets all update round-to-round — this is Meta’s answer to “RL once vs N times” for preference optimization: DPO N times (N=6), never PPO-style online RL in the main loop.

Iterative DPO more broadly is a named, established pattern, not a DeepSeek/Llama-specific quirk: Self-Rewarding Language Models (arXiv:2401.10020) runs Llama 2 70B through 3 iterations where the model generates and judges its own new preference pairs each round, instruction-following improving monotonically for those 3 rounds specifically because both the judge and the generator are re-derived from the current policy each time, not frozen. Apple’s AFM iTeC (arXiv:2407.21075) goes further — instead of committing to one linear stage order, it keeps a committee of RS/DPO/IPO/online-RL variants alive every round and lets round-over-round evaluation pick which optimizer propagates. STaR (arXiv:2203.14465) is the theoretical minimum-viable ancestor of all of this: “generate rationales… fine-tune on all the rationales that ultimately yielded correct answers; repeat” — no reward model, no PPO, just sample-filter-tune-repeat, with the outer loop drawn explicitly as a loop back through fine-tuning in the paper’s own Figure 1.

Confidence: high — DeepSeek-R1’s round-trip and Llama 3’s 6 rounds are both verbatim, primary-source quotes; iterative DPO / committee patterns are independently corroborated across three more labs.

Continue from checkpoint, or restart from base?

This is where the practice genuinely splits, and the split is not random — it correlates with which stage is being revisited, not with “how iterative the lab is” in general.

• Continue-from-latest is the default for RL. Llama 3 samples “from the latest models” every round; DeepSeek-R1’s RL₂ “retains most of the parameters from the first stage” (Nature companion paper, s41586-025-09422-z); Apple’s iTeC committee explicitly includes “the best models from previous iterations.” Continuing is cheap and is what “successive rounds” literally means for RL.
• Restart-from-base is the disclosed convention for SFT-revisits. DeepSeek-R1’s SFT₂ explicitly “retrain[s] the DeepSeek-V3-Base model” — the RL₁ checkpoint’s job is purely to generate the data; its weights are discarded, and a clean copy of base is fine-tuned on the newly curated set. This exact asymmetry is independently reconfirmed by Havrilla et al., Teaching LLMs to Reason with RL (arXiv:2403.04642): “for both EI and RCRL we generate data with the SFT checkpoint but reset training to start from the pretrained base model… we find this model resetting is crucial for achieving best performance.” — and by STaR before either. Three independent groups converge: when you revisit SFT, restart from the clean base using freshly-generated data; when you continue RL, continue from the current best checkpoint.
• ReST-EM is the disclosed exception that restarts everything — it fine-tunes from the base pretrained model at every outer iteration, generate-and-improve, “to mitigate task-specific overfitting” (arXiv:2312.06585). Flag this as a genuine, disclosed disagreement: restart-vs-continue is contested as a global policy, but converges cleanly once you split it by stage type (see below).
• A controlled A/B on exactly this variable, University of Chicago, “Iterative Finetuning is Mostly Idempotent” (arXiv:2605.01130, 2026, 0 citations — promising, not yet independently replicated): “trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle.” Mechanism: restarting each cycle makes the dataset — not accumulated weight-drift — the only thing carried forward, so errors don’t compound; continuing creates a persistent optimization trajectory where each round’s update stacks on the last one’s direction, which is exactly what you want for RL (build on exploration gains) and exactly what you don’t want for SFT (narrow imitation of one round’s possibly-idiosyncratic data compounding on top of already-drifted weights).

loop_takeaway: “continue vs restart” is not one global policy for the whole loop — it is a per-stage decision. Restart for SFT-revisits (use fresh, RL-generated data against a clean base). Continue for RL-revisits (build on the current policy). Confidence: high on the convergence (four independent groups); medium on the precise mechanism (the idempotence paper is single-team and studies persona drift, not CTF capability — a well-argued generalization, not a domain-proven fact).

3. The constraints on revisiting — five things that make a loop non-free

A loop is not a free repeat button. Five independent, converging bodies of evidence bound how you can run it.

(a) Forgetting accumulates, and severity is data-scale-dependent, not a fixed law. Continual instruction-tuning shows catastrophic forgetting is the norm and — counterintuitively — worsens with model scale in the 1B–7B range studied (arXiv:2308.08747). A more recent, larger study (arXiv:2510.17776) refines this: forgetting is not a single scalar that always gets worse — RL/SFT from a base model shows moderate-to-large backward transfer (net gain) with low/moderate forgetting elsewhere, while RL/SFT applied to an already instruction-tuned model is data-scale sensitive (“mixed, warrant further study” at large extra data-scale). Model merging does not reliably mitigate forgetting (yet) per the same source — don’t treat souping as a free pass. Confidence: high that CF is real; contested on exactly how it scales with round count.

(b) Diminishing returns, then instability, past some round count. The first systematic study of overoptimization dynamics across iterations of iterated RLHF (arXiv:2505.18126) finds overoptimization itself decreases over successive iterations (the reward model increasingly approximates ground truth) but performance gains diminish over time — the classic curve. Self-rewarding iterative DPO shows the same shape empirically: gains shrink each iteration on ~7B models due to “accumulated bias in the reward system” (arXiv:2410.12735). Left unmanaged, prolonged RL degrades further into outright instability — entropy collapse, KL spikes — not just plateauing (ProRL, §c below).

(c) Fresh, on-policy data is needed each round — reusing stale rollouts is the single most common silent failure. Off-policy preference optimization “often suffers from a distributional gap between the policy used for data collection and the target policy” and even reweighting fixes don’t fully close it (arXiv:2406.11827); a theoretical result shows DPO’s gradient-signal magnitude scales with how much the generating distribution underrepresents high-reward responses relative to the current policy — multi-round online DPO, resampling from the updated policy each round, is “a principled justification, not a heuristic” (arXiv:2506.04272). loop_takeaway: if you revisit an RL/DPO stage, regenerate the preference/rollout data from the CURRENT checkpoint before that round starts — reusing round-1’s rollouts in round-3 is now mechanistically explained, not just folklore.

(d) KL / reference-policy resets — even inside a single “RL stage,” you’re already looping. NVIDIA’s ProRL (arXiv:2505.24864, NeurIPS 2025) names the failure directly:

“as training progresses, the KL term may increasingly dominate the loss, leading to diminishing policy updates. To alleviate this, we introduce… reference policy reset: periodically, we hard-reset the reference policy π_ref to a more recent snapshot of the online policy π_θ, and reinitialize the optimizer states.”

Every ~200–500 steps, ProRL hard-resets π_ref ← π_θ (current). This is a round boundary inside a single nominal RL stage — the same continue-vs-reset mechanics from §2 recur at a finer grain than the macro-level loop this chapter otherwise describes.

(e) Model merging is a real alternative to sequential stacking — not a solved one. Task Arithmetic (arXiv:2212.04089) → Model Soups (arXiv:2203.05482) → WARM (souped reward models, arXiv:2401.12187) → WARP (arXiv:2406.16768) is a well-cited genealogy of “instead of chaining rounds sequentially, train branches independently and merge in weight space.” WARP is the sharpest instance: it merges inside every RL iteration (EMA anchor, spherical interpolation across independently-RL’d policies, linear interpolation back toward the pretrained init) and runs the whole procedure iteratively — merging used as the revisit mechanism, not a post-hoc alternative to it. But the (a)-cited 2510.17776 finding — “model merging does not reliably mitigate forgetting (yet)” — is a direct, recent tension with WARP’s own framing. Confidence: high that merging exists and works in cited settings; contested on how general the forgetting-mitigation benefit is.

Non-commutativity: a stage applied late is not the same operation as applied early

This is the sharpest, most directly answerable part of the question, and it now has a formal proof, not just observation. Niu, Bai, Han, Zhang, On the Non-decoupling of SFT and RL in Post-training (arXiv:2601.07389, Jan 2026) prove:

1. SFT-then-RL coupling (Thm 3.1): even if SFT has already converged (loss no longer decreasing), a subsequent RL phase still measurably increases SFT loss — RL’s reward-seeking updates degrade what SFT had converged to.
2. RL-then-SFT coupling (Thm 4.1): symmetrically, if RL has already converged (reward no longer improving), a subsequent SFT phase degrades the reward RL had achieved.

These are two different theorems with different mechanisms, not mirror images — confirmed empirically on Qwen3-0.6B (abrupt degradation in cross-entropy or reward exactly at the transition point, both directions). The proof is very recent (0 citations at verification time — promising, not yet peer-validated), but it formalizes a pattern already independently observed four other ways:

• DeepSeek-R1’s own ablation. R1-Zero (RL, no cold-start SFT) reaches the same eventual reasoning capability as R1 but a materially worse behavior profile (poor readability, language mixing) — the paper’s own stated reason cold-start exists: “starting RL training from an uninitialized model can lead to instability and slow convergence” (2501.12948). Order changes the character of the result, not just a scalar.
• Llama 4’s explicit reversal of Llama 3’s emphasis — thin SFT/DPO, intensive RL, because “SFT and DPO can over-constrain the model, restricting exploration during the online RL stage” (Meta blog, already cited in The recipe is a sequence).
• Scale-dependence within one family: Magistral Medium runs pure GRPO, zero SFT (AIME’24 pass@1 26.8→73.6 from RL alone); Magistral Small (24B) needs SFT-then-GRPO because RL-only underperforms at that scale (arXiv:2506.10910) — the “right” order is conditional on how much latent capability the base already carries, not a fixed law even inside one lab.
• Excessive SFT measurably reduces subsequent RL plasticity — over-confident, sharper output distributions from over-long SFT make the checkpoint harder for RL to reshape (arXiv:2606.09932); a companion result shows RL-after-SFT’s benefit is often a restoration of OOD capability that SFT itself degraded, not new capability (arXiv:2509.12235) — order and dose, not just presence/absence, determine the reachable end-state.
• Meta FAIR’s >1M-GPU-hour study: the best post-SFT checkpoint is often NOT the best pre-RL checkpoint — over-trained SFT can do worse after RL than a less-optimized SFT checkpoint, or worse than skipping SFT entirely (arXiv:2510.01624).

loop_takeaway: treat “which stage, applied at which revisit-point” as a designed intervention with a predictable directional effect (RL heals OOD forgetting from SFT; more SFT reduces RL’s subsequent plasticity), not an interchangeable menu item. This is the concrete mechanism behind why an iterative loop targeting the current bottleneck beats a fixed unidirectional roadmap — the roadmap can’t know in advance which stage, at which point, will help or hurt, because that depends on the accumulated forgetting state the loop itself creates. Confidence: high on the empirical pattern (independently replicated across five groups); promising, not yet peer-validated on the formal proof itself.

4. Planning to fix N behaviors: iterative-target-the-bottleneck, not one-per-stage

Given N known behaviors to fix (this project’s own F1–F4: discovery/exploration, exploit-skill, tool-use, pivot/long-horizon — see Diagnosing the gap), the naive plan is a unidirectional roadmap: assign F1 to stage A, F2 to stage B, F3 to stage C, F4 to stage D, run each once in order, done. It is attractive because it reads like a project plan with a finish line. It is the wrong shape, for three concrete reasons, each backed by evidence already established above:

1. It assumes the behaviors are independently addressable. §3’s non-commutativity evidence says otherwise — an earlier stage’s fix for one behavior can foreclose exploration a later stage needs (Llama 4’s own warning), and a later stage’s technique can silently erode what an earlier stage already fixed if old-round data isn’t remixed in (Llama 2’s own documented regression, below).
2. It commits the technique before measuring which behavior is actually the current bottleneck. A technique correctly matched to the dominant gap in round 1 may be the wrong match by round 3, once the policy has moved — the same technique (e.g. staged/dense reward) helps a weak policy and washes out as the policy strengthens (arXiv:2603.21972, already cited in decomposition-vs-monolithic). “Which of F1–F4 is dominant” is an empirical, round-to-round question, not a fixed assignment.
3. It has no natural stopping rule. “Run each stage once” either stops before the gain saturates or keeps running stages that have already plateaued — burning compute and forgetting budget for near-zero marginal return (§3b/§3a).

A concrete, documented cost of skipping the “keep old data in the mix” discipline: early Llama 2 RLHF, confined to rejection-sampling only from the latest round’s data (not pooled across all prior rounds), produced a silent regression — “RLHF V3 struggled more than previous versions to compose rhyming lines in poems” (arXiv:2307.09288). They fixed it by pooling across all prior iterations, not by restarting from base. This is forgetting accumulating specifically because of round-order discipline, independent of any single stage’s technique.

This is the decomposition-vs-monolithic axis, projected onto the sequence

The already-settled verdict for architecture is: decompose the eval unconditionally (cheap, safe, diagnostic), but do not decompose training into independently-trained sub-policies wholesale — gate any training-side change on what the eval funnel actually shows (decomposition-vs-monolithic). Projected onto time instead of architecture, the identical logic reads: decompose the measurement into per-round funnel snapshots (cheap, safe, diagnostic) but do not commit training to a rigid, pre-planned, one-behavior-per-stage roadmap — gate each round’s technique choice on what that round’s funnel snapshot shows. “How many sub-policies” and “how many rounds, fixed in advance vs re-decided” are the same decision, and both resolve the same way: decompose the cheap diagnostic layer freely; keep the expensive training-loop commitment conditional and re-evaluated, never frozen in advance.

Every frontier recipe surveyed in this chapter is itself built this way. GLM-4.5’s difficulty-curriculum switch is explicitly signal-triggered, not schedule-triggered — switch to harder problems “once static data goes stale,” measured as zero reward-variance, not a pre-committed step count. Llama 2’s RLHF-V1→V5 reward-model retraining is paced by “as we received more batches of human preference data,” not a calendar. Kimi K2’s Toggle mechanism alternates budget-limited vs unconstrained-scaling phases gated on a measured accuracy threshold. The empirically dominant pattern is not “pick one of {roadmap, loop}” — it is a coarse-grained roadmap (decompose behaviors into a small number of named stages, in a rough intended order — informed by §3’s non-commutativity finding, so front-load anything with a foreclosure risk) whose transitions are gated on measured bottleneck signals, not fixed in advance.

A loop-exit criterion — without one, a loop is an open-ended compute sink

Four converging, citable stopping signals:

1. Explicit validation-saturation rule (Expert Iteration). Havrilla et al. (arXiv:2403.04642): “repeated until performance on a validation set saturates” — for their tasks, saturation happened at n=5 rounds. The round count is an output of the loop, not a pre-committed input.
2. Measured, not assumed, diminishing returns at scale. A 20+-model scaling study (arXiv:2412.06000) finds RLHF gains “improve remarkably in the early stage… but additional data yields only marginal gains despite increasing training rewards” — RLHF scales less efficiently than pretraining. Track marginal pass@1/pass@k gain per round; route away once it flattens.
3. A cheap go/no-go gate BEFORE spending RL compute on the next round. Held-out generalization loss + Pass@64 on the post-SFT checkpoint predict post-RL Pass@1 better than post-SFT Pass@1 itself (arXiv:2510.01624) — compute this before committing to another round.
4. A general-RL-theory precedent for continue-vs-restart AND loop-exit combined. AlphaGo Zero’s self-play loop only ever promotes a challenger network to “current best” if it clears a measured win-rate bar in held-out games; otherwise the loop discards the challenger and tries again with fresh data (Silver et al., Nature 550, 2017 — no arXiv id, general RL theory, not fabricated). Never accept a round’s output on faith — gate promotion on a measured bar.

Concrete exit rule: exit (or route away from) a given technique/stage when any two hold: (a) the targeted behavior’s measured delta is within noise of the previous round, (b) Pass@64 on the post-round checkpoint is flat vs. the previous round’s, (c) a forgetting guardrail (held-out general-capability eval, or a non-targeted behavior’s own measured rate) has dropped since the previous round.

5. A concrete iterative-loop template for this project’s F1–F4 case

flowchart TD
    A["Round start: run the frozen\nfunnel + pass@1/pass@k\nagainst the CURRENT checkpoint"] --> B{"Which of F1-F4\nis the current\nbottleneck?"}
    B -->|"F1 discovery"| C["Route: curriculum sequencing\nor milestone shaping\n(no reward-contract change)"]
    B -->|"F2 exploit-skill"| D["Route: rejection-sampling SFT\non fresh own-verified solves --\nRESTART from clean base/\ncold-start checkpoint (S2)"]
    B -->|"F3 tool-use"| E["Route: elicitation-ladder first\n(prompt -> light SFT) before\ncommitting SFT capacity"]
    B -->|"F4 pivot/long-horizon"| F["Route: continue current\nGRPO/RLVR run -- CONTINUE\nfrom current checkpoint (S2)"]
    C --> G["Retrain the routed stage with\nfresh data from THIS round's\ncheckpoint (S3: on-policy)"]
    D --> G
    E --> G
    F --> G
    G --> H["Re-measure: funnel + pass@k +\nforgetting guardrail (held-out\ngeneral-capability + non-targeted\nbehaviors)"]
    H --> I{"Exit test: >=2 of\n(a) delta in noise\n(b) Pass@64 flat\n(c) guardrail dropped?"}
    I -->|"No -- still improving"| A
    I -->|"Yes -- plateaued\nor regressing"| J["Stop this stage's technique;\nmix this round's data into next\nround's SFT/preference sets\nregardless (S4: anti-forgetting);\nreport per-round, not one\nclosing number"]

Per round: (1) run the frozen eval suite against the current-best checkpoint at zero training-loop cost; (2) identify the dominant F1–F4 bottleneck this round — not a bottleneck assigned in advance, cross-check against a cheap elicitation ladder before assuming any gap needs a training-loop response at all; (3) route to the matching technique; (4) apply the §2 asymmetry — restart from clean base if this round revisits SFT, continue from current checkpoint if this round continues RL; (5) re-measure with the same frozen suite and apply the §4 exit test; (6) regardless of the exit-test outcome, mix this round’s data into the next round’s SFT/preference sets — the one documented failure mode of not doing this is a silent capability regression (Llama 2’s poem-rhyming loss), and it costs nothing to avoid; (7) report per-round which behavior was dominant, what was routed, what moved, what the exit test said — a running log, not a single closing verdict, because the honest answer is a split verdict by challenge subtype, not one number (see Diagnosing the gap).

Confidence summary

Cross-links: The recipe is a sequence, not a pick for the stage skeleton this chapter loops; One problem, or many? — monolithic vs decomposed for the architecture-side twin of this chapter’s sequence-side question; Diagnosing the gap — a scientific framework for the funnel/pass@k measurement machinery that routes each loop round; Before you train — instrumentation & data readiness for what must be in place before a loop round can even be measured; Where you are & the forks ahead for how this loop-vs-pipeline view resolves into this project’s next concrete decision.