Ordering rules: interleaving stages & fixing N problems

Is the recipe a loop? is the macro finding: post-training isn’t a one-shot pipeline, it’s a loop over a small set of anchors (base model, reward contract) with an iterated tail (SFT/RS → preference → RL, repeated, gated on measurement). This chapter is the micro companion — it does not re-derive the loop shape. It answers the question the macro chapter deliberately left as “routing logic”: once you know you’re in a loop, which stage-type is safe to apply after which, and does batching N problems into one round actually work, or is that a different failure mode than the loop shape already covers?

Three questions, precisely:

1. Are the stage-types — {continued/domain pretraining, off-policy SFT, on-policy SFT/rejection-sampling, DPO/preference, RL (any variant)} — interchangeable across rounds, or is there a hard ordering constraint?
2. Does off-policy SFT after on-policy RL erode the RL gains — and if DeepSeek-R1 does “SFT after RL” successfully, what’s actually different about it?
3. Given N pass@k-identified problems, do you fix them one at a time or all at once?

Stance held throughout, per project standing rule: no claim below is grounded in an academic cybersecurity-LLM project — none appear as evidence. Grounding is frontier-lab technical reports and general RL/ML theory, verified live via Exa on 2026-07-02. Confidence is stated per claim; “contested” is used honestly where sources disagree.

1. The palette, and the one axis that governs all of it

The five stage-types in play, and where each sits on the on/off-policy axis:

The axis, not the label, is what predicts safety. Two stages with the same name (SFT) can be opposite operations depending on whether the training targets came from π_θ itself or from something else. This is the single fact the rest of this chapter is built on — see the foundations chapter for the general theory (DAgger’s O(εT²) vs O(εT) compounding, arXiv:1011.0686).

Why RL is structurally different from SFT, not just “SFT with a reward term”: RL’s own policy- gradient update is implicitly biased toward staying KL-close to whatever policy produced the reward signal — this is RL’s Razor (Shenfeld, Pari, Agrawal, arXiv:2509.04259): “among all ways to solve a new task, RL prefers those closest in KL to the original model.” SFT has no such restoring force; it will happily converge arbitrarily far from the current policy if the target data says to. This mechanism — not folklore — is why what data an update pulls from matters more than which loss function it uses. Confidence: high — cross-validated on LLMs and robotic policies, ~110 citations at 10 months old.

A second, independent mechanism compounds this: RL monotonically collapses policy entropy early in training (Cui et al., The Entropy Mechanism of RL for Reasoning LMs, arXiv:2505.22617) — R = -a·exp(H) + b, entropy drops sharply, performance saturates as a direct consequence. By the time an RL stage “converges,” the policy is a narrow, overconfident distribution — the worst possible state to hit with a foreign gradient. Confidence: high (mechanistic + broad empirical replication, code public).

2. The ordering-rules table — the centerpiece

“Interchangeable” is the wrong frame. The constraint is directional and about data provenance, not a rigid stage-name sequence. Eleven concrete transitions, rated safe / conditional / erosive:

flowchart LR
  CPT["Continued/domain\npretraining\n(off-policy, coarsest)"] --> OffSFT["Off-policy SFT\n(foreign demos,\ncold-start -- LIGHT ONLY)"]
  OffSFT -->|"light dose"| RS["On-policy SFT / RS\n(self-generated,\nverified)"]
  RS --> DPO["DPO / preference\n(prefer on-policy-\nanchored pairs)"]
  DPO --> RL["RL -- GRPO/RLVR\n(sharpens; entropy drops;\nKL-anchored)"]
  RL -.->|"SAFE -- rejection-sample\nfrom THIS checkpoint"| RS
  RL -.->|"SAFE if LIGHT --\ncorner-case polish"| DPO
  RL -. "EROSIVE -- foreign demos\nonto an entropy-collapsed,\nKL-anchored policy" .-> OffSFT

  classDef coarse fill:#3a2a10,stroke:#f5b942,color:#fff3d6;
  classDef safe fill:#132b22,stroke:#34d399,color:#eafaf3;
  class CPT,OffSFT coarse;
  class RS,DPO,RL safe;

loop_takeaway: coarse/foreign-distribution work goes early and stays light; on-policy/self- distribution work goes late and gets repeated; the one edge that is never safe at full strength is foreign data flowing back onto a policy an RL stage has just sharpened. Everything else — DPO’s exact position, how many RL rounds, whether cold-start SFT exists at all — is a flexible, lab-specific choice within that constraint. Confidence: high on the directional rule (four independent frontier recipes converge — R1, Llama 3, Tülu 3, Llama 4); medium on DPO’s precise micro-placement (genuinely unsettled across labs).

3. The crux: does off-policy SFT after on-policy RL erode the RL gains?

Yes — for foreign off-policy data. Not for on-policy rejection-sampled data. The constraint is data-distribution, not the SFT loss function.

The mechanism

Three independent, converging explanations for why the foreign case is bad:

1. Covariate shift (DAgger lineage). A model trained purely on a fixed demonstrator’s states diverges from that distribution once it starts acting on its own — errors compound at up to O(εT²) (arXiv:1011.0686). A post-RL policy is a specific, sharply-peaked distribution (§1’s entropy-collapse mechanism); foreign SFT data was drawn from whatever produced it (a human, a bigger teacher, an older checkpoint) — none of which match the current post-RL policy.
2. RL’s Razor. RL is implicitly KL-anchored to the policy that generated its training signal (arXiv:2509.04259). Foreign SFT has no equivalent restoring force — it will pull the model toward the demonstrator’s distribution regardless of how far that is from the model’s current behavior.
3. CHORD’s named failure curve. Zhang et al. (Alibaba) directly ablate this: training on expert data that diverges from the model’s established patterns produces a “shift → readapt → overfit” three-phase degradation, which is why naive sequential SFT-then-RL “does not consistently outperform the pure RL approach” (arXiv:2508.11408).

A formal proof exists that some erosion happens even in the safe case: Niu, Bai, Han, Zhang (Huawei), On the Non-decoupling of SFT and RL in Post-training (arXiv:2601.07389), Theorem 4.1 — if RL has converged, any subsequent SFT phase provably degrades the achieved reward, regardless of whether the SFT data is on- or off-policy. This is the important caveat: the theorem doesn’t distinguish provenance. What it does not say is the magnitude — and that’s exactly where CHORD, the DAgger lineage, and the empirical record (below) diverge sharply between the two cases: on-policy erosion is small and recoverable; off-policy erosion can be large and, past a threshold, non-recovered (“RL Is Neither a Panacea Nor a Mirage,” arXiv:2508.16546 — RL-FT restores moderate SFT-induced OOD damage but cannot rescue a checkpoint pushed into a “markedly different representation regime”). Independent mechanistic corroboration for why SFT collateral damage is generally larger than RL’s: SFT makes bigger parameter updates, hits mid-layer MLPs harder (Scalpel vs. Hammer, arXiv:2507.10616 — “GRPO amplifies existing capabilities, SFT replaces them”).

Two more data-alignment papers converge on the same practical fix: Towards On-Policy SFT (arXiv:2602.12222) — “when the data distribution deviates from the model’s, no improved SFT strategy can completely eliminate the effects of catastrophic forgetting” — and Mind the Gap (arXiv:2509.15157), whose operational recipe is directly stealable: keep on-policy-correct solutions as-is; for problems the model still gets wrong, have the current model rewrite a foreign demonstration into its own voice before training on it, rather than training on the raw foreign demo. PEAR (arXiv:2602.01058) does the same alignment at the loss-weighting level (down-weight SFT tokens implausible under the current policy) instead of the data-rewriting level — two independent implementations of the identical fix.

Resolving the R1 “paradox”

R1’s own pipeline is SFT → RL → SFT → RL (arXiv:2501.12948), quoted directly: “Upon nearing convergence in the RL process, we create new SFT data through rejection sampling on the RL checkpoint… combined with supervised data from DeepSeek-V3 in domains such as writing, factual QA, and self-cognition.” Two components, two different safety profiles:

• ~600k reasoning-domain samples: rejection-sampled from the RL checkpoint itself, filtered for correctness. On-policy by construction. This is the domain RL just sharpened — and it’s exactly the safe case above.
• ~200k non-reasoning samples: reused foreign DeepSeek-V3 SFT data (writing, roleplay, self-cognition). Genuinely off-policy — but for capabilities the reasoning-RL stage never touched, so there’s no sharpened distribution there to erode.
• A further RL pass (“all scenarios”) immediately follows, re-anchoring whatever the mixed SFT stage perturbed — mirroring the Panacea-or-Mirage finding that RL can restore moderate SFT damage.

R1 never violates the ordering constraint. It looks like “SFT after RL” only if you read stage- names; read the data provenance and it’s on-policy-for-the-touched-domain + foreign-only-for-disjoint-domains + an RL cleanup pass. Kang et al.’s >1M-GPU-hour study reinforces why this discipline matters even when done carefully: high SFT scores are not reliably predictive of post-RL gains — RL on a “better” SFT checkpoint can substantially underperform RL on the raw base model, because SFT optimized for the wrong objective (SFT-stage accuracy, not what it leaves for RL to build on) (arXiv:2510.01624).

Confidence: high — this is the best-triangulated claim in this whole chapter (mechanism × dosage study × a real, heavily-scrutinized frontier pipeline that avoids the hazard by construction). Medium on the exact magnitude of “how small is small” for the safe case — Huawei’s theorem proves nonzero, doesn’t bound it generally; that’s an open, project-specific measurement, not a literature constant.

4. Fixing N problems: mix-and-replay vs erosive one-at-a-time

Mix, don’t sequence. Fixing N=10 pass@k-identified problems one at a time — a separate fine-tune per problem, moving to the next once the current one improves — is the same failure class as §2’s last table row, just at a coarser grain, and it’s directly documented, not inferred:

• Llama 2’s own regression: confining rejection-sampling to only the latest round’s data (not pooled across prior rounds) caused “RLHF V3 struggled more than previous versions to compose rhyming lines in poems” (arXiv:2307.09288) — the targeted capability kept improving while an untouched one silently regressed.
• Catastrophic-forgetting literature converges: sequential single-task continual fine-tuning is the norm-case failure, and — counterintuitively — worsens with scale in the 1B–7B range (arXiv:2308.08747).
• The forgetting is biased, not uniform: models fine-tuned sequentially forget earlier-tuned behavior to a greater extent than the reverse order, and this disproportionately affects certain categories rather than degrading everything equally (arXiv:2412.16469).
• A purpose-built ablation (the DMT paper, OpenReview 6M5G5hNiAU) confirms the contrast directly: sequential training across skills is “prone to catastrophic forgetting”; pure multi-task mixing avoids forgetting but can hit capability-conflict ceilings at high data volume. Their fix — mix the specialized skills first, then add back a small slice of general-ability data at the end — is a mix-within-a-stage-plus-final-replay recipe, not a waterfall.

Every frontier recipe surveyed mixes, none waterfalls. Tülu 3’s public SFT mixture is 939k samples across 18 named skill sources combined into one stage, with mixing ratio itself treated as a first-class hyperparameter to sweep (arXiv:2411.15124). R1’s SFT#2 corpus mixes ~600k reasoning + ~200k non-reasoning samples in one training pass, not sequential single-domain passes (arXiv:2501.12948). Llama 3 layers multiple named capabilities (tool-use, coding, reasoning, multilingual) into the same round’s SFT/DPO batch across all 6 iterative rounds, never one-capability-per-round (arXiv:2407.21783).

Replay, if you must revisit across rounds. If problem 10 is only discoverable after fixing 1–9 changed the model enough to reach it, don’t leave problems 1–9 out of round 10’s data — but the replayed data should itself be on-policy, not the original off-policy demonstration:

• On-Policy Replay (arXiv:2605.29495): on Qwen2.5-7B-Instruct, sequential SFT alone measures backward-transfer at −13.93; a 10% on-policy replay budget (roll out the current checkpoint on old prompts, filter by task reward, replay only the surviving self-generated pairs) lifts that to −0.65 — the active ingredient is demonstrated to be the on-policy-ness of the replay, not response quality alone (vanilla off-policy replay at matched budget is a materially weaker baseline).
• Older, independently-anchored precedent for the same idea at coarser grain: mixing a modest general- instruction slice into each continual fine-tuning stage recovers held-out knowledge scores (26.8% → 30% MMLU-human vs. 34.72% original, arXiv:2308.08747); Episodic Memory in Lifelong Language Learning (arXiv:1906.01076) is the classical experience-replay mechanism this all descends from.

Model merging — a legitimate alternative to mixing, unvalidated at this scale. Instead of building one joint dataset, you could fix each problem (or cluster) independently and merge the resulting weight deltas. Task Arithmetic (arXiv:2212.04089) shows summed task vectors can improve multiple tasks at once; Model Soups (arXiv:2203.05482) shows weight-averaging independently fine-tuned models often beats the single best one, at zero added inference cost. Both were demonstrated on vision/classification/light-NLP settings, not RLVR-fine-tuned reasoning/agent policies at your scale — treat as a cheap, worthwhile pilot on 2–3 problems, not a default. The macro chapter flags a direct tension worth carrying here: a larger-scale study found model merging does not reliably mitigate forgetting in the settings it tested — don’t treat souping as a free pass.

Confidence: high that mixing-within-a-round beats sequential one-at-a-time, and that on-policy replay beats off-policy replay when rounds must revisit old problems (both are directly documented, multi-source-convergent claims). Medium on exact replay ratios and scheduling (open research area). Medium-low on model merging’s applicability to this project’s RLVR-on-a-dense-agent setting specifically (mechanism is proven elsewhere, not yet tested here).

5. The concrete plan for our ~10 F1–F4 problems

This section applies the ordering rules and the batching verdict to this project’s own diagnosis framework — F1–F4: discovery, exploit-skill, tool-use, pivot/long-horizon. It does not repeat the macro chapter’s round-by-round template — read that for the loop mechanics (measure → route → apply → re-measure → exit-test). What follows is the stage-ordering and batching layer that plugs into it:

1. Cluster, don’t sequence, by F-label. The ~10 problems get grouped by which F1–F4 species dominates their failure this round — a batching key, not a per-problem schedule. Two problems that both route to rejection-sampling SFT go into one mixed SFT batch, not two fine-tunes.
2. Before routing a cluster to on-policy SFT/RS, confirm ≥1 verified own-solve exists (a pass@64–128 check). Zero verified solves means the cluster isn’t SFT-ready yet — it’s an upstream discovery gap (F1) — and this is precisely the moment someone is tempted to substitute a foreign teacher demonstration instead. Don’t — that’s exactly the erosive edge from §2’s table. Route to more sampling / curriculum first.
3. F3 (tool-avoidance) clusters run the elicitation ladder before consuming any SFT-batch capacity — the behavior may already be latent; only clusters that fail all rungs graduate to training.
4. If clusters in the same round route to different stage-types (some to SFT, some continuing RL), §2’s ordering applies: any genuinely new off-policy capability-injection (a true knowledge gap) runs before the RL continuation in that round, never after; the on-policy RS batch and the RL continuation use the restart-for-SFT, continue-for-RL asymmetry already established in the macro chapter.
5. Every SFT batch this round = (mixed, verified, on-policy rejection-sampled trajectories for the currently-routed clusters) + (a replay slice of prior rounds’ verified solves for the clusters NOT routed this round). This is §4’s mixing verdict plus its replay corollary, applied directly.
6. Re-measure all ~10 problems, not just the routed subset, every round. A technique aimed at one cluster can silently regress an untouched one — this is the only way to catch it (Llama 2’s poem- rhyming case, §4).
7. Expect a small, real, on-policy erosion cost even when everything above is followed correctly (Huawei’s Theorem 4.1, §3) — budget an RL “clean-up” pass after any SFT injection, mirroring R1’s own final RL pass after its combined SFT#2 stage.

One honest caveat carried from the macro chapter and worth restating here: which F-label dominates by round 3 is explicitly not predictable in advance — this is why step 6 is a per-round re-measurement, not a one-time classification of the 10 problems. The exact severity of off-policy-after-RL erosion, and the exact replay ratios, are also this project’s own measurements to make once a real RL loop exists — every number cited above (CHORD’s magnitude, the 10% replay budget, Huawei’s proof) comes from math-reasoning or general-chat domains, not a ~100-turn CTF agent specifically.

Confidence: high on the structure (direct multi-recipe convergence: R1, Llama 3, Tülu 3, plus this project’s own already-settled diagnosis framework); medium on whether the numeric thresholds (replay %, exit-test deltas) transfer cleanly to this project’s scale and domain — flagged as the first thing to measure, not assumed.

Confidence summary

Cross-links: Is the recipe a loop? for the macro loop shape this chapter’s ordering rules route within; The recipe is a sequence, not a pick for the underlying stage skeleton; The one axis that predicts everything for the on/off-policy theory this whole chapter is an application of; One problem, or many? — monolithic vs decomposed for the architecture-side twin of this chapter’s batching question; Diagnosing the gap for the F1–F4 funnel that supplies the routing signal §5 depends on; Where you are & the forks ahead for how this resolves into this project’s next concrete decision.