Data mixing, ratios & not forgetting how to think

Ordering rules answered which stage-type is safe to run after which. This chapter answers the question one level below that: even with the right stage order, how much data from each stage — and what ratio of replay against earlier-stage data — is the difference between a capability gain and a silent format collapse.

0. The anecdote

Someone SFT’d a reasoning-capable, <think>...</think>-style model on naive off-policy synthetic trajectories — teacher-generated (prompt, response) pairs where the responses were terse, answer-only, no reasoning trace — using LoRA. After training, the model stopped emitting CoT entirely, even on prompts where it used to think. The expectation going in was that LoRA’s well-known “learns less, forgets less” property (Biderman et al., arXiv:2405.09673) would make this safe. It didn’t.

One-line diagnosis, verified rather than assumed: this is format/behavioral collapse, not weight destruction. The model didn’t lose its reasoning circuitry — it learned, from the data, that the correct response shape is “no thinking.” LoRA didn’t save it because LoRA bounds how far the weights move (magnitude), not what direction a small move takes them (behavior/format) — and “always skip <think>” is exactly the kind of coarse, low-information-content policy switch a small move is cheap to encode. This chapter verifies both halves of that claim against the literature, then builds the general theory of data mixing that prevents it: replay ratios, LoRA discipline, template hygiene, and a mandatory per-checkpoint “does it still think?” probe.

Stance, as everywhere in this book: no academic cybersecurity-LLM paper is used as evidentiary ground below — grounding is frontier-lab reports and general RL/ML/continual-learning theory. All arXiv ids were verified live via Exa on 2026-07-02; confidence and evidence-type (mechanistic-argument vs measured) are marked per claim, and “contested” is used honestly where sources disagree.

1. Why SFT erases reasoning — the mechanism, cited

1.1 It’s a named, measured phenomenon: “reasoning-trace collapse”

Twist, Yannakoudakis, Zhang (King’s College London), “Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning,” arXiv:2605.21127 (2026). This is the closest thing in the literature to a direct writeup of the anecdote. Their claim, verbatim: fine-tuning a reasoning model on “ordinary instruction–response data that contains no [reasoning] traces” induces collapse — the model “continues to produce plausible final answers while losing the structurally valid explicit reasoning traces that made it a reasoning model in the first place… a model can minimise its loss by learning to produce only the final answer, effectively treating the absence of reasoning as the desired behaviour.” Across four open-weight reasoning models: standard SFT can rapidly suppress valid reasoning traces, and — the load-bearing methodological point — answer-only accuracy hides the failure: in several settings, accuracy conditional on emitting valid reasoning stays high while the rate of emitting valid reasoning collapses. The cheap fix, and it doesn’t need teacher CoT at all: loss-mask the non-reasoning-format tokens so terse demonstrations don’t actively supervise “the correct move here is skip thinking.” Confidence: high — measured, multi-model, framework released, though a brand-new (2026), not-yet-independently-replicated preprint.

Independent, peer-reviewed corroboration: Lobo, Agarwal, Lakkaraju, “On the Impact of Fine-Tuning on Chain-of-Thought Reasoning,” NAACL 2025, arXiv:2411.15382. SFT (including QLoRA rank-16) on non-reasoning datasets reduces both CoT accuracy and CoT faithfulness (whether the reasoning is causally load-bearing vs. post-hoc rationalization), general across most tested datasets, worse in smaller models — i.e. not a LoRA-config artifact, and worse at exactly the scales a project like this actually deploys. Confidence: high (peer-reviewed, corroborating).

1.2 Why: the answer is load-bearing, not decorative — so suppressing CoT isn’t free

Zhang, Lin, Rajmohan, Zhang (Microsoft), “From Reasoning to Answer,” arXiv:2509.23676 (2025). Across three distilled DeepSeek-R1 models: reasoning consistently improves answer quality; dedicated mid-layer “Reasoning-Focus Heads” track the reasoning trajectory; and activation patching on reasoning tokens causally alters the final answer. This is why format collapse is a capability loss, not a cosmetic one — later tokens are mechanistically consuming the reasoning computation. Complementary quantified evidence: Zhang, Morris, Shmatikov (Cornell Tech), ICML 2026, arXiv:2603.07267 — fine-tuning Qwen-2.5-7B-Instruct on answers+summaries only vs. with reasoning traces present (even synthesized/inverted ones) gives a large measured delta: MATH500 56.8% → 77.6%, JEEBench 11.7% → 42.3%. Confidence: high (mechanistic-interpretability + a clean, large, quantified delta).

1.3 The mechanistic bridge: the <think> delimiter is a fragile, learnable contract

Zhu, Zhang, Wang, Xu, Lyu, Wu, “To Think or Not to Think,” arXiv:2502.12202 (2025). Forcing an LRM to see an empty <think></think> block flips it straight to the final answer — >90% attack success rate, ~80% relative performance drop across mainstream LRMs — and this same behavior is backdoorable during ordinary SFT/DPO, no adversarial intent required: if training data contains empty/absent think blocks at any non-trivial rate, “empty think block → answer now” generalizes as a policy, not a per-example fact. Naive off-policy SFT that never emits <think> at all is, from the chat-template’s perspective, in-kind indistinguishable from data that emits an empty think block — the model sees “prompt → nothing between the delimiters → terse answer” as the target distribution often enough to generalize it as the mode. Confidence: high for the vulnerability itself (adversarial- robustness paper, careful methodology); mechanistic-argument, not directly measured, for “accidental naive SFT recreates this by accident” — a strong, well-supported inference, not a controlled experiment on the non-adversarial case.

Qwen’s own fine-tuning documentation treats exactly this as a named risk: the official chat template uses an empty <think>\n\n</think>\n\n block to signal non-thinking mode at inference, and Qwen’s guidance for mixing non-CoT data recommends explicit flags (ignore_empty_think, add_non_thinking_prefix) rather than letting formats mix implicitly in a batch. Concrete knob: if you must include non-reasoning examples, tag them explicitly as a selected non-thinking mode rather than silently omitting the think block. Confidence: medium (primary docs + secondary diagnosis, not a controlled experiment; the underlying principle is durable, the specific flag names may drift).

1.4 Compounding factor: off-policy distribution shift

The failure is not just “no reasoning tokens in the data” — it’s compounded by the data being off- policy relative to the student. This is a direct instance of the on/off-policy axis: DAgger’s classical result is that a fixed, off-policy demonstrator distribution gives no corrective signal for the states the learner’s own policy actually visits, and imitation error compounds up to O(εT²) (Ross, Gordon, Bagnell, arXiv:1011.0686); the sequence-level, purely-supervised restatement is exposure bias / scheduled sampling (Bengio et al., arXiv:1506.03099); the modern, LLM-native formalization is on-policy distillation (GKD, Agarwal et al. (DeepMind), arXiv:2306.13649), which exists specifically to kill the train/inference mismatch fixed-dataset KD creates. A synthetic teacher trajectory is, by construction, off-policy — the student never would have generated that terse response itself — so the moment it does start a <think> block (from its pre-SFT prior), the SFT training distribution offers no guidance for continuing it, biasing continuation toward “wrap it up fast.” Confidence: high as established theory (canonical, widely-cited references); its direct quantitative transfer to modern chat-format LLM SFT is a well-established analogy in the field, not a re-derivation on LLMs specifically — flagged as well-established analogy, not measured for this exact setting.

1.5 Format/style is learned — and unlearned — from remarkably little data

Zhou et al. (Meta AI), LIMA, arXiv:2305.11206 (2023). A 65B LLaMA SFT’d on just 1,000 curated (prompt, response) pairs reaches response quality competitive with GPT-4/Bard on a large fraction of human judgments — the “Superficial Alignment Hypothesis”: most of a base model’s knowledge is learned in pretraining, fine-tuning mostly teaches format/style, and format is cheap to teach from few examples. This cuts both ways. If style is learnable from ~1,000 examples in the beneficial direction, it’s exactly as learnable in the harmful direction — a modest terse-teacher dataset is easily enough to overwrite a <think>-emission habit. This is why “even LoRA, even without huge data” was sufficient to break the anecdote’s model. Confidence: medium-high — the directional claim (format is cheap to (un)learn) is uncontested and is all this chapter needs; the strong-form hypothesis (“fine-tuning teaches only format, never new capability”) is contested — later RLVR results (DeepSeek-R1) show post-training stages can add non-superficial capability too. Also complementary, InstructGPT’s own disclosed “alignment tax” (Ouyang et al., arXiv:2203.02155) — “minimal performance regressions on public NLP datasets” as the measured cost of RLHF — establishes that any behavior-reshaping post-training stage carries a nonzero capability-erosion risk; the anecdote is a severe, unmitigated instance of exactly that named tax, made worse by off-policy terse data actively targeting the very behavior for removal, with no replay guardrail in place.

Synthesis: SFT loss doesn’t distinguish “correct answer, reasoning genuinely absent from the target distribution” from “correct answer, reasoning omitted for brevity” — it minimizes loss on the tokens present, and if the target tokens never populate a <think> block, zero-reasoning becomes the argmin policy. That policy switch is coarse, low-rank, and cheap to learn (§1.5), the chat-template delimiter makes it a single learnable decision boundary (§1.3), and off-policy demonstration data gives no corrective signal once the model starts drifting toward it (§1.4) — while the loss it is skipping is one later tokens causally depend on (§1.2). None of this requires any weight destruction.

2. Why LoRA did not save it

2.1 The misconception, stated precisely

Misconception: “LoRA prevents catastrophic forgetting, so a LoRA SFT run can’t meaningfully damage the base model’s behavior — at worst it under-learns the new task.”

What’s actually true: LoRA reduces the magnitude of the weight perturbation relative to full fine-tuning, and that magnitude constraint empirically correlates with less forgetting on held-out, aggregate capability benchmarks. That is a statistical claim about weight-space drift — not a guarantee about any specific behavior surviving. Output format/policy (“always think before answering” vs. “answer directly”) is encoded in a direction, not a magnitude, and a low-rank update has more than enough directional freedom to flip a single high-level behavioral switch even while aggregate drift stays small.

2.2 The primary source, read precisely

Biderman et al. (Columbia + Databricks Mosaic), “LoRA Learns Less and Forgets Less,” TMLR 2025, arXiv:2405.09673 (448 citations — well-validated, not fringe). Llama-2-7B/13B, code and math domains, both continued-pretraining and instruction-FT regimes. Their own findings, read carefully, are the crux of “LoRA did not save it”:

• Instruction fine-tuning (IFT) — your regime — forgets more than continued pretraining at matched scale. A synthetic-trajectory SFT run is squarely IFT, the higher-forgetting regime.
• Rank controls the forgetting-protection dial, and it’s a Pareto trade, not a free lunch. r=16 (the common default) forgets least; at r=256 on math IFT, LoRA forgets nearly as much as full fine-tuning (LoRA r=256: 0.567 vs. full-FT: 0.559 at epoch 16 — statistically the same floor).
• LoRA needs a higher LR than full-FT to learn comparably — their recommended range is 5e-5 to 5e-4, about an order of magnitude higher than typical full-FT LR for the same setup. This is the double-edged knob: crank LR up to make LoRA actually learn the target task, and you’re simultaneously moving toward the higher-forgetting regime.
• Full fine-tuning learns perturbations with effective rank 10–100× greater than typical LoRA configurations — this is the structural reason the aggregate-forgetting protection exists at all.
• What the paper’s “forgetting” metric actually measures: held-out, general-capability benchmark scores (commonsense/world-knowledge), aggregated across many unrelated skills. It says nothing about whether one narrow behavioral switch — “emit <think> before answering” — survives. A metric that stays flat in aggregate can hide one behavior collapsing to zero if that behavior is a small slice of what the benchmark measures.

Independent LR corroboration (non-academic, frontier-lab practitioner source): Thinking Machines Lab, “LoRA Without Regret” (2025) — “the optimal LR for LoRA is consistently 10x the one used for FullFT in the same application, for both supervised […] and RL.” Confidence: high (two independent sources converge on ~10x, one peer-reviewed/TMLR, one frontier-lab operational report). A very recent theory paper formalizes why the optimal LR moves with rank — Maximal-Update Adaptation, arXiv:2602.06204 — confidence: medium, very new, not yet broadly stress-tested.

2.3 The mechanistic “why”: intruder dimensions

Shuttleworth, Andreas, Torralba, Sharma (MIT), “LoRA vs Full Fine-tuning: An Illusion of Equivalence,” arXiv:2410.21228 (2024). This is the paper that directly refutes “LoRA forgets less” as an unconditional safety guarantee. LoRA-trained weight matrices develop “intruder dimensions” — new, high-ranking singular vectors absent from the pretrained model’s SVD structure — that full fine-tuning does not produce (FFT perturbs existing directions; LoRA adds new dominant ones). Causally validated: scaling down the intruder-dimension singular values post-hoc reduces forgetting with minimal downstream-task cost, i.e. forgetting is concentrated there, and it gets worse across sequential/continual LoRA rounds. High alpha/LR is exactly where intruder-dimension risk grows (§2.2’s rank-vs-forgetting curve is the same phenomenon from a different angle).

Direct empirical proof the mechanism is real, not hypothetical, in an adjacent domain: Lermen, Rogers-Smith, Ladish, “LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B,” arXiv:2310.20624. Using QLoRA, a single GPU, under $200, they took Llama-2-Chat 7B/13B/70B and Mixtral-Instruct from RLHF safety-refusal down to ~1% refusal rate on two benchmarks — while retaining general-capability benchmark scores. This is the identical shape of failure as “model forgot to think”: a single global behavioral policy (there: refuse harmful requests; here: emit reasoning before answering) collapses to its opposite under LoRA SFT, precisely while the LoRA magnitude constraint is doing its job of preserving general capability. Confidence: high, directly on point — the field has independently proven “LoRA preserves aggregate benchmarks” ≠ “LoRA preserves a specific behavioral policy” in an adjacent domain, using the same mechanism.

Confirming, very recent representation-level evidence: “Representation Collapse in Sequential Post-Training of Large Language Models,” arXiv:2605.30524 (2026) — “LoRA updates from different stages occupy overlapping subspaces,” and “long chain-of-thought tuning may carve a strong reasoning-format manifold… [that] mostly affects late generated tokens” — the literature is converging on modeling format/reasoning-style as its own low-dimensional direction, separable from general knowledge, which is exactly the shape a low-rank update is cheap to capture or destroy. Confidence: medium (very recent, low-citation, but consistent with §2.2/§2.3’s converging picture).

2.4 What LoRA protects vs. does NOT protect

2.5 Independent confounder worth ruling out: rank/alpha instability

If a run used non-default rank with the conventional α/r scaling, it may sit in a genuinely different failure regime: Kalajdzievski, “A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA),” arXiv:2312.03732 (~254 citations, now use_rslora in HF PEFT) shows conventional α/r scaling causes gradient collapse / stunted learning at high rank; the fix is γ = α/√r. A garbled, undertrained high-rank LoRA can also present as “degraded/collapsed output behavior,” distinct from — and worth checking for, alongside — the clean format-collapse mechanism above. Separately, DoRA (arXiv:2402.09353) and PiSSA (arXiv:2404.02948) are both learning-capacity accelerants (they close the LoRA-vs-FFT gap faster/further) — they move LoRA’s behavior closer to FFT’s, which is the wrong direction for behavior preservation specifically; none of the LoRA variants are a forgetting fix. The fix has to come from data composition, next section.

2.6 Concrete knobs, consolidated

None of this is a substitute for §3–4. LoRA discipline reduces risk; it does not remove it.

3. The core lever: mixing ratios + replay/rehearsal

3.1 Replay ratio — the most quantitatively converged lever in this thread

Consolidated knob: 1–10% replay is the empirically converged band for SFT/continual-fine-tuning- scale forgetting mitigation. Scale up toward 5–10% with distribution-shift strength (Ibrahim); prefer general/pretraining-style or reasoning-specific samples over “just more of the old task” if broad- capability retention (not just narrow old-task retention) is the goal (GeRe); if you can afford rollouts, on-policy replay beats vanilla off-policy replay at equal or lower % budget (OPR) — this is the direct fix for §1.4’s off-policy-distribution-shift compounding factor, not just a separate lever. Confidence: Scialom = peer-reviewed, foundational, high confidence but small model (T0-3B) — generalizability to modern 7–70B dense models is argued, not re-verified here. Ibrahim = peer-reviewed TMLR, largest scale, high confidence, but continual pretraining, not SFT — mechanism transfers, exact % may not transfer 1:1 to SFT’s smaller-dataset/larger-shift-per-token regime. GeRe/OPR/Spiegelhalter/ Kotha/Marek are single, very recent (2025–2026), low-citation preprints — directionally strong, structurally sound, not yet independently replicated — “promising, not yet broadly validated.”

3.2 Data-mixing laws — port the principle, not the exact numbers

These are pretraining-domain-mixture papers, not SFT-capability-mixture papers — cited for the general theory that mixture ratio is a first-class, optimizable hyperparameter with a measurable, non-obvious optimum, i.e. the theoretical backbone for “don’t eyeball your SFT mix”:

• DoReMi (Xie et al., NeurIPS 2023, arXiv:2305.10429) — a small (280M) proxy model with Group-DRO finds domain weights transferable to an 8B model; beats the default heuristic mixture by 6.5 points, even when it downweights a domain — the intuitive mixture is measurably wrong.
• RegMix (Liu et al., ICLR 2025 Spotlight, arXiv:2407.01492) — frames mixture selection as regression over hundreds of cheap tiny proxy runs; matches DoReMi at ~10% of the compute. The single most transferable finding for SFT mixing: “domains interact in complex ways often contradicting common sense” — you cannot assume “add X% reasoning data” composes linearly with “add Y% cybersec data.” This is exactly why §5’s per-checkpoint probe, not a static ratio, is non-negotiable.
• Scaling Laws for Optimal Data Mixtures (Shukor et al., Apple, NeurIPS 2025, arXiv:2507.09404) — extends scaling laws to solve analytically for optimal domain weights given a budget, generalizing the proxy-sweep approach further.

Confidence for porting the principle to SFT capability-mixing: moderate — mechanistically sound (same “loss is a smooth function of mixture weights” assumption should hold), but none of these three ran an SFT/instruction-capability-mixing experiment themselves — an explicit extrapolation, flagged as such.

3.3 Instruction-tuning mixture methodology — Tülu’s process, not its numbers

Tülu 2 (arXiv:2311.10702) and Tülu 3 (AI2, COLM 2025, arXiv:2411.15124) disclose the most transparent public SFT-mixture- construction process, not a fixed ratio table: (1) build skill-specific mixtures first, keeping whichever maximizes that skill’s own eval; (2) merge into one candidate mix, then add/remove entire datasets and re-measure the whole suite average — dataset-level, not fine-grained percentage-level, tuning; (3) a directly transferable negative-ratio finding — Tülu 2 explicitly downsampled the oversized FLAN dataset and dropped Dolly entirely for hurting the mix average, i.e. “more data” from one dominant source can be actively harmful, don’t let raw count dictate weight. Confidence: high for methodology (open, reproducible, widely used baseline); exact numeric ratios are project-specific to Tülu’s own skill set — port the iterate-and-ablate process, not the table.

3.4 The capability-balance tradeoff — replay is a dial with two failure modes

Ibrahim et al. also find that at an “extreme amount of replay” the model adapts less to the new domain — there is a real Pareto frontier: too little replay (0–0.5%) forgets; too much (25%+) under-fits the new capability, with diminishing/negative returns above roughly ~10–25% for a strong shift. Biderman’s rank result (§2.2) is the LoRA-specific version of the same tradeoff — a second, orthogonal dial on the same frontier. Confidence: the existence of the tradeoff is high confidence (multiple independent papers, same qualitative curve shape); the exact “sweet band” is regime-specific (continual pretraining vs. SFT vs. LoRA) — re-verify on your own forgetting probe (§5) rather than assume it transfers exactly.

4. Merging/soups vs. mixing — a decision aid

Model merging (combining independently fine-tuned checkpoints in weight space) is the alternative to in-data mixing. Four load-bearing references, all verified live:

• Task Arithmetic (Ilharco et al., ICLR 2023, arXiv:2212.04089) — task vector τ = θ_finetuned − θ_pretrained; negate to suppress a behavior, sum to combine tasks; θ_new = θ + λτ, λ tuned on held-out data. Works because task vectors from different tasks are typically close to orthogonal — the assumption that breaks down when the tasks (e.g. “cyber tool-use” and “keep-CoT”) both touch the same output-formatting circuitry.
• TIES-Merging (Yadav et al., NeurIPS 2023, arXiv:2306.01708) — fixes redundant-magnitude interference and sign interference from naive vector summation via trim→elect-sign→disjoint-merge; reference config: top-k retention ≈20%, λ ≈ 0.8–2.5. What you’d reach for if you had two independently-trained adapters (a cyber-tool-use LoRA and a CoT-preserving LoRA) and wanted to combine without one silently erasing the other in overlapping regions.
• DARE (Yu et al., Alibaba, ICML 2024, arXiv:2311.03099) — SFT delta parameters are extremely redundant; drop 90–99% and rescale, minimal impact, larger models tolerate more drop. Contested at <13B: community reports of incoherent output at aggressive drop rates on 7B models — the safety margin shrinks exactly at the sizes most Sequence-B-style projects run.
• Model Soups (Wortsman et al., ICML 2022, arXiv:2203.05482) — models fine-tuned from the same checkpoint with different hyperparameters land in one low-error basin; weight-averaging (greedy soup) often matches or beats picking the single best, zero extra inference cost. This is the same-task, same-init degenerate case of task arithmetic — the cheapest merge to reach for if you already have N seeds of the same objective.

Decision aid: merge vs. mix

Verdict for the reasoning-retention problem: mix, don’t merge, for this stage. Reserve merging for a later stage (souping multiple seeds of an already-good recipe, or reversible removal of a bad capability). Confidence: high on the qualitative reasoning (orthogonality assumption is well-established and clearly violated here); medium-low on applicability of DARE/TIES numeric operating points to a <13B dense model specifically — none of the four papers were run on an RLVR-fine-tuned reasoning/agent policy at this project’s scale, this is an extrapolation.

5. The practical anti-forgetting recipe (for Sequence-B)

Given: dense open-weight base, LoRA-based cyber-tool-use SFT on synthetic trajectories, a prior naive attempt that already lost thinking behavior.

1. Never generate/accept synthetic teacher trajectories that strip <think> content for brevity or cost. §1.2’s MATH500/JEEBench deltas make this a measured, large capability cost, not a stylistic nicety — and §1.3 shows an absent think block is directly learnable as “the mode.” If a trajectory genuinely has no reasoning trace (e.g. a terse tool-call-only example), loss-mask the answer region rather than let it supervise the think/no-think decision (§1.1’s cheapest, most directly-evidenced fix).
2. Enforce exact template/format consistency with whatever schema the base checkpoint was reasoning-SFT’d/RLVR’d with. If mixing in any non-reasoning examples, tag them explicitly as a selected non-thinking mode (§1.3) — don’t let the model infer “sometimes no-think” from silent omission.
3. Reserve a 5–10% replay slice of reasoning-preserving data every training step/epoch, per §3.1’s converged band — on-policy if compute allows (roll out the current checkpoint on old/generic prompts, filter by a reasoning-format+correctness check, replay the survivors — OPR’s 42–46% BWT improvement over vanilla replay is directly attributable to this), else self-generated replay (arXiv:2605.26097) or a fixed curated reasoning-trace set (arXiv:2508.04676) as cheaper fallbacks. Grid {5, 10, 15%} if budget allows (Spiegelhalter et al.).
4. If staying on LoRA, treat the rank/alpha/LR discipline in §2.6 as risk-reduction, not a forgetting fix. For a task this behaviorally demanding (tool-use + reasoning retention), rank 64–128, α≈2×rank, LR in the 5e-5–5e-4 band, is the reference operating point — but §3’s replay ratio is the lever that actually dominates; if full-FT is available and affordable, it removes the LoRA-rank-vs-forgetting variable entirely and shifts the whole burden correctly onto replay ratio.
5. Prefer rejection-sampled/on-policy-adjacent trajectories over pure off-policy teacher dumps for the cyber-tool-use data itself where feasible — sample from your own base/reasoning checkpoint, verify tool-call correctness against ground truth (this project’s own non-negotiable — see the handbook), keep only correct + format-clean completions. DeepSeek-R1’s own stage-3 SFT corpus does exactly this: ~600K reasoning-domain samples rejection-sampled from the RL checkpoint itself (arXiv:2501.12948) — the on-policy-for-the-touched-domain pattern this whole book’s ordering-rules chapter already establishes as the safe transition. Mention-only, per this project’s standing stance against academic cybersec-LLM grounding — cited as frontier-lab practice, not as an evidentiary source for cybersec-specific claims.
6. Build the “does it still think?” structural probe before the first real SFT run, and run it every checkpoint, not just at the end. Mirror Twist et al.’s valid/empty/missing/truncated trace-rate metric, and report reasoning-conditioned pass@1 alongside raw pass@1 — this is the single cheapest guardrail with direct empirical backing, because §1.1’s central point is that answer-accuracy alone hides the collapse until it’s severe. Concretely, per checkpoint: (a) classify each eval generation’s reasoning-trace validity; (b) track median <think> token count against a fixed eval set — a monotonic collapse toward near-zero is the earliest, cheapest signal, cheaper than a full downstream benchmark; (c) hold a fixed general-capability retention check constant from before the cyber SFT started. Treat a falling think-presence rate as a stop-training signal independent of whether the cyber metric is still improving — “cyber metric up, think-rate down” is exactly the collapse trap the literature documents directly.
7. If you suspect the damage was shallow (task-alignment, not knowledge), a cheap diagnostic exists before committing to a full retrain: Zheng et al.’s spurious-forgetting result (arXiv:2501.13453, ICLR 2025) shows old-task performance can be restored by briefly training on as few as ~10 anchor/alignment instances — none from the original dataset. If a quick recovery probe like this snaps performance back, the failure was mixture/format (squarely this chapter’s levers), not irreversible weight damage.

What I’d change first, concretely, for this project’s actual pipeline: rebuild the current cyber-SFT corpus so every trajectory keeps a <think> block in the exact template the base checkpoint uses; add a 5–10% on-policy (or self-generated) reasoning-replay slice as the default, not an afterthought; and stand up the think-presence probe as a first-class metric before the next training run, not a post-hoc autopsy on the next collapse.

Confidence summary

Cross-links: Ordering rules: interleaving stages & batching for the stage-provenance rules this chapter’s replay recommendation composes with (on-policy replay is the same “restart-for-SFT” discipline, applied within a stage rather than across rounds); Is the recipe a loop? for the macro iterate-and-measure shape the per-checkpoint probe in §5 plugs into; The recipe is a sequence, not a pick for where this single-stage data-composition question sits inside the full stage skeleton; PEFT is orthogonal for the LoRA-as-mechanism framing §2 depends on; Continued pretraining on an instruct model for the closely-related but distinct failure mode (raw CPT breaking instruction-following/alignment format, not reasoning specifically) and its own CPT-on-base-then-re-instruct mitigation; The one axis that predicts everything for the on/off-policy theory §1.4 and §3.1’s on-policy-replay result are direct applications of.