One problem, or many? — monolithic outcome-RL vs staged decomposition

Every other chapter in this book asks “which algorithm” (SFT vs DPO vs GRPO). This chapter asks a question one level up, specific to a sequential, multi-stage task with a single sparse terminal reward: should the CTF solve be trained as one end-to-end outcome-RL problem (flag reward only, let RL discover the stages), or decomposed into sub-problems — evaluated per stage and, more contentiously, trained per stage? The two halves of “decompose” turn out to have very different answers, and conflating them is the single easiest way to get this wrong.

All citations below are carried over, unmodified, from five research threads run 2026-07-02 (artifacts/overnight-decomposition/research/{monolithic-case,decomposition-case,staged-eval,pentest-ctf-rl,credit-assignment-theory,verdict}.md) — no id below was invented for this chapter. Re-grounding pass, same date: per the project’s standing rule, no conclusion in this chapter may rest on a domain-specific academic CTF/pentest training or benchmark paper (CTF-Dojo, Cyber-Zero, Pentest-R1, HackSynth-GRPO, AutoPenBench, Cybench, NYU CTF Bench, EnIGMA, InterCode-CTF, DRLRM-PT, node-fragility shaping, the kill-chain-staged-reward paper) — every such work is demoted to a labelled context-only mention below, and every claim that had rested on one is re-anchored on general frontier-lab / RL-theory evidence or this project’s own data instead. Two new citations were added and independently verified live for this pass: STaR (arXiv:2203.14465) and ReST-EM (arXiv:2312.06585), both general (non-security) self-training literature, replacing CTF-Dojo/Cyber-Zero as the basis for §5’s rejection-sampling-SFT recommendation.

1. The pipeline, and why the failure isn’t uniform

A CTF solve is not one action, it’s a chain:

flowchart LR
  R["Recon /\nenumeration"] --> E["Endpoint\ndiscovery"]
  E --> V["Identify the\nvulnerable endpoint"]
  V --> X["Exploit it"]
  X --> P["Post-exploitation /\npivot"]
  P --> F(("Flag\n{0,1}\nground-truth verified"))

  classDef stage fill:#132b22,stroke:#34d399,color:#eafaf3;
  class R,E,V,X,P stage;

Only the last box is ground-truth-checkable today. Observed failures cluster by where in the chain the agent dies, not uniformly across it — the project’s own F1–F4 taxonomy, which turns out to be a CTF-specific instance of failure clusters the general agent-eval literature keeps independently rediscovering (§4).

The credit-assignment theory thread makes the split precise: a ~100-turn trajectory with reward only at the end is hard for three separable reasons — exploration burden (F1, upstream of everything else), credit-assignment variance (Monte-Carlo/GRPO-style returns smear one scalar across all turns — arXiv:1506.02438, GAE), and compounding distributional drift (a policy trained once, on one static snapshot, drifts off-distribution as a rollout gets longer — arXiv:1011.0686, DAgger, classically O(εT²) uncorrected). F1 is the first problem; F2–F4 are flavors of the second and third. Don’t expect one fix (a denser reward) to solve both (arXiv:2312.01072, the credit-assignment-vs-exploration survey).

2. Side A — the monolithic case, steelmanned

The pattern across every lab that tried both is consistent: outcome-only + scale beats hand-built process supervision, every time it’s been A/B’d.

• DeepSeek-R1-Zero rejects process reward outright, in its own failure-experience writeup. Pure RL, no SFT, rule-based outcome-only reward; reasoning behaviors emerge as a side effect. Verbatim: “a model-based PRM… inevitably leads to reward hacking, and retraining the reward model needs additional training resources and it complicates the whole training pipeline.” arXiv:2501.12948. This is the single strongest evidence against a learned/neural per-stage reward — note precisely what it doesn’t rule out: a deterministic, ground-truth per-stage check is a different animal (§4).
• OpenAI Deep Research shipped a long-horizon, tool-using agent trained end-to-end on outcome/rubric reward, and its own team says why: “End-to-end training beats manual orchestration… constructing a graph of operations… is the common approach to building agents [but] Deep Research is trained end-to-end… This allows the model to develop flexible strategies… that would break if scripted manually.” (OpenAI Deep Research system card, no arxiv id — flagged as such.) Closest real-world analog to this project’s shape: long-horizon, tool-using, sparse/rubric-graded.
• Kimi k1.5 gets SOTA reasoning results with no PRM, no MCTS, no value function — substituting long-context scaling for explicit search. arXiv:2501.12599. Second independent lab, same conclusion as R1.
• Llama 4’s own post-training team found heavy SFT/DPO caps the ceiling of the subsequent RL stage — verbatim: “SFT and DPO can over-constrain the model, restricting exploration during the online RL stage.” They responded by pruning >50% (95% for Behemoth) of their SFT data (ai.meta.com blog, no arxiv id). This is the project-critical citation: a hand-imposed stage boundary is itself a form of prescriptive pre-RL structure, and this is the general mechanism by which imposed structure narrows a policy’s exploration before RL gets to use it.
• Academic, cited for context only, not a basis (per project standing rule — no domain-specific academic CTF/pentest training paper has produced a frontier cybersecurity model): in the CTF/pentest domain specifically, the same monolithic-wins pattern is also reported by academic training papers — CTF-Dojo (arXiv:2508.18370), Cyber-Zero (arXiv:2508.00910), Pentest-R1 (arXiv:2508.07382), HackSynth-GRPO (arXiv:2506.02048). None of these is load-bearing here — the actual basis for Side A is the frontier-lab evidence directly above (R1, Kimi k1.5, OpenAI Deep Research, Llama 4, Bitter Lesson), which independently converges on the same conclusion without needing a CTF-specific data point.
• The Bitter Lesson (Sutton 2019, incompleteideas.net) is the intellectual ancestor of all of the above: hand-built structure plateaus, general search+learning wins at scale. High confidence the historical pattern is real; medium-low confidence it transfers directly to a 1000-challenge corpus with real per-rollout infra cost — that’s exactly the disanalogy the honest limits below press.

The honest limits — where the monolithic case’s own literature admits it breaks down

Bottom line for Side A: thick, convergent, in-domain evidence that monolithic-outcome-plus-better-data/curriculum wins whenever it’s been tried against a decomposed alternative in LLM-CTF training specifically. The genuine open risk it must own is Go-Explore’s — whether the ~100-200/1000 solve rate reflects “still finding it eventually with more rollouts” (favors monolithic) or “structurally not finding it” (favors an exploration-specific intervention) is an empirical question literature alone cannot resolve.

3. Side B — the decomposition case, steelmanned

Framing. Monolithic outcome-RL is implicitly betting on four things at once: (1) the base policy already puts non-zero mass on the correct trajectory shape for every stage on a large fraction of challenges, (2) the RL algorithm can correctly attribute a late reward to the right subset of ~100 turns, (3) one scalar is expressive enough to teach four qualitatively different skills (exploration breadth, exploit depth, tool discipline, chaining) without one skill’s gradient starving another’s, and (4) “more outcome RL” is uniformly the right lever for F1 through F4 alike. Every technique below is a documented failure mode of at least one of these assumptions.

The menu of decomposition mechanisms

The honest limits — where decomposed training breaks, concretely

The reward-hacking evidence against naive per-stage reward is thick and convergent, not one paper. The moment a “verifier” stops being a deterministic ground-truth check and becomes a learned/judge score, this project’s own confirmed lesson (SFT-induced FLAG{} confabulation from a loose format-matcher) generalizes into a much larger literature:

• PURE / Stop Summation (arXiv:2504.15275) names the mechanism precisely: the canonical summation-form credit assignment (additive, per-step reward) “easily induces LLMs to hack steps with high rewards.”
• Reward Under Attack (arXiv:2603.06621) shows SOTA PRMs function as “fluency detectors rather than reasoning verifiers” — >0.9 PRM reward on trajectories with <4% ground-truth accuracy.
• Gao et al. (arXiv:2410.15115) — combining a learned PRM/ORM with success reward can hurt relative to success-reward-only, via “repeating correct but unnecessary steps.”
• PRIME’s own authors (arXiv:2502.01456) state the central open problem is that process labels are “prohibitively expensive… making [PRMs] particularly vulnerable to reward hacking,” and route around a separately-trained PRM entirely for this reason.
• MONA (arXiv:2501.13011, DeepMind) generalizes this: multi-step reward hacking can occur even when no single step looks bad to a human/judge overseer.
• The ancestor of all of it: the bicycle-shaping failure (Randlov & Alstrom, ICML 1998, pre-arxiv) — a non-potential-based “looks like progress” bonus taught an agent to ride in tight circles farming the bonus instead of reaching the goal. Same species as this project’s own SFT confabulation lesson: reward emitting the right-looking pattern rather than deterministically verified success, and the policy learns to farm the pattern.

Ceiling-capping is a real cost of decomposed training too, symmetric to Llama 4’s SFT/DPO warning. HIRO (arXiv:1805.08296) needed an explicit off-policy correction specifically because a high-level subgoal’s meaning drifts as the lower-level policy improves during training — without it, the system converges on subgoals that are locally useful but cap out below the true optimum. Translated to F1–F4: an “exploit-only” sub-policy trained against a synthetic “endpoint identified” subgoal risks converging on the shallowest exploit that satisfies the boundary — which is precisely the F2 shallow-probing failure this project already observes, not a hypothetical.

Academic, cited for context only, not a basis: the domain-specific CTF/pentest training papers named above happen to be monolithic-outcome or curriculum-decomposed rather than reward-decomposed, and the one adjacent paper doing genuine staged reward in an LLM+RL security loop (arXiv:2605.17075) is unreplicated — but neither observation is the basis for caution here. The actual, load-bearing case against naive per-stage reward is the general reward-hacking convergence immediately above (PURE, Reward Under Attack, Gao et al., PRIME, MONA, HIRO) — that literature alone is sufficient to warrant the conditional verdict in §5, independent of what the CTF-training corpus does or doesn’t show.

4. The key split — eval-decomposition vs training-decomposition

This is the load-bearing distinction. They are not the same decision, and the evidence supports very different confidence levels for each.

The theory’s own framing of why these are different decisions: per-stage evaluation is just better logging — nothing is being optimized against it, so it carries none of the correctness burden. Per-stage reward is where every failure mode above lives, because now something in the loop is being optimized against the signal. This is why the project brief is right to force these into two separate decisions.

The one empirical finding that turns this from philosophy into an operational rule. A controlled study across the RL design space on TravelPlanner finds: “reward and algorithm choices are scale-dependent — smaller models benefit from staged rewards and enhanced exploration, whereas larger models converge efficiently with simpler dense [outcome-only-adjacent] rewards.” arXiv:2603.21972. This is directly checkable via the eval funnel: is the current baseline “occasionally stumbles onto stage 3” (favors staged help) or “reliably reaches stage 3/4, fails to convert” (favors leaving outcome reward alone and attacking execution depth via data/SFT)? The project’s own diagnosis — “largely an execution gap” — leans toward the latter, but this is an empirical call the funnel should confirm, not an assumption to bake in from literature alone.

5. The verdict for this project

The concrete next step this whole verdict depends on

The project’s own PTES matcher schema (benchmark/challenges/*/challenge.json, ptes.<phase>.steps[]) was arrived at independently, before this literature review, on the general non-security basis in §4 (AgentBoard, MAST, AgentErrorTaxonomy). Academic, cited for context only, not a basis: it happens to structurally resemble the Cybench-subtask / AutoPenBench-milestone design too. What’s missing is aggregation: run the existing triage subagent over completed runs and emit one funnel row per challenge (five phase-reached booleans + an exploit-given-vuln-found conditional rate), rolled up into a corpus-level funnel. This single aggregation is the input every downstream decision below depends on — it settles empirically whether the ~100-200/1000 solve rate is F1-dominated, F2/F3-dominated, or F4-dominated, which the flag_verified column alone cannot supply no matter how much data accumulates.

The recommended recipe, cheapest and safest first

1. Decompose the eval fully, now, unconditionally (§4). Cross with the project’s own pass@k methodology rather than one aggregate pass@5, per tau-bench’s pass^k precedent.
2. Keep the terminal flag reward as the ground-truth backbone, unconditionally. Nothing in this dossier argues for demoting it below a milestone signal.
3. Run rejection-sampling SFT on own verified solves as already planned — but keep it light. The general, non-security basis for this move is the frontier self-training line: STaR (arXiv:2203.14465, Zelikman et al. 2022) — the seminal “generate, keep only what’s verified correct, fine-tune, repeat” loop — and ReST^EM (arXiv:2312.06585, Singh et al., DeepMind 2023) — the frontier-lab scaling result showing this expectation-maximization-style self-training on a model’s own correct samples beats training on human data alone, on math/code reasoning, no cybersecurity domain involved. Academic, cited for context only, not a basis: CTF-Dojo and Cyber-Zero report the same pattern (~500 verified trajectories → double-digit gains, no staged reward) inside the CTF/pentest domain specifically — a reassuring domain-match, not the reason to do this. Respect the Llama 4 warning: don’t over-train on the easy/repetitive subset — it narrows the exploration space the subsequent RL stage needs.
4. Add curriculum sequencing before touching the reward function at all. The single lowest-risk lever available — no new reward, hence none of §3’s hacking surface. Order by whichever axis the funnel identifies as the bottleneck.
5. Only if the funnel shows an F1 (exploration)-dominated bottleneck, and only via a ground-truth mechanism: add potential-based milestone shaping on top of the terminal reward. Define Φ(s) as a monotonic count of deterministically-verified stage completions (same verification contract the flag oracle already uses — server-side checks, not judge opinions), paid once per stage-transition, never re-collectable. Do not build this for stage 3 (vuln identification) specifically — VPR’s own authors flag exactly this stage-shape (“identify which of several candidates is vulnerable”) as the “open, unstructured” regime their method doesn’t yet solve well; keep that stage eval-only until a genuine deterministic check exists.
6. Explicitly do NOT build a learned/LLM-judge per-stage reward model. Every citation in §3’s honest-limits section converges on this being the failure mode to avoid.
7. If the funnel instead shows an F2/F3 (execution-depth / tool-policy)-dominated bottleneck — which the project’s own current diagnosis (“largely an execution gap”) suggests is more likely — the evidence base points away from reward decomposition and toward better trajectory curation and more/better SFT data, not a training-loop change.
8. When entropy collapses under GRPO (the project’s own stated graduation trigger), watch stage-transition tokens specifically — a badly-shaped milestone reward is an easy, low-entropy shortcut to farm, and would accelerate collapse.

Deliberately not recommended as a first move: standing up four independently-trained sub-policies with four separate critics (the full options/ArCHer/HiPER/FeUdal-style architectural decomposition). Real, actively converging in the literature, and MiRA’s 6.4%→43.0% number is the strongest existence-proof in this whole dossier that monolithic reward can leave a large gap on the table — but every one of these is validated on web-navigation or generic agentic benchmarks with crisp, cheap-to-verify milestones, not on an offensive-security CTF corpus. Highest-upside, least-validated-for-this-domain lever here — a candidate for a later, small, gated experiment, not step one.

The decision, as a diagram

flowchart TD
  Start["Failing challenge / corpus\nunder diagnosis"] --> EvalDecomp["Step 1 — decompose the EVAL\n(PTES funnel, near-free)\ndo this unconditionally"]

  EvalDecomp --> Funnel{"Funnel shows which\nbottleneck dominates?"}

  Funnel -->|"F1: rarely reaches\nthe vulnerable endpoint"| ScaleCheck{"arXiv:2603.21972 —\nweak policy, capacity-limited?"}
  Funnel -->|"F2/F3: reaches it,\nfails to convert / clumsy tools"| Mono["Stay monolithic.\nInvest in trajectory curation +\nrejection-sampling SFT data\n(STaR / ReST-EM pattern)"]
  Funnel -->|"F4: no pivot after\na foothold"| Curric["Curriculum first\n(1-hop before 2-hop pivot chains,\nno reward change)"]

  ScaleCheck -->|"yes — occasionally\nstumbles onto it"| Shape["Potential-based milestone\nshaping ON TOP OF the flag reward\n(Ng/Harada/Russell 1999 — provably\npolicy-invariant), NOT stage 3"]
  ScaleCheck -->|"no — already capable,\njust unreliable"| Mono

  Shape --> Guard["Guard: ground-truth verifier only,\nnever a learned/LLM-judge score\n(PURE / Reward-Under-Attack / MONA)"]
  Mono --> Entropy["Watch entropy at GRPO\ngraduation regardless of path taken"]
  Curric --> Entropy
  Guard --> Entropy

  classDef safe fill:#132b22,stroke:#34d399,color:#eafaf3;
  classDef risk fill:#3a1414,stroke:#f87171,color:#fde8e8;
  class EvalDecomp,Curric,Mono safe;
  class Shape,Guard risk;

Contested point, stated plainly: there exists one paper doing genuine kill-chain-staged RL reward inside an LLM+RL hybrid (arXiv:2605.17075, cyber-defense red-teaming) that on its face looks like support for training-decomposition on an adjacent task. Per the project’s standing rule it carries no evidentiary weight here regardless — it’s academic cybersecurity-LLM work, cited for context only. The Medium-confidence, conditional verdict above does not rest on it; it rests on the general theory (potential-based shaping’s policy-invariance guarantee, HIRO’s ceiling-capping mechanism, the PRM-hacking convergence) and on this project’s own diagnosis. Demoting this citation changes nothing about the verdict.

Cross-links

• Diagnosing the gap — a scientific framework — the pass@k / Pass@(k,T) / Cover@τ protocol that tells you which gap (knowledge / execution / exploration) a challenge subtype actually has; run this before deciding whether an F1-dominated funnel result calls for exploration-RL or just more samples. That chapter’s routing test and this chapter’s eval-funnel are complementary diagnostics, not competing ones.
• RL that creates value — long-horizon, exploration, reasoning, novelty — the mechanics of how to fix an exploration or credit-assignment gap once diagnosed here (GiGPO step-level credit, DAPO, entropy instrumentation, ArCHer, curriculum-band filtering) — this chapter answers whether to decompose; that one answers how to execute the fix on the training-loop mechanics.
• Agentic & multi-turn RL — the missing category — the training-loop shape (turn as the unit of advantage) that any of §5’s mechanisms (potential-based shaping, curriculum) has to be implemented inside.
• Contested edges & landmines — the “does RL create capability or just amplify it” fight this chapter’s scale-dependence finding (arXiv:2603.21972) directly informs.

Bibliography (all traced to a verified source file, 2026-07-02)