Before you train — instrumentation & data readiness

This chapter is a prerequisite, not a fix. Diagnosing the gap gives you the routing test (knowledge vs. execution vs. exploration); behavioral audit → training signal maps observed failure shapes to methods. Both assume you can already say which stage of a run failed. For most agentic RL projects, you can’t — not because the telemetry is missing, but because nobody has written the small amount of code that turns existing telemetry into a stage verdict. This chapter is a general method for answering “what do I already have, and what’s the minimal thing to build first” for your own harness, worked through on a concrete, representative example rather than left abstract.

Everything below is written as a recommendation to hand to whoever owns the harness — the point is the method and the worked example, not a specific diff against a specific codebase. State your confidence per claim; where you can’t confirm something from source, say so.

1. Why per-stage signal is the prerequisite for the whole diagnosis

The diagnosis framework’s routing test — does the correct action ever appear at high N, and does RL (not matched SFT) expand it? — is defined per challenge or per challenge-subtype, not per portfolio. Run it on the portfolio in aggregate and you get one number that averages over four structurally different failure modes (F1 exploration / F2 skill / F3 tool-use / F4 long-horizon), which is exactly the “collapsing a split verdict into one sentence” anti-pattern Diagnosing the gap §0 warns against. You cannot segment a portfolio you cannot localize. Per-stage signal is what turns an aggregate pass-rate delta into “F1 dropped from the dominant failure mode to a minority, F2 is now the dominant failure mode” — the only shape of finding that tells you which lever (SFT curriculum, tool-use reward, RL exploration term) to pull next.

flowchart LR
    A["event stream<br/>(already emitted)"] --> B["stage extractor<br/>(usually NOT built)"]
    C["per-challenge manifest<br/>(usually NOT built)"] --> B
    B --> D["F1-F4 attribution<br/>per run"]
    D --> E["Diagnosis framework<br/>routing test, per subtype"]
    E --> F["method choice:<br/>SFT curriculum / tool reward / RL exploration term"]

2. What a typical harness already emits — confirm this by source read, don’t assume it

Source: your event-schema module, the code that wires event emission into the agent loop, and whatever doc of record you maintain for the observability contract. In a harness worth building this method on, every event is turn-indexed and tool-call/tool-result pairs join on a stable call id — this is the load-bearing fact that makes any stage-localization script possible with zero changes to the agent loop. If your harness doesn’t have this property yet, that’s the actual prerequisite, ahead of anything below.

Tool surface bounds what a “stage” can even look like: a small, fixed action space (shell exec, file read/write, web search — the usual set for an agentic security or ops harness, confirmed from your own tool registry) with no structured http_request-shaped tool call is the common case. All HTTP interaction with a target is then embedded in free-text shell commands and free-text shell output — there is no machine-readable status/URL/body field. This is the single biggest reason “endpoint discovered” and “vuln identified” are not already derivable cleanly in most such harnesses — any stage parser must regex/parse free text, not read a field.

Confidence: high, if and only if you’ve done the source read — this table is a template; verify it against your own event schema, don’t assume it transfers.

3. Per-stage ground-truth verifier design

3.1 What already satisfies “ground-truth-verified, never transcript-matched”

The common non-negotiable rule for RL on agentic tasks — reward from real environment/tool-output state, never format/regex-matched on the transcript — is usually already met at the terminal (success/flag) stage and nowhere else:

• A deterministic flag/solution scanner classifies every success-string sighting as retrieved (in a tool-result, absent from that call’s own args) / echoed (present in both) / model_claim (only in model text). This is a provenance signal — it proves the string came back from the environment, not the model’s mouth — but is not a byte-compare against the real, known-good answer. A retrieved value from a decoy or off-target leak still reads solved:true.
• The actual byte-compare against a held-out ground truth (call it a verified check, distinct from retrieved) typically lives outside the agent runtime: a tracked ground-truth file plus held-out reference solvers per challenge. Verifying against these is often a manual, credential-gated, human-authorized step — not wired into the harness or any automated post-run check. Grepping the harness source for that verification concept and coming back with zero hits is a common, confirmable finding — a real absence, not a naming mismatch.
• Documented failure modes any pre-terminal verifier must not reproduce: model-claim fabrication after repeated errors (a fine-tuned model claiming solves that a replay disproves is a recurring, real failure mode); a hardcoded success-format regex producing false negatives on an off-format but genuine success string; a points-inflation gap between a cheap proxy signal and the actual verified signal on a leaderboard.

3.1a Tamper-isolation requirement (not just determinism)

Determinism (§3.1) stops proxy-gaming but not reward-tampering — an agent with tool/shell access can subvert the verifier itself, not just the signal it scores (see contested.md §5 for the Denison et al. finding and the isolation requirement). The operational consequence for this harness: whenever your terminal/stage scanner is file- or process-based (reads the event log, reads sandbox tool output) and reachable by the same shell the agent uses to solve the challenge, it must additionally verify from env state the agent cannot write, out-of-band (e.g. the held-out reference solver plus ground-truth file living outside the agent runtime, per §3.1) — not merely a deterministic scan over artifacts the agent’s own tool calls can touch.

3.2 The proposed per-stage predicate design

Phase names below follow a standard penetration-testing execution phase taxonomy (recon → enumeration → detection → exploitation → lateral movement), mapped onto an F1–F4-style failure taxonomy. The mechanism is a direct reuse of the terminal scanner’s proven shape: a pure, read-only, post-hoc scan over the event stream — no new sandbox instrumentation, no changes to the agent’s execution path.

Two things this design deliberately does NOT do, because both violate the common reward rule above:

• Does not reuse an LLM-judge / rubric-matcher mechanism, even if one already exists in your challenge-authoring tooling. Such a mechanism is explicitly a judge (“a span satisfies a matcher when the description’s intent is met, not merely when the regex matches”) — a mid-tier rung on the reward-gameability ladder, and in practice most of your live challenge roster won’t even carry the metadata file that mechanism depends on. Use phase names from that convention if you like; don’t reuse its judging mechanism.
• Does not treat free-text “intent” capture fields as evidence of “identified the vuln.” If your harness has an opt-in reasoning/intent-capture field, treat it as documented observability metadata, low-faithfulness, off by default, and strip it before any training use.

3.3 Potential-based shaping — the caveats if this ever feeds RL reward

If a stage-scan result is ever turned into dense RL signal (rather than just a diagnostic readout), the only shaping form proven not to change the optimal policy is F(s,a,s') = γΦ(s') − Φ(s) for any potential function Φ — Ng, Harada & Russell, “Policy Invariance Under Reward Transformations,” ICML 1999 (no arXiv id — this predates arXiv’s routine ML use; ACM DL 10.5555/645528.657613, verify live before depending on this citation). Two subtleties are easy to get wrong:

1. Φ must be a monotone “best stage reached so far” running max, not the instantaneous current-turn stage — otherwise re-triggering an already-reached signature, or a later turn’s evidence going quiet because the agent moved on, can pay a spurious negative shaping reward for forward motion.
2. Φ must be defined identically across every terminal branch (stop_reason ∈ {stop, max_turns, error}, all real values in a typical harness) — otherwise the invariance proof breaks across the different termination paths variable-length episodes actually produce. Recommended sidestep: apply shaping only over non-terminal transitions; let the terminal reward (unchanged) carry all outcome signal at the very last transition.

Domain-adjacent SOTA, verify live before depending on this table — as of this writing none tick all three boxes (domain-specific + proven invariant + validated against a ground-truth terminal verifier), so this remains an unfilled niche, not a solved-elsewhere problem. Treat any cybersecurity-LLM-training paper you cite here as context, not as a basis for a specific claim/recipe/verdict, unless it produced a frontier-comparable model — academic domain papers are cited for the shaping idea, not as evidence the design below is validated.

Recommended default if this is built: keep the terminal reward and the dense stage-shaping term as two separate additive components, never merged into one function — this is both what makes the invariance argument clean (Ng, Harada & Russell, ICML 1999, cited above) and what a decoupled dense-process-signal + sparse-ground-truth-outcome-signal reward-design doctrine independently favors.

Confidence: high on the harness-reuse mechanism and the Ng et al. invariance result itself (25-year-old, well-established). Medium on the “running-max Φ” / “terminal-consistency” recommendations — applied reasoning from the theorem plus a variable-horizon episode shape, not lifted verbatim from a paper.

4. What training/eval data you already have, per candidate move

Scope: your git-tracked benchmark assets plus wherever the actual trajectory takeouts live (a local run-artifact directory, object storage, or both). If your project has been running sweeps for a while, on the order of hundreds to low thousands of individual agent trajectories across many distinct challenge definitions likely already exist — this is a mining problem, not a collection problem, for most of the four candidate moves below.

Cross-cutting gotchas that apply to all four moves:

• Ground truth typically exists for only a minority of your corpora. Corpora without a held-out success file have only the harness’s own retrieved classification as their “solved” signal — a weaker epistemic tier than exact-match. Flag this explicitly in anything built downstream of them.
• retrieved (tool-result-not-in-own-args) is a strong genuineness signal but is still transcript-level heuristic, not an out-of-band verifier query — the common reward rule is not automatically satisfied just because the harness flagged something retrieved.
• Reconcile any “raw completed count” against a separately-cleaned count before using either quantitatively — a stale local mirror silently diverging from the canonical cleaned number is a common, easy-to-miss trap.

Confidence: high on the general shape of this inventory-and-readiness exercise; the specific counts above are illustrative, not a stand-in for your own numbers — row-count your own corpora directly before using any number quantitatively.

5. The minimal instrumentation gap — the recommended first step

Collapsing §2–§4 into one actionable delta: in most agentic-RL projects, the harness’s process telemetry is already complete for stage attribution. Nothing in the event-emission or agent-loop code needs to change. What’s missing is entirely semantic, and it splits into two independent, additive pieces of work that correctly sit on different sides of a harness/content ownership boundary:

flowchart TD
  A["Pick ONE representative challenge<br/>(one with a fully documented<br/>solve chain)"] --> B["content owner:<br/>author stage_oracle.json<br/>from the reference solver — a<br/>handful of predicates,<br/>editorial work, not infra"]
  A --> C["harness owner: write a<br/>stage-scan module,<br/>same shape as the terminal scanner —<br/>pure io.Reader -> StageScan,<br/>no side effects"]
  B --> D["Run the stage scan over existing<br/>event logs from already-<br/>collected runs"]
  C --> D
  D --> E["Validate: does the stage vector<br/>match what a human reading<br/>the same trace concludes?<br/>(same discipline as the terminal-<br/>scanner validation)"]
  E -->|"holds"| F["Generalize to the rest of the<br/>ground-truth-backed corpus,<br/>then the larger unlabeled pools"]
  E -->|"doesn't hold"| G["Refine predicates before<br/>trusting any F1-F4 number"]

1. Harness side: a deterministic (no-LLM, no-confabulation) URL/path extractor over tool_call.args
   • tool_result.output, scoped to the shell/exec tool only. Emit as a derived per-run artifact, not a new harness event — keeps the harness itself free of challenge-specific semantics.
2. Content-authoring side: one stage-oracle file per challenge — one entry per phase, each a deterministic predicate over (method, path_regex, status_code, body_signature), authored directly from that challenge’s own reference solver. This is editorial work (one person reads each reference solver and writes a handful of predicates), not new infrastructure — the ground-truth reference already exists, it’s just not lifted into a machine-readable file.
3. Prototype on ONE challenge before committing to the whole roster. Pick whichever challenge already has its solve chain fully narrated in prose elsewhere in your notes — its reference solver’s own success checks (e.g. a status-code-plus-token-field check for a bypass step; a success-string regex on a follow-up endpoint for the pivot step) are directly reusable as the exploitation-stage and lateral-stage predicates verbatim. Run the prototype over runs already sitting in your existing results store — no new sweep needed to validate the mechanism.
4. A narrower, more concrete companion gap on the terminal side: turn a retrieved classification into a true verified boolean via an offline exact-match diff against your already-tracked ground-truth file, for the subset of your challenge roster that has one. This needs no live credentials, no manual step, and closes the one place today’s “ground-truth-verified” claim is actually a provenance proxy.

None of this requires RL infrastructure, a new sandbox tool, or a change to the agent’s execution path — it is a read-only scan over data that already exists, validated against traces already collected, before any RL/RLVR reward design depends on it.

Cross-links

• Diagnosing the gap — the routing test this chapter’s stage signal feeds.
• From behavioral audit to training signal — what to do once a failure is localized to a stage.
• Method → Data — the data-object framing this chapter grounds in an actual corpus.