Hint-guided bootstrapping — put the walkthrough in the prompt, then train it away

The kinds-of-sft taxonomy just gave you the axes a training row is built from — who produced the trajectory, executed vs. synthetically authored, verifier-filtered or not. This chapter takes one specific point on that grid — on-policy, executed, hint-seeded — and follows it end to end: from a plausible cold-start idea, to the four-decade-old literature it belongs to, to its measured failure modes, to a concrete recipe.

You’ve independently arrived at an idea for cold-starting a sparse-reward agentic task: a fresh policy solves ~0% of the hard cases, so there’s no gradient, nothing to learn from. Your fix — put the solution/walkthrough for that exact task in the system prompt, let the model execute the task for real against the actual environment (real tool calls, real stdout, a real pass/fail check), then mask the walkthrough out of the recorded trajectory and fine-tune the model, supervised, to produce that same executed trajectory when given only the bare task. Your own framing: it’s an exploration task; hand the agent the map, harvest its own walk to the goal, then take the map away.

This chapter’s first job is to tell you: this is not novel, and that’s good news. You’ve reinvented a family the literature has been building out under at least four different names since 2009, each naming a different facet of the same operation. The second job is the harder one — the family has real, measured failure modes, and at least one of them lands directly on the specific design choice (“full walkthrough,” not an abstract cue) you proposed. This chapter names the family, shows the mechanism precisely, tables the evidence, and gives you the honest risk register before you spend compute on it.

Framing: general research reference — the technique family, not any one project’s pipeline. Status: every arXiv id below was verified live against arxiv.org abstract pages (title, authors, date cross-checked against body text, not a bare crawler title) during the research pass this chapter draws on; confidence is stated per section, and — per NuRL’s own ablation (§4) — the chapter does not conclude “this works, go build it.” It concludes “this is a real family with one specific, well-evidenced caveat that lands on your exact design,” which is a more useful and more honest answer.

1. The idea, and its real names

Four bodies of work converge on the same four-step move, each catching a different piece of it:

• STaR-rationalization (Zelikman, Wu, Mu, Goodman, arXiv:2203.14465) — the generate-under-hint-then-mask mechanism at the level of a single scalar hint: append the correct final answer to the prompt, let the model generate a rationale that reaches it, train on the (question → rationale) pair with the hint stripped out.
• Context / prompt distillation (Askell et al., arXiv:2112.00861; Snell, Klein, Zhong, arXiv:2209.15189) — the same mechanism scaled up to a rich hint: a whole instruction block, scratchpad, or system prompt, internalized into the weights by training the model to reproduce hint-conditioned outputs without the hint present.
• Hint-guided exploration for sparse reward (NuRL, arXiv:2509.25666; HiLL, arXiv:2604.00698; ReGFT, arXiv:2603.01223) — the RL-training-loop framing: privileged guidance is what recovers a non-degenerate reward/advantage distribution when every rollout in a group currently fails, and the field has explicit vocabulary and metrics for whether that guidance was genuinely necessary and whether it transfers once removed.
• Learning Using Privileged Information / asymmetric actor-critic (Vapnik & Vashist, Neural Networks 2009, DOI:10.1016/j.neunet.2009.06.042; Pinto, Andrychowicz, Welinder, Zaremba, Abbeel, arXiv:1710.06542) — the formal name for “the teacher sees information at train time the student will never see at deployment,” which is the abstraction all three techniques above are instances of.

None of these is “solved” in the sense of a settled best-practice recipe. All four have real evidence they work, and all four have documented conditions under which the same recipe silently produces a model that looks trained but hasn’t actually learned the thing you think it has. That’s the substance of this chapter.

2. The mechanism: exploration compiled into supervision

What the hint is doing. An untrained (or weak) policy facing a sparse/binary reward has to find the solution by search — an exponentially large space of tool calls and token sequences, almost none of which lead to a verified success. A hint (an answer, a scratchpad, a full walkthrough) collapses that search: it narrows the space down to a specific, concrete, correct path at generation time. Once that path exists as a real (input → output) pair, imitating it is a supervised problem — dramatically cheaper and more stable than making the model rediscover the same path from scratch via RL exploration in a sparse-reward regime. This reframing — “the hint pays the exploration cost once, then supervision harvests it” — is the single sentence that unifies STaR-rationalization, context distillation, and hint-guided RL.

Grounded-and-executed beats confabulated-and-narrated. There are two very different ways to produce the (task, trajectory) pair the hint enables. One: ask a model to write a plausible-looking solve — no environment involved, nothing actually ran. Two: give the model the hint and let it act against the real environment — real tool calls, real stdout/stderr, a real pass/fail check. Your proposal insists on the second. This is a genuine, load-bearing design choice, not a detail: a narrated rationale can confabulate a result that never happened (this is the exact failure mode the field calls unfaithful chain-of-thought — biasing a prompt toward an answer produces plausible-but-non-load-bearing reasoning text, dropping accuracy up to 36% on BIG-Bench Hard when the bias points at a wrong answer, Turpin, Michael, Perez, Bowman, arXiv:2305.04388) — while an executed, verifier-checked trajectory structurally cannot silently hallucinate its own success. That’s real protection against one failure mode. It is not, on its own, protection against the other one (§4) — a real, verified trajectory can still have been shaped by the hint in ways that don’t transfer.

Mask-the-hint is the training-time move, formalized twice.

Snell et al. give the exact loss for the general case — teacher (with hint) supervises student (without hint), same underlying model, hint stripped by an extractor f before the target is built:

L(θ_student) = E_x~D [ E_y~P_θteacher(·|T_teacher(x)) [ log P_θstudent( f(y) | T_student(x) ) ] ]

T_teacher(x) = [hint/instructions/walkthrough] + x        # what generates y
T_student(x) = x                                            # what the trained model sees
f(y)         = y with the hint/scratchpad stripped out      # what the trained model must produce
θ_teacher stays frozen — it's the same model, hint present, at generation time only

STaR/TRICE give the same move a probabilistic reading, one level more mechanistic. Treat the rationale (or trajectory) z as a latent variable; the real objective is the marginal likelihood of the correct outcome y given the task x, marginalizing over all possible z: log p_θ(y|x) = log Σ_z p_θ(y|x,z) p_θ(z|x). You can’t compute that marginal directly — you need a sample from the posterior p_θ(z|x,y), “the trajectory distribution given that we already know it succeeds.” The hint-conditioned generation step is exactly an approximation of drawing from that posterior. TRICE (Phan et al., arXiv:2312.02179) formalizes STaR-rationalization as one-sample, uncorrected MCMC against that posterior, and proves it is a biased stochastic-EM estimator: the true gradient of the marginal log-likelihood is p_θ(y|x) ∇_θ log p_θ(y|x) — weighted by how likely the unhinted model already was to succeed. That weighting means the gradient signal shrinks toward zero exactly on the hardest examples — the ones where the hint is doing the most work relative to the model’s own competence. This is the formal version of “did I really teach it something, or did the hint just do the work” — and it’s the mathematical spine under NuRL’s elicit-vs-expand framing (§4).

LUPI names the shape of the whole thing. Vapnik’s framing: training triplets (x_i, x*_i, y_i) where x* (privileged info) is available only at train time; the learned function f: X → Y must use only x at deployment. Your writeup is x*; the executed-and-masked trajectory is the (x, y) the model is trained on. The theoretical payoff, when privileged info is genuinely informative, is a faster generalization-error convergence rate (O(1/n) vs O(1/√n), the “Oracle SVM” analysis) — but the empirical follow-up is candidly mixed: randomly-generated “privileged” features can perform similarly to genuinely meaningful ones in some SVM+ settings (Serra-Toro, Traver, Pla, Pattern Recognition Letters 2014) — privileged-at-train-time is not automatically privileged-and-useful. The asymmetric actor-critic lineage (Pinto et al., arXiv:1710.06542) is the deep-RL sim-to-real instance of the same idea done right: the critic sees full simulator state, the actor only ever sees what it will have at deployment — and it’s now the standard recipe precisely because the privileged channel there (ground-truth physics/pose) is verifiably informative, not noise.

3. Named techniques — what each contributes, and where each breaks

4. Failure modes — named, measured, and the one that lands on your exact design

Three distinct risk classes, not one — conflating them leads to the wrong mitigation.

(a) Shortcut learning / Clever-Hans / hint-copying

The general phenomenon: a decision rule that performs well on the training distribution by exploiting a cue that isn’t the actual task (Geirhos et al., “Shortcut Learning in Deep Neural Networks,” arXiv:2004.07780 — 1650 citations, named for Clever Hans, the horse that appeared to do arithmetic by reading its handler’s involuntary cues). In NLP specifically: BERT-class NLI models adopting fallible syntactic heuristics that collapse on a controlled challenge set (HANS, McCoy, Pavlick, Linzen, arXiv:1902.01007); the same effect on COPA, with the sharp finding that the same recipe can produce a Clever-Hans learner or a genuine one depending on architecture (Kavumba et al., arXiv:1911.00225); a detection technique — strip words from the input while confidence stays unchanged, meaning the model wasn’t reading the content at all (RAWR, Feng et al., arXiv:1804.07781).

The version specific to this proposal is sharper still. Peng et al., “Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation,” arXiv:2602.14469 studies exactly “synthesize a trace from a known (query, answer) pair” — precisely “write the trajectory knowing the destination” — and formalizes “answer as cognitive anchor” across lexical/entropic/probabilistic axes. Their most important negative result: the naive mitigation, “tell the model to ignore the answer while generating,” backfires — it reduces surface lexical overlap with the hint but increases entropic/probabilistic anchoring, meaning the model leans on the hint just as much internally, only hides it better. Direct implication: masking the hint text out of the trajectory is necessary but not sufficient. The model’s tool-call sequence itself — target selection, the order it tries things — can still be suspiciously well-aimed relative to what an unhinted agent would ever discover, even with every trace of the writeup’s literal text gone from the training row.

(b) Distribution mismatch / off-policy exposure bias

Named explicitly by OPCD: the classical Askell/Snell recipe trains on a frozen, once-sampled teacher-conditioned completion — the student never generated it — which produces exposure bias (trained on teacher-shaped sequences, must autoregress its own at test time) and mode-covering forward-KL (spreads mass across all teacher behaviors instead of committing to a coherent policy). ReGFT names the same failure from the SFT side: raw expert/reference text sits outside the model’s own reasoning distribution, and training on it directly teaches imitation of an alien voice, not the underlying skill — their explicit fix is to have the model itself generate the continuation, guided-but-not-copying the reference. Your proposal’s insistence on executing the trajectory rather than distilling the writeup’s text directly is already the correct mitigation here, by construction — keep it.

(c) Elicit-not-expand

The sharpest open question, and the one with a formal answer (§2’s TRICE result) and an empirical instrument (NuRL’s pass@1024 metric): does hint-guided training expand what the policy can do, or does it just make an already-latent, rarely-sampled solution more probable? NuRL’s headline finding is that done right (self-generated, abstract hints, gated on genuine 0%-pass-rate difficulty, applied late — after unhinted training has already converged) it does raise the model’s true ceiling (pass@1024), which vanilla GRPO cannot move. But their own ablation is the load-bearing caveat for you specifically: revealing the gold answer/full solution directly hurt performance relative to an abstract hint — a full CTF walkthrough sits at exactly the dangerous end of that spectrum, closer to “here’s the answer” than to “here’s the abstract cue.” This is not a reason to abandon the idea; it’s the single most concrete, most actionable finding this literature offers against the specific form (“full walkthrough in system prompt,” not “abstract hint”) you proposed.

Detection — ablate the hint, don’t trust the narration

• HiLL’s hint-reliance check, operationalized cheaply: re-roll the same task from the same checkpoint without the writeup, N times; check whether the hinted trajectory’s key decision points (target selection, tool choice, the specific exploit path) show up in any of the unhinted rollouts. Low overlap = high reliance = the hint did the work, not the model.
• NuRL’s pass@1024-style check: hold out tasks that never had a writeup generated for them at all (not just masked at inference — never authored, so the eval set itself can’t leak task-specific info from writeup-authoring). Compare the SFT’d model’s pass rate on those to the pre-SFT base model’s. A genuine capability gain shows up there; a hint-shortcut gain does not.
• Do not trust the trajectory’s own narrated reasoning as evidence of non-reliance. RL optimization pressure is documented to increase reliance on prompt-present hints without proportionally increasing disclosure of that reliance in the model’s own chain-of-thought — the model can use the hint and simply not say so. The ablation checks above are load-bearing precisely because introspection is not sufficient evidence.

Mitigate

1. Mask thoroughly, then audit for residual fingerprints — not just “is the writeup’s literal text gone,” but whether the tool-call order and target selection look anomalously well-aimed compared to the model’s other, unhinted successful trajectories on structurally similar tasks (Peng et al.’s finding that masking-only backfires against internal anchoring).
2. Fade the hint, don’t cut it binary. Multiple independent lineages converge on the same fix: graded hint depths rather than one full walkthrough (Zhang et al., “Multi-level Stepwise Hints,” arXiv:2507.02841); scheduling easy-to-hard curricula to prevent overfitting to the assistance signal (E2H Reasoner, arXiv:2506.06632; AdaRFT, arXiv:2504.05520); and explicit annealing of an auxiliary demonstration loss as the policy’s own competence overtakes the demonstrator (Nair, McGrew, Andrychowicz, Zaremba, Abbeel, arXiv:1709.10089) — the clearest classical precedent for “hint fading” as a design pattern, not a novelty.
3. Verify unaided, on tasks that never had a hint at all, per the detection section above — the single check that actually distinguishes expansion from elicitation.

5. Verdict, and the concrete recipe

Honest verdict. The mechanism is real and well-precedented across four independent lineages spanning 2009–2026; “hint-conditioned execution, then strip-and-train” reliably converts intractable exploration into tractable supervision, and grounding it in a real, verifier-checked environment (rather than a narrated rationale) is a genuine, non-trivial improvement over the weakest member of this family (vanilla STaR-rationalization against a string-match check). But this is contested at the specific design point you proposed, not in general: the one paper in this family that directly ablates “full answer/solution vs. abstract hint” (NuRL) found the full-solution end of that spectrum hurts relative to an abstract cue, and no paper surveyed here tests the exact setting — a full external walkthrough, executed in a live tool-using sandbox, verified against a real target — end to end. Treat the mechanism as validated and the specific recipe (full walkthrough, not graded/abstract hints) as the open empirical risk to test, not to assume away.

The concrete recipe, incorporating the mitigations above rather than the naive one-shot version:

graph LR
  A["Writeup / solution<br/>in SYSTEM PROMPT<br/>(privileged hint)"] --> B["Agent EXECUTES<br/>in the real environment<br/>(real tool calls, real output)"]
  B --> C{"Verifier check<br/>(e.g. flag_verified)"}
  C -->|"fails"| D["Discard, or keep as a<br/>hinted-failure negative<br/>(V-STaR-style contrast signal)"]
  C -->|"passes"| E["MASK the writeup out of<br/>the recorded trajectory"]
  E --> F["Hint-reliance filter:<br/>re-roll unhinted N times,<br/>drop trajectories with<br/>near-zero unhinted overlap"]
  F --> G["SFT: task-without-hint<br/>-> executed, masked trajectory"]
  G --> H["FADE hint strength across<br/>a curriculum: full walkthrough<br/>-> abstract cue -> none"]
  H --> I["VERIFY UNAIDED on held-out<br/>tasks that never had a<br/>writeup authored at all"]
  I -->|"gap holds vs. pre-SFT base"| J["Genuine capability expansion"]
  I -->|"gap closes / vanishes"| K["Hint-shortcut — degrade the<br/>hint toward abstract, re-run"]

Reading it end to end: the loop is not “run this once and trust the flag check.” The flag check (step C) rules out confabulation, which is real but is only one of the two failure modes this family documents. Step F and step I are what rule out the other one — hint-copying and elicit-not-expand — and neither is optional if you want to trust the resulting model’s behavior on a task it’s never seen a writeup for, which is presumably the entire point of doing this at all.

Cross-links

• The kinds of SFT — it is the data, not the algorithm — this chapter’s masked, executed trajectory is a specific instance of that chapter’s Axis A/B/C (self-generated, executed, verifier-filtered) — the highest-signal cell in that taxonomy, with one extra wrinkle (the trajectory was generated under privileged information) that chapter doesn’t cover and this one does.
• Method → Data (your real bottleneck) — hint-guided bootstrapping produces the same “full trajectory” data object that chapter’s SFT row already names; the difference is entirely in the generation process (hint-conditioned, then masked), not in the shape of the resulting row.
• RL that creates value — long-horizon · exploration · reasoning · novelty — the [E]/[N] exploration and novelty-tagged techniques there are the RL-native cousins of the same problem this chapter solves via a one-shot SFT harvest; read that chapter for the online-RL version of “recover a gradient when a group is all-fail.”
• Is the recipe a loop? — the fade-the-hint curriculum in §5 (full walkthrough → abstract cue → none) is a specific instance of that chapter’s question: does a stage run once or get revisited, and from which checkpoint does the next hint-level start.
• Contested edges & landmines — the elicit-not-expand tension named here in §4(c) is the same fault line that chapter’s §1 covers for vanilla RLVR (pass@1 vs. pass@k, base-model boundary); this chapter’s TRICE result and NuRL’s pass@1024 check are the SFT-side and hint-injection-side versions of that same unresolved question.

Verified citation registry

Confidence calibration: High on every id/abstract above (verified live, not recalled from training memory) and high on the mechanism itself (four independent lineages, 2009–2026, converge on the same generate-under-privilege → strip → supervise structure). Contested, explicitly: whether a full walkthrough specifically (as opposed to an abstract hint or partial reference prefix) is the right hint granularity — NuRL’s own ablation says it isn’t, and no paper surveyed tests the exact executed-in-a-live-sandbox setting end to end, so treat that as the open empirical question, not a settled negative. Several cited ids are very recent (2602.x–2606.x, weeks to months old at verification time) — their mechanisms and diagnostics (hint reliance, teacher flip rate, pass@1024 checks) are more load-bearing for a design decision than their specific benchmark numbers, which have not yet accumulated independent replication. Not grounded on any academic cybersecurity-LLM paper — every citation is general ML/RL data-construction, exploration, or shortcut-learning literature, per this chapter’s brief.