Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Contested edges & landmines

The places where confident-sounding claims are actually unsettled, plus the terminology traps that cause real planning errors. Cited so you can check me.

1. “RL can’t create capability” — contested, not a law

  • Elicit-not-expand (the base claim): RLVR raises pass@1 but the base model beats the RL model at large pass@k — the paths RL finds were already in the base distribution; the reasoning boundary narrows with training. Distillation from a stronger teacher does expand it; RL does not (Yue et al., “Does RL Really Incentivize Reasoning Capacity Beyond the Base Model?”, arXiv:2504.13837, NeurIPS 2025). Reproduced for vanilla GRPO by others (e.g. NuRL notes plain GRPO leaves pass@1024 ≈ base, arXiv:2509.25666).
  • The counter: prolonged RL + KL control + reference resets expands the boundary even on problems the base never solves (ProRL, arXiv:2505.24864); entropy/exploration bonuses and parameter-space noise (arXiv:2602.02555) show similar. Also a metric critique: pass@k over-credits lucky-but-wrong CoT (CoT-Pass@K, arXiv:2506.14245).
  • Safe framing: vanilla RLVR at a normal budget elicits; sufficient compute + explicit exploration-preservation can expand — recipe-dependent, not settled. So “you can’t RL your way out of a missing capability” holds for standard-recipe RL only. Don’t state it as a law.

2. The “RFT” terminology landmine

Two different things share the acronym; conflating them mis-scopes a whole plan:

  • Rejection-sampling Fine-Tuning (STaR-family) = “RL without RL,” positives-only SFT on your own verified samples. Cheap.
  • Reinforcement Fine-Tuning (OpenAI/Fireworks product term — and what the project handbook calls “RFT”) = actual online RL / GRPO against a grader. Expensive.

“Start with RFT before RL” only parses under the first. Say “rejection-sampling SFT” for the cheap thing so you don’t accidentally spec a GRPO run.

3. On-policy distillation is promising, not lab-proven

Earlier framing called it “the sleeper.” Correction after a 2026 verification pass: the efficiency numbers (~9–30× cheaper than RL) come from GKD + Thinking Machines’ own blog (arXiv:2306.13649; thinkingmachines.ai 2025-10-27). No frontier lab has stated it as their production recipe. Real and attractive; treat the numbers as directional, not settled.

4. Don’t over-SFT before RL (2026 lesson)

Meta’s Llama 4 recipe deliberately keeps SFT and DPO lightweight around an intensive online-RL core, with the explicit finding that heavy SFT/DPO restricts RL exploration (ai.meta.com/blog/llama-4-multimodal-intelligence). If RL is your capability driver, a big SFT stage can cap your ceiling — counter to the naive “more SFT is safer.”

5. Reward must be ground-truth-verified, never format-matched

A project-empirical finding: SFT on trajectory data trained the model to emit FLAG{…}-shaped strings on unsolved challenges — confabulation — and a loose regex matcher fired on the model’s own reasoning/tool-args, not real server output (lessons/post-training/sft-induced-flag-confabulation.md, lessons/security-agent/flag-detection-false-positives.md, shared memory). Rule: scan tool output / server state for the flag and verify against ground truth; never reward format. On the gameability ladder, a deterministic verifier (level 1) has no parameters to exploit — stay there; PRM (level 5) was rejected for R1 for exactly this (arXiv:2501.12948).

Tampering caveat — determinism stops gaming, not tampering. A deterministic ground-truth verifier closes off proxy-gaming (exploiting slack in a misspecified reward), but it does not by itself close off reward-tampering — an agent with tool/shell access directly editing, spoofing, or otherwise subverting the verifier or its inputs. Anthropic’s curriculum study found LLM assistants trained on a sequence of easily-discovered specification-gaming environments generalize, zero-shot and non-negligibly often, to rewriting their own reward function or checker (Denison et al., “Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models,” arXiv:2406.10162). Common gotcha: a file/process-based deterministic flag verifier is often reachable from the same shell the agent operates in — so “make the verifier deterministic” is necessary but not sufficient. The verifier must also be isolated from the agent’s action space: verify out-of-band (a separate process/host the agent has no write path to), read env/server state the agent cannot mutate, or run outside the agent’s sandbox entirely. Determinism defeats gaming; isolation is the separate, additional requirement that defeats tampering.

6. “Three knobs” was a teaching scaffold

The on/off-policy axis and the imitation/preference/reward paradigm split are canonical. Packaging them as “N independent knobs you toggle” is a scaffold that over-reached — the axes aren’t independent (signal + policy largely determine what changes), so free combinations produce non-methods. Learn the one axis + the fixed method presets, not a combinatorial grid.

7. Pass@k-as-diagnostic has its own landmines — don’t trust a bare crossover plot [E][R]

Point 1’s crossover test (base pass@large-k beats RL pass@large-k → RL only reweighted, didn’t teach) is the closest thing to a standard instrument in this literature, but it is contested on at least four fronts — the lucky-final-answer confound (CoT-Pass@K), the guessing-vs-reliability conflation (Cover@τ), the vanishing-gradient trap in optimizing pass@k directly (fixed by PKPO), and the task-structure-dependent agentic crossover (Pass@(k,T)) — all worked through in full, with citations, in Diagnosing the gap §2 (“The core instrument: pass@k → Cover@τ → Pass@(k,T)”).

Safe framing: treat the pass@1-vs-pass@k gap as a diagnostic to segment by challenge type, not a single portfolio-wide verdict — never trust an aggregate plot. The CTF-domain analogue of the CoT-Pass@K confound: a verifier-passed trajectory can still contain wasted turns or an ungrounded critical guess before the winning move, so filtering your rejection-sampling SFT set on flag==1 alone reproduces the same lucky-answer-credit problem one level up. A shrinking gap with flat pass@1 is the entropy-collapse warning sign (point 8 below), not “the model learned the task.”

8. Long-horizon credit assignment: turn-level vs trajectory-level vs sequence-level — no consensus on the right granularity [L]

Every flagship reasoning-RL recipe (GRPO, DAPO, Dr.GRPO) assigns one advantage to the whole trajectory — fine for a single-turn math answer, but it starts to matter once episodes run 100 turns with a terminal-only flag reward. The literature has responded with at least three different, non-convergent fixes, each validated on a different (mostly non-CTF, mostly short) domain:

  • Go finer — turn-level advantage. arXiv:2505.11821 shows trajectory-level GRPO applied naively to multi-turn tool-use can fail to teach tool invocation at all (baselines get 20-30% exact-match and never learn to call tools; their turn-level MT-GRPO variant hits 100% tool-execution success). Turn-PPO, arXiv:2512.17008 independently argues PPO-with-a-critic, reformulated so the MDP’s base unit is a turn (not a token), is more robust than GRPO for long-horizon agentic tasks — a direct challenge to defaulting to critic-free GRPO. Both are recent, small-scale (workshop poster / 0-citation preprint, toy benchmarks WebShop/Sokoban) — promising, not settled.
  • Go coarser — sequence-level ratio. GSPO, arXiv:2507.18071 goes the opposite direction: clip and optimize at the whole-response (sequence) level instead of per-token, because token-level importance ratios compound multiplicatively over long sequences and destabilize MoE RL training at scale — credited with letting Qwen3’s RL stage not destabilize. Backed by a shipped frontier model, not a toy benchmark — stronger evidence than Turn-PPO’s, but solving a different problem (numerical stability of the ratio, not credit assignment across turns): the two proposals are not mutually exclusive, but they are not the same fix either.
  • Fix the critic, don’t change the granularity. VAPO, arXiv:2504.05118 keeps trajectory-level PPO but argues the real problem is an unreliable critic at long/heterogeneous horizons — fixed with value-model pretraining + length-adaptive GAE (λ tuned per response length) — reporting zero training crashes across independent runs, directly disputing the “value-based RL is unstable for LLM reasoning” folklore GRPO was invented to route around.
  • Scale the horizon itself, skip the value function entirely. Kimi k1.5, arXiv:2501.12599 treats context/horizon length as a first-class RL scaling axis (not a constraint), using partial rollouts (checkpoint/resume mid-episode) to make 128k-context RL tractable — explicitly avoiding MCTS, value functions, and PRMs. Reframes “the episode is long” as an opportunity, contingent on whether security-agent-<family>/pdq can support partial-rollout checkpointing (unanswered today).
  • Curriculum over the horizon. AgentGym-RL / ScalingInter-RL, arXiv:2509.08755 sidesteps the granularity debate entirely: cap the allowed turn budget low early in training and relax it toward the full target (100 turns) as training proceeds, reporting this prevents the long-horizon collapse that training at full horizon from step one causes.

Designed to fix a common failure: agents commonly prefer raw shell/HTTP over provided higher-level tools, leaving much of the tool surface unused (tool-selection reliability is itself an open problem — cf. arXiv:2505.18135) → turn-level credit assignment / tool-use RL. The turn-level papers’ headline finding (trajectory-level GRPO can fail to teach tool invocation at all) is a plausible root-cause mechanism for tool disuse, not just a scaffolding/prompting issue, if this project ever RL-trains on tool-use.

Confidence: contested by construction — no single paper compares turn-level vs sequence-level vs trajectory-level-with-a-better-critic head-to-head on the same long-horizon agentic benchmark. Most of this cluster is 2025 H2–2026 preprints with 0 citations at verification time, validated on toy environments (WebShop, Sokoban) or math/code, not a 100-turn CTF setting. Treat as “candidate designs to pilot cheaply,” not a default architectural choice — and note turn-level, sequence-level, and critic-repair are not mutually exclusive; a future GRPO/RLVR run here could combine GSPO’s sequence-level clipping (stability) with a turn-level auxiliary tool-invocation reward without contradiction.

9. Do exploration bonuses genuinely EXPAND [N] the boundary, or just elicit what the base model already has? — open question, no paper has run the decisive test on this domain

Point 1 already flags “RL expands vs. only reweights” as contested between Yue et al. and ProRL. The exploration-specific literature (DIVER, CDE, MERCI, PSN-RLVR) all claim their intrinsic-reward/parameter-noise mechanism helps the policy “escape local routines” or “discover better solutions” — language asserting [N] (boundary expansion) rather than mere elicitation. That claim deserves the same skepticism point 1 applies to vanilla RLVR:

  • None of the exploration-bonus papers ran the decisive ablation. The rigorous test for “did this expand the boundary or just elicit/redistribute” is a pass@large-k comparison against the base model (point 1’s own protocol) — DIVER (arXiv:2509.26209), CDE (arXiv:2509.09675), MERCI (arXiv:2510.16614), and PSN-RLVR (arXiv:2602.02555) all compare against vanilla-GRPO/DAPO baselines, not against a very-large-k base-model ceiling. Beating a collapsed-entropy baseline is a much lower bar than beating the base model’s own pass@1024 — and per Spurious Rewards, arXiv:2506.10947, even a completely wrong reward can look like it’s “unlocking” capability on the right base model (Qwen2.5-Math specifically; does not replicate on Llama3/OLMo2) — a stark warning that “the policy now solves things it didn’t before” is not sufficient evidence of genuine novelty without a same-model-family, large-k, base-vs-trained comparison.
  • The capability-elicitation / AI-safety literature already built the falsification protocol for exactly this question — password-locked models (arXiv:2405.19550), harder circuit-broken organisms (The Elicitation Game, arXiv:2502.02180), and the “elicit within <1% of training cost” operational definition of latent capability (AI Sandbagging, arXiv:2406.07358) — all built around known-ground-truth hidden capabilities specifically to distinguish “the technique surfaced something already there” from “the technique taught something new.” No exploration-bonus paper in the RLVR-entropy literature has been tested against a model organism with known injected/withheld capability the way this sub-field requires before it will accept an expansion claim.
  • The one paper that ran something close to the decisive test, on an agentic/compositional task, found genuine expansion — but it’s a single, very recent result. Pass@(k,T), arXiv:2604.14877 (point 7 above) shows RL pulls ahead of base-model pass@k on compositional, sequentially-gated tasks, and — critically — that matched-data SFT on the same tasks regresses the capability boundary (net −4 vs RL’s net +4), isolating self-directed exploration during RL, not exposure to more data, as the causal factor. This is the strongest evidence in the entire corpus that exploration specifically (not RL in general, not more data) is what expands the boundary — but it is one paper (2026-04-16), one research group, unreplicated, studying retrieval-style compositional tasks, not cybersecurity.

Designed to fix a common failure: tool avoidance, and confident guessing that only fails on the hard cases → exploration-bonus RL. DIVER’s pairwise-diversity-of-a-group reward and CDE’s perplexity-based actor bonus are both pitched as countering exactly these behavioral patterns. That framing may be correct as an elicitation mechanism (surfacing tool-diverse or better-calibrated behavior the base model can already produce with a nudge) even if the “expands the boundary” language in the papers’ abstracts is not yet earned.

Confidence: genuinely open. The honest position for this project: exploration bonuses are worth piloting (cheap, mechanistically motivated, all portable per the exploration research thread) — but do not claim any of them “expand the capability boundary” [N] until you’ve run a pass@large-k-vs-base-model check on the same challenge subset (point 1/7’s protocol), ideally segmented by task compositionality the way Pass@(k,T) recommends. Absent that check, the safer verb is “elicit” (base capability present, surfaced more reliably), matching this project’s own competence/performance framing, rather than “expand” (base capability genuinely absent, newly created).