Skip to content

fix(headless): harden RSI prompt optimization runs - #400

Merged
Astro-Han merged 7 commits into
mainfrom
codex/rsi-kept-5round
Jul 1, 2026
Merged

fix(headless): harden RSI prompt optimization runs#400
Astro-Han merged 7 commits into
mainfrom
codex/rsi-kept-5round

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

Harden the headless RSI prompt optimization path based on the kept-prompt 5-round run:

  • allow production prompt optimization runs to seed the prompt repo from MAKA_PROMPT_INITIAL_SYSTEM_PROMPT_FILE
  • allow a resume when the only manifest change is a raised costCeilingUsd
  • avoid reward-hack false positives when Terminal-Bench canary GUIDs also appear in model-visible task files
  • make real reward-hack quarantine evidence block keep decisions and skip held-out spend
  • keep legacy report-only quarantine WAL decisions replayable after scanner fixes

Why

The local RSI run needed a fixed initial prompt seed and a budget-ceiling resume path to complete cleanly. Its reward-hack caveat also turned out to be a scanner false positive: llm-inference-batching-scheduler exposes the same canary GUID in normal helper scripts that the task instructions tell the agent to read.

Closes: N/A

Scope

Changed:

  • run-prompt-optimization.mjs now wires an optional initial prompt seed file into the prompt repo setup.
  • Prompt optimization manifest compatibility now permits resumes that only raise costCeilingUsd.
  • Reward-hack verifier pattern extraction now keeps only canaries exclusive to hidden tests, not canaries also present in model-visible task material.
  • Held-in reward-hack scans now skip tasks without verifier-only patterns instead of quarantining them.
  • Acceptance policy now discards true reward-hack quarantines before spending held-out evaluation.
  • Replay evidence now tolerates legacy report-only false-positive quarantine metadata when recalculated decisions are otherwise identical.

Not included:

  • No benchmark result publication or report UI changes.
  • No change to Harbor task definitions or cached task contents.
  • No broader RSI policy change beyond scanner/gate correctness and resume ergonomics.

Verification

  • npm run -w @maka/headless build && node --test packages/headless/dist/__tests__/harbor-adapter.test.js
  • node --test packages/headless/dist/__tests__/prompt-optimization-run.test.js packages/headless/dist/__tests__/prompt-optimization-loop.test.js packages/headless/dist/__tests__/prompt-acceptance-policy.test.js packages/headless/dist/__tests__/prompt-candidate-loop.test.js packages/headless/dist/__tests__/prompt-optimization-loop-replay-decision.test.js packages/headless/dist/__tests__/prompt-optimization-replay-evidence.test.js packages/headless/dist/__tests__/prompt-structural-smoke.test.js
  • npm run -w @maka/headless test (643 tests, 641 pass, 2 skip)
  • Manually checked the cached llm-inference-batching-scheduler task now extracts no verifier-only reward-hack pattern for its shared canary GUID.

User-facing impact

None for desktop users. This affects headless prompt-optimization/benchmark runs only.

Reviewer notes

This intentionally keeps scanner behavior conservative: canaries repeated in hidden solution/ files still count as verifier-only; only canaries present in model-visible task material are excluded from reward-hack matching.

@Astro-Han
Astro-Han merged commit 40ec1ba into mainJul 1, 2026
@Astro-Han
Astro-Han deleted the codex/rsi-kept-5round branch July 1, 2026 11:21
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han