RFC: Harness RSI loop — autonomous system prompt optimization

[Astro-Han]

RFC: Harness RSI loop — autoresearch structure for system prompt optimization

Context

maka's eval framework is ready (PR #62/#63). The goal is closed-loop harness self-improvement: an agent autonomously optimizes its own system prompt against a benchmark task set, no human in the loop, runnable until stopped.

Prior art: Karpathy's autoresearch (minimal loop, single agent), HarnessX/AEGIS (heavy 4-stage engine, +14.5%), Self-Harness (regression-gated edit loop).

Architecture (verified)

RuntimeRunner in-container + Harbor verifier. Harbor provides the Docker container (from TB task Dockerfile). Inside that container, a run-cell.mjs entrypoint runs maka's RuntimeRunner directly — the agent works in-place at /app (the Dockerfile WORKDIR), using file tools + Bash all in-container (one filesystem). After the agent finishes, Harbor runs tests/test.sh which checks /app state.

Controller (host):
 1. For each held-in task: harbor run --path <tb-task> --agent-import-path maka_agent:MakaAgent
 --ae MAKA_SYSTEM_PROMPT=<prompt> --ae DEEPSEEK_API_KEY=<key>
 2. Parse Harbor result.json → per-task reward (0/1)
 3. Parse adapter's errorClass output → per-task errorClass

Harbor container (per task):
 1. Build from task Dockerfile (WORKDIR /app, has tools, test env)
 2. MakaAgent.run() → adapter starts run-cell.mjs in /app
 → RuntimeRunner runs agent (file tools + Bash, all in /app, one filesystem)
 → RuntimeRunner records trajectory to runtime-events.jsonl
 → adapter extracts errorClass from InvocationResult
 3. Harbor runs tests/test.sh → checks /app state → reward 0 or 1

Why not runExperiment? Verified: runExperiment copies fixture to mkdtemp('/tmp/maka-headless-ws-XXXX'), runs verifier there, deletes it. But TB test scripts write absolute paths to /app (e.g. Path("/app/regex.txt") in test_outputs.py). 89/89 Dockerfiles use WORKDIR /app. runExperiment's temp workspace would never be checked by TB tests. RuntimeRunner works in-place at /app — no copy, no sync problem.

Why not Harbor verifier only? We need maka's errorClass (max_tokens / verification_failed / runtime_error) for signal enrichment. The adapter extracts this from RuntimeRunner's InvocationResult and writes it alongside Harbor's reward.

File/bash filesystem split (v5 P0-2, resolved): the entire run-cell.mjs process runs in-container. RuntimeRunner's file tools (Write/Edit/Read) and Bash tool (toolExecutor) all operate on the container's filesystem. No host-container split.

Prompt delivery: controller passes system_prompt.md content via MAKA_SYSTEM_PROMPT env var. run-cell.mjs reads it, passes to Config.systemPrompt. Runtime stamps a hash of the effective Config.systemPrompt string (at backend constructor, after resolveSystemPrompt) into trajectory events. Controller verifies the hash round-trips — if trajectory hash ≠ hash of prompt sent, the round is invalid (plumbing failure).

Implementation prerequisite: systemPromptHash does not exist in runtime yet (verified: only requestShapeHash exists). Must add a small instrumentation: hash the resolved Config.systemPrompt and write to a runtime event field. Pure addition, no existing code changes.

Harbor/container failure ≠ benchmark failure. Container crash, image build failure, Harbor timeout → no result.json or invalid output. Controller treats this as infra error. Harbor reward + adapter errorClass only decide benchmark score.

Signal quality

Fix 1 — Coverage-aware gate. max_tokens/runtime_error tasks should count as fail (not drop from denominator). Gate uses pass / eligible (unscored = fail), NOT pass / scored.

keep = held_in_pass_rate (pass/eligible) improved beyond noise_band_held_in
 AND held_in_coverage (scored/eligible) not degraded
 AND held_out_pass_rate non-inferior vs original baseline (>= -noise_band_held_out)
 AND held_out_coverage not degraded
 AND no infra_errors (any infra error → no keep)

Controller computes pass/eligible itself. Harbor gives per-task reward (0/1); adapter gives per-task errorClass; controller combines: pass = reward==1, eligible = true (all tasks eligible unless infra error), scored = errorClass != 'max_tokens' && errorClass != 'runtime_error'.

Fix 2 — Dual noise band. noise_band = wilson_half_width(N, p_mean) × √(1 + 1/n_baseline) — CI of the difference, not single-proportion CI. Separate bands for held-in and held-out.

• Layer 0 (free): dual noise thresholds.
• Layer 1 (cheap): 60 held-in + 20 held-out from TB 2.0's 89 tasks.
• Layer 2 (deferred for v1): repeat sampling. v1 goal is structural validation.

Fix 3 — Agent reads failing trajectories. Controller extracts per-task digest from runtime-events.jsonl (raw events): last 2 tool calls (from function_call events with args), errorClass, steps/duration, verifier stdout/stderr first 3 lines.

Overfitting guard

Held-in vs current kept baseline (monotonic improvement). After KEEP: held_in_reference = max(previous_reference, new_run_pass_rate − noise_band_held_in). Banks real improvements without enshrining lucky peak, never falls below prior state.

Held-out vs original baseline (fixed floor). Floor = original_held_out_mean − noise_band_held_out. Prevents cumulative drift.

Held-out physical isolation. Held-out task content, trajectories, results.jsonl live outside agent cwd. Agent sees only program.md, system_prompt.md, results.tsv, held-in digests.

Reward-hack quarantine gate. Controller scans function_call.args in raw runtime-events.jsonl for verifier-specific patterns (expected-output strings, not filenames). Flagged → QUARANTINE (discard). Scanner fails closed if raw events missing.

Infra error handling

Any infra error → no KEEP. Infra = missing result.json, container crash, image build failure, Harbor timeout. Retry once, else discard+log.

>20% infra → stop. Stop condition, not keep-allowance.

Infra classification: model API failure (DeepSeek 429/500) → runtime_error (scored=false, counts in eligible). Container/Docker/Harbor failure → infra (excluded from scoring).

Budget

Controller sums costUsd from runtime-events.jsonl token-usage events. costUsd exists on RuntimeEvent. BUILTIN_PRICING keyed as deepseek:deepseek-chat (provider:model format) — controller must compose key as ${provider}:${config.model}.

Zero-cost guard: if summed costUsd = 0 but tokens > 0 → plumbing failure (pricing not wired). Do NOT count as free round.

Cost ceiling: $30 (default). 80 tasks × single-sample × ~10 rounds ≈ $15-25.

Controller / agent separation

Controller (host-side Node.js): Harbor invocation, result parsing, metric computation, budget, git state machine, keep/discard, crash/resume WAL (append jsonl LAST as source of truth; reconcile last_kept_commit from jsonl on resume). Diff guard AFTER agent edit, BEFORE commit — only system_prompt.md (tracked) may change; controller files gitignored.

Agent (meta-agent): scripted LLM call (deepseek-chat). Reads results.tsv + digests → outputs prompt edit + description. Log IS its memory. No interactive tools.

Implementation entrypoint

Not maka-headless eval CLI (only wires fake backend). run-cell.mjs in container directly imports RuntimeRunner from @maka/runtime:

import { RuntimeRunner, AiSdkBackend, ... } from '@maka/runtime';
// agent runs in /app (Dockerfile WORKDIR), in-place
// RuntimeRunner → InvocationResult → extract errorClass
// write errorClass + trajectory to /output/

maka-agent repo cloned + built at /opt/maka-agent (outside /app).

Verified facts

Question	Answer	Source
TB test.sh writes /app?	Yes — test_outputs.py uses Path("/app/regex.txt") etc.	grep 89 tasks
runExperiment compatible?	No — prepareWorkspace uses mkdtemp('/tmp/...'), can't be /app	sandbox.ts:20
systemPromptHash exists?	No — only requestShapeHash	grep runtime/src
pricing key format?	provider:model (deepseek:deepseek-chat)	builtin-pricing.ts:14
Harbor + Docker verified?	Yes — oracle reward 1.0, maka adapter runs, verifier works	smoke test 6/19
file/bash split resolved?	Yes — whole process in-container	architecture decision

Directory layout

~/.local/maka-eval/
├── harbor-adapter/
│ ├── maka_agent.py # Harbor agent adapter
│ └── run-cell.mjs # in-container entrypoint (RuntimeRunner)
├── harness-rsi/
│ ├── program.md # loop instructions (agent reads)
│ ├── system_prompt.md # optimization target (agent edits ONLY)
│ ├── controller.mjs # state machine (host-side)
│ ├── results.jsonl # canonical truth (OUTSIDE agent cwd)
│ ├── results.tsv # derived view for agent (IN agent cwd, gitignored)
│ └── baseline/ # 3× baseline calibration
├── ds-key.txt
└── fixtures/

The loop

INIT:
 1. Run baseline 3× on held-in + held-out
 2. Compute means + observed_spread
 3. noise_band = max(wilson_half_width(N, p) × √(4/3), observed_spread)
 4. held_in_reference = baseline_held_in_mean
 5. Save original_held_out_mean (never updated)

LOOP:
 1. Write pending round record
 2. Agent: read results.tsv + digests → edit system_prompt.md
 3. Controller: diff guard (after edit, before commit) — only system_prompt.md
 4. Controller: git commit
 5. Controller: run held-in via Harbor → parse reward + errorClass per task
 6. Check infra — any infra → no keep, retry once, else discard
 7. Compute held_in_delta vs held_in_reference
 8. |delta| < noise_band → DISCARD (inconclusive)
 9. delta <= 0 → DISCARD (regression)
 10. delta > noise_band:
 a. Run held-out via Harbor
 b. Check held-out infra
 c. held_out_delta vs original_held_out_mean
 d. < -noise_band → DISCARD (regression)
 e. coverage degraded → DISCARD
 f. reward-hack scan → QUARANTINE if flagged
 g. >= -noise_band AND coverage OK → KEEP
 h. held_in_reference = max(held_in_reference, pass_rate − noise_band)
 11. Append to results.jsonl (source of truth)
 12. Advance last_kept_commit (if KEEP, AFTER jsonl append)
 13. Update results.tsv
 14. Check budget (sum costUsd, zero-cost guard)

Success criteria (structural validation)

• Loop runs unattended ≥10 iterations without human intervention
• Zero keeps is passing — v1 validates structure, not prompt quality
• If keeps occur: no held-out regression, no unreviewed reward-hack flags
• Cost ≤ $30
• Prompt round-trip hash verified every round

Open questions

1. Task selection: which 60 of 89 for held-in? Filter by speed/stability.
2. Parallelism: --n-concurrent — determinism? Key results by task id.
3. Simplicity criterion: deletion + non-inferior = KEEP? Needs own branch.
4. Pricing wiring: verify in-container path registers pricing lookup.

Non-goals (v1)

• No tool description optimization
• No control flow changes
• No AEGIS 4-stage pipeline
• No repeat sampling / pass@k
• No runExperiment (RuntimeRunner + Harbor verifier instead)

[Astro-Han]

Approved by @likun666661 . Will start implement.

[Astro-Han]

Implementation split proposal

I propose implementing this RFC as six flat PRs, split by concern and verification boundary rather than line count. Each PR should be independently reviewable and end-to-end verifiable on its own.

PR1: Runtime cell facts

Add the runtime facts the harness must be able to trust:

• first-class systemPromptHash for the effective resolved system prompt
• costUsd on runtime token-usage / trajectory events, using the existing pricing lookup
• tests proving the prompt hash and cost survive the RuntimeRunner / AiSdkBackend path

This PR should also pin the prompt-hash contract explicitly: whether the controller verifies the input prompt hash, the resolved prompt hash, or both.

Non-goals: Harbor adapter, host controller, prompt editing, acceptance policy.

PR2: Harbor cell contract

Add the single-cell execution surface:

• run-cell.mjs inside the task container
• Harbor maka_agent.py adapter
• one shared typed cell-output schema for status, errorClass, runtime-events path, prompt hash, token/cost summary, steps, and duration

The controller should later import this schema rather than re-declare it.

Non-goals: controller loop, prompt editing, KEEP/DISCARD policy.

PR3: Fixed-prompt controller + WAL

Add the host-side controller for a fixed system_prompt.md:

• invoke Harbor for a configured task list
• parse Harbor result plus adapter output
• classify infra errors separately from benchmark failures
• aggregate cost and enforce zero-cost-with-tokens as a plumbing failure
• write append-only results.jsonl
• derive results.tsv
• support resume/reconcile from WAL

Important: results.jsonl should start as a discriminated append-only event log, not a flat result table, so later KEEP/DISCARD and commit records can be added without migration.

Non-goals: meta-agent prompt edits, acceptance policy.

PR4: Meta-agent containment + candidate loop

Add the prompt-editing loop:

• program.md
• system_prompt.md
• scripted meta-agent call
• held-in trajectory digest extraction
• candidate prompt commit with round id
• strict diff guard: only system_prompt.md may change

This PR should include both write containment and input/read containment: the meta-agent should receive only held-in digests / derived results, never held-out task content or held-out trajectories.

Non-goals: statistical KEEP/DISCARD policy.

PR5: Acceptance policy + baselines

Add the acceptance policy:

• baseline calibration
• held-in / held-out partitions
• dual noise band math
• pass / eligible scoring
• coverage non-degradation
• held-in monotonic reference
• held-out original floor
• KEEP/DISCARD transitions
• last_kept_commit promotion after WAL append

Verification should be mostly fixture-driven with synthetic result records covering improvement, flat/noise, regression, coverage degradation, infra failures, and resume after KEEP.

Non-goals: reward-hack quarantine, concurrency, large unattended run.

PR6: V1 safety and scale hardening

Add the v1 hardening layer:

• reward-hack quarantine scan over raw runtime function_call.args
• physical held-out isolation checks
• task stability / selection filter
• deterministic concurrency keyed by task id, if needed
• >20% infra stop
• budget ceiling enforcement
• unattended 10-iteration structural smoke/report

V1 success is structural: zero keeps is acceptable if the loop runs unattended, respects the budget, verifies prompt plumbing every round, and does not accept held-out regressions when keeps occur.

Notes

This deliberately does not route Terminal-Bench through runExperiment: the current runner copies fixtures into a temporary workspace, while TB tasks verify /app in-container. The harness cell path should use RuntimeRunner in-place inside the Harbor/TB container.

[Astro-Han]

PR6 merged: V1 safety and scale hardening

PR6 is merged: #70 (4236af8e). The original six-PR split is now complete at the code layer:

PR	Scope	Status
#65	runtime cell facts	merged
#66	Harbor cell contract	merged
#67	fixed-prompt WAL controller	merged
#68	meta-agent candidate loop	merged
#69	acceptance policy + baselines	merged
#70	V1 safety / scale hardening	merged

PR6 added:

• reward-hack quarantine scanner over model-visible runtime events
• physical artifact isolation for controller-only results and held-out artifacts
• stable task selection / baseline evidence guards
• infra-rate and cost-ceiling stop guards
• deterministic guarded concurrency
• structural smoke report binding commit, task evidence, decision order, prompt hash, candidate hash, and scan evidence
• CLI git containment for dirty baseline drift, side edits, deleted side files, and HEAD drift

Verification before merge:

• npm run -w @maka/headless test — 197 tests pass
• npm run -w @maka/headless typecheck
• npm run -w @maka/headless build
• git diff --check origin/main...HEAD
• final fresh-eye review: PASS, no P0/P1, no worthwhile pre-merge P2/P3
• final Claude review: PASS, no P0/P1

One integration note for the real run: the hardening primitives are now in place and fail closed when scan evidence is missing, but the final unattended API-key eval should still verify that the real orchestrator path passes genuine rewardHackScan results into acceptance and structural smoke, with verifier patterns sourced from the task/verifier data.

Suggested next validation, after supplying API keys: run the 10-iteration structural eval once on merged main, with the budget ceiling enabled, then inspect the smoke report for:

• no prompt plumbing/hash failures
• no missing reward-hack scan evidence
• no held-out artifact visibility
• no HEAD/diff guard violations
• cost within ceiling
• zero keeps is acceptable for V1 structural validation

[Astro-Han]

Dry structural smoke evidence

After PR #70 merged into main, I refreshed local dependencies and ran the no-API dry structural smoke check on merged main:

• npm install — installed the newly declared overlayscrollbars dependency needed by current main
• npm run build — passed for all workspaces
• node --test --test-name-pattern "passes after ten unattended discard decisions under budget" packages/headless/dist/__tests__/prompt-structural-smoke.test.js — passed

That dry smoke fixture covers the V1 success shape: 10 unattended rounds, all discard decisions, clean reward-hack scan evidence, task evidence present after candidate commits, total cost under ceiling, and zero keeps still passing structural validation.

[Astro-Han]

Enhancement: feed an aggregated run-trace failure signal into the meta-agent input

Today the loop tunes the system prompt mostly from scalar outcomes — pass rate, reward, cost (prompt-optimization-loop.ts). The per-task TrajectoryDigest the meta-agent receives (prompt-candidate-loop.ts) already carries some causal context: errorClass plus the last two tool calls (recentToolCalls, .slice(-2)). What it lacks is a round-level aggregate of why tasks failed — above all, recurring tool failures.

Verified — the capture already exists; the gap is only in what the digest reads:

• Each trial persists two ledgers. runtime-events.jsonl is the message stream (author/content/role/turnId); events.jsonl is the trace stream ({type, data, ...}) and already contains tool_failed, turn_started, etc. recordRunTrace is wired end-to-end (harbor-cell.ts:319, task-agent-controller.ts:740), and inspectAgentRunReadModel (agent-run-inspect.ts:62) already folds the trace ledger into a structured summary.
• But extractTrajectoryDigest reads runtimeEventsPath → the message stream, so tool_failed / abort / permission / budget signals never reach the meta-agent today.

So this needs no new capture — just point the digest at the trace ledger (or reuse inspectAgentRunReadModel) and aggregate.

Proposal — phase it:

• Phase 1 (do first): aggregate tool_failed across the round — which tool failed, how many times, the top arg/error shapes — deterministic (sorted, top-N capped, mirroring the existing .slice(-2)), threaded into the meta-agent prompt as a round-level block. This is the most directly actionable signal for a rewrite ("the agent kept failing X with Y → tell it Z").
• Phase 2 (defer until phase 1 proves out): token / context-budget waste patterns and permission / abort hotspots. Noisier and less directly actionable at the prompt layer.

Validate, don't assume: the meta-agent here is a cheap model (deepseek-v4-flash); a richer prompt can dilute as easily as it sharpens. A/B it — run with and without the trace signal and compare KEEP/DISCARD rates and the held-out floor via the loop's existing acceptance machinery.

Out of scope: per-run in-loop self-check. The SelfCheckPolicy hook in packages/headless/src/autonomous-agent-loop.ts is still dormant (no caller installs a policy), so the advisory-feedback path doesn't run today. Revisit only if the prompt-layer signal proves useful.

[Astro-Han]

v1 done + follow-up plan

The v1 RSI loop is implemented and validated end-to-end — PR #149. Full unattended run (59 held-in / 20 held-out, 3 baseline sweeps, 10 rounds, real DeepSeek v4-flash + Docker, ~20.5h):

status: pass | rounds 10/10 | decisions keep=0 discard=10
task_events: completed=475 infra_failed=32 plumbing_failed=0
rounds_without_task_evidence=0 | reward_hack_quarantine=0 | cost_usd 4.76 / 30

keep=0 is the intended v1 pass criterion (structural validation): the whole pipeline runs end-to-end — generate → commit → evaluate held-in/held-out → reward-hack canary scan → acceptance policy → discard/reset → guards → report — with zero plumbing failures and zero reward-hack leaks.

Root cause of the all-discard outcome (useful signal, not a bug): 9 of 10 rounds were coverage_regressed. A blind prompt edit occasionally makes one held-in task (recurringly cobol-modernization, also large-scale-text-editing, extract-elf) stop producing a scorable artifact, so coverage drops below the reference (1.0 → 0.947) and the policy hard-vetoes the candidate — even in rounds 1–3 where the candidate's pass rate on everything it did score was actually higher (0.842 vs 0.789). The 10th round was held_in_within_noise. Measured held-in pass-rate noise band ≈ 0.185 (wide).

Follow-ups (ordered)

1. Feed an aggregated tool_failed trace signal into the meta-agent input. This is the dominant discard cause and the highest-leverage fix — the meta-agent edits blindly today and can't see which task its edit broke. Verified plan (capture already exists in events.jsonl; phase 1 = tool_failed aggregate) is in the enhancement comment above (#issuecomment-4770124004). Do this first.

2. A/B harness — compare a hand-authored new baseline prompt (B) vs the current seed (A). Design:

• Reuse the existing acceptance policy: run as a 1-round loop with A = lastKept, B = candidate → a KEEP/DISCARD verdict with the same rigor as the loop. No new scoring code.
• Metrics: held-in pass rate, held-out pass rate, coverage, cost; record paired per-task win/loss.
• Decision rule: B wins iff Δpass-rate > noise band and coverage does not regress and the held-out floor holds.
• Power constraint (size this before authoring B): held-in noise band ≈ 0.185 is wide → either B must beat A by a large margin, or increase reps / task count to tighten the band, otherwise the A/B can't detect a modest real improvement.

3. Author the new baseline prompt — replace/strengthen the current 5-rule seed (introduced in #62), validated via the A/B above.

Suggested order: merge #149 → (1) trace signal → (2) A/B harness → (3) new baseline.