feat(headless): validate tool result pruning with A/B benchmark

[Astro-Han]

Summary

We should run a first-phase non-inferiority A/B benchmark for archive-backed active tool-result pruning.

The question is not whether prune is mathematically lossless. The question is whether enabling prune causes any unacceptable official benchmark pass-rate regression under the same Maka runtime, prompt, model, tools, task set, timeout, and execution budget.

Primary comparison:

• A: Maka with tool result prune off
• B: Maka with archive-backed active tool-result prune on, plus archive retrieval enabled

For this phase, prune on means archive-backed active tool-result pruning in the Harbor / Terminal-Bench path. Stale prior-context pruning is not part of the phase-1 conclusion.

The smoke run exposed a validity blocker in the previous framing: the current single-turn Harbor cell can hit the runtime tool-call step cap (maxSteps=50) and end as incomplete_tool_calls before the task has a normal chance to finish. A formal run dominated by step-cap truncation cannot answer whether active prune regresses official verifier pass rate. It mostly measures the harness cap.

Therefore, phase 1 needs a benchmark-safe continuation path before formal evidence, unless an explicit shared step budget is proven sufficient. The preferred path is deterministic continuation: split the same task execution across multiple neutral runtime turns, keep the same task workspace, do not run or reveal verifier feedback between turns, and run the official verifier only once at the end.

OpenCode can be used as external context, but it is not part of the primary A/B. OpenCode differs in agent loop, prompt, tools, architecture, and product behavior, so it cannot prove whether Maka's prune switch is non-inferior.

Current status

PR #323 landed the phase-1 active-prune reporting foundation:

• Harbor cell output now carries active context-budget diagnostics.
• Harbor context-budget policy snapshots include activeToolResultPrune.
• Runtime-policy A/B reports now summarize active-prune diagnostics and existing archive retrieval/failure diagnostics.
• Activated subset reporting is now a B-active paired slice: B attempts where activePrunedToolResults > 0, plus the matching A (taskId, rep) attempts.
• Active subset markdown includes task count, attempts, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Stale-only prunedToolResults no longer counts as active activation.

PR #323 did not add archive reason-count reporting, did not add benchmark-safe continuation, did not run the formal non-inferiority benchmark, and does not close this issue.

Active-prune smoke already ran:

• Run root: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

Smoke interpretation: this is reporting and activation evidence only. Both B attempts ended as incomplete_tool_calls after hitting the single-turn 50 tool-call step cap, so it is not non-inferiority evidence. The result means active prune diagnostics/reporting worked, and it shows the formal benchmark path needs a cap-safe continuation strategy before it can answer the pass-rate question.

Goal

Validate that enabling archive-backed active tool-result prune does not cause a large official verifier pass-rate regression, while collecting evidence about token and cost savings.

First-phase quality bar:

• Use official benchmark verifier pass rate as the primary metric.
• Use a 10 percentage-point non-inferiority margin for the first phase.
• Treat token/cost reduction and trace evidence as benefit signals, not as the hard pass/fail gate.
• Treat cap-dominated incomplete_tool_calls as a validity failure, not as evidence for or against prune.

This means we are trying to rule out a large regression first. A stricter 5 percentage-point claim would require a much larger sample and should be a follow-up experiment if the first phase is clean.

Continuation requirements

The formal phase-1 run may use multi-turn continuation only if it is benchmark-safe and does not add verifier or coaching confounds.

Required properties:

• Keep the same task workspace across continuation turns.
• Continue only after a runtime/tool-call step cap or a declared neutral continuation condition.
• Use a fixed neutral continuation prompt, for example: Continue the same benchmark task from the current workspace state. Do not restart. If the task is complete, provide the final response.
• Do not run the official verifier between continuation turns.
• Do not reveal verifier results, scorer taxonomy, hidden test output, or pass/fail feedback to the model between turns.
• Do not use self-check coaching, autonomous retry feedback, or verifier-feedback repair prompts.
• Run the official verifier only once, after the final continuation turn or budget exhaustion.
• Apply the same continuation policy, max turns, max total runtime steps, wall timeout, model, tools, prompt, and task set to A and B.
• Record continuation diagnostics in the report: turns used, total runtime steps, per-turn step-cap hits, max turns, max total steps, wall timeout, and cap-exhausted attempts.

The existing autonomous loop architecture can be reused for budgeting, task-run events, and projection ideas, but the current default autonomous retry semantics cannot be used directly for phase-1 evidence because it runs verifier after each attempt and feeds retry-oriented feedback back into the model.

A benchmark-only explicit maxSteps override may still be useful as a secondary control, but it must be applied equally to A and B, recorded in the report, and not tuned after seeing formal A/B outcomes. It should not replace continuation if normal task trajectories still saturate the cap.

Statistical framing

Use a non-inferiority test, not a generic "difference/no difference" test.

• H0: prune on is worse than prune off by more than 10 percentage points in official pass rate.
• H1: prune on is not worse than prune off by more than 10 percentage points in official pass rate.

Do not interpret "no statistically significant difference" as proof that prune is safe. The test should directly ask whether B is worse than A by more than the agreed margin.

pp means percentage points. For example, if prune off passes 40% and prune on passes 35%, the delta is -5pp.

Task set discussion

The exact first formal slice is still open, but expert-time slicing alone is not enough to solve the step-cap problem. The selected slice must run under the same continuation policy and must report continuation/cap diagnostics.

The current local cached Terminal-Bench task pool has 89 tasks by expert_time_estimate_min:

Slice	Task count
<= 30 min	34
<= 45 min	40
<= 60 min	53
<= 90 min	55
> 90 min	33
Missing estimate	1

The previous prompt A/B formal run used the <= 30 min slice with 34 tasks and 3 reps.

For this active-prune A/B, decide whether the first formal run uses <= 30, <= 45, or <= 60, but do not rely on a shorter slice as the primary fix for incomplete_tool_calls. The formal run must either complete under the declared continuation policy or explicitly report that the evidence is blocked by cap saturation.

Candidate	Attempts per arm at 3 reps	Tradeoff
<= 30 min	102	Lowest cost, close to the existing prompt A/B setup, but weaker for a 10pp non-inferiority claim.
<= 45 min	120	Better fit for first-phase 10pp non-inferiority while still bounded.
<= 60 min	159	Stronger first-phase signal, higher cost, still not enough for a 5pp claim if observed rates are roughly tied.

A rough baseline pass rate around 32% implies that proving 5pp non-inferiority when observed rates are roughly tied would require hundreds of attempts per arm. Even <= 60 min × 3 reps is still a first-phase 10pp experiment, not a 5pp "lossless" proof.

Report requirements

Minimum report contents:

• Overall A/B summary: task count, reps, pass rate, pass-rate delta, non-inferiority decision, budget timeout counts, infra failure counts, plumbing failure counts.
• Activated subset: B-active paired slice where activePrunedToolResults > 0, plus matching A (taskId, rep) attempts.
• Active-prune diagnostics: activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures.
• Existing archive diagnostics where present: archivePlaceholders, archiveWriteFailures, retrievedArchiveToolResults, retrievedArchiveEstimatedTokens, archiveRetrievalSkipped, and archiveRetrievalFailures.
• Archive reason counts where present: archivePlaceholderReasonCounts, archiveRetrievalSkippedReasonCounts, and archiveRetrievalFailureReasonCounts.
• Token/cost summary: input tokens, cache hit/miss input, output tokens, total cost, and duration by arm.
• Active subset token/cost summary: full token/cost plus attempts, observed, missing, and coverage by arm.
• Trace references: include runtimeEventsPath and traceEventsPath for activated attempts, B losses, budget-discordant pairs, and infra/plumbing-discordant pairs where available.
• Continuation/cap diagnostics: turns used, total runtime steps, per-turn step-cap hits, max turns, max total runtime steps, wall timeout, and cap-exhausted attempts.
• Formal run report: publish a redacted issue comment or artifact summary after the first formal A/B run.

Artifact policy: keep raw traces local by default. Share a redacted summary in the issue or follow-up comment. Do not paste full raw traces into GitHub unless they have been reviewed for local paths, secrets, and irrelevant private data.

The activated subset is important because a full task set may include tasks that never trigger prune. A/B equality on tasks that never prune does not tell us whether pruning itself is safe.

Implementation checklist

Completed:

• Add Harbor cell support for runtime policy wiring so benchmark runs can pass contextBudget into AiSdkBackend.
• Add archive persistence hooks in the Harbor cell path: archiveToolResult and readToolResultArchive.
• Add a runtime-policy A/B runner/config layer on top of the existing generic A/B core, without reusing prompt A/B's prompt-path assumptions.
• Extend cell output and report extraction so active context-budget diagnostics can be summarized per attempt.
• Extend A/B markdown/JSON reports with active-prune diagnostics, B-active paired subset summaries, token/cost summaries, and trace references.
• Run a small wiring/activation smoke to verify that active prune triggers and diagnostics appear.

Remaining for this issue:

• Add minimal archive reason-count schema/reporting if the runtime diagnostic fields are present in events.
• Add benchmark-safe deterministic continuation for the Harbor / Terminal-Bench path, with the same workspace and neutral continuation prompt before final verification.
• Extend reports with continuation/cap diagnostics.
• Decide the formal task slice (<= 30, <= 45, <= 60, or another declared slice) after the continuation/cap strategy is fixed.
• Run the formal first-phase A/B.
• Publish the redacted formal report and explicitly scope the conclusion to archive-backed active tool-result pruning under the declared continuation/cap policy.

Follow-up, not phase 1:

• Design a benchmark-safe stale prior-context pruning experiment with replay or continuation that does not claim stale-prune non-inferiority from active-prune results.

Non-goals

• Correct the active current-turn tool-result prune default threshold to 2048 in this issue, so active pruning matches the measured threshold and the existing stale tool-result prune default. Archive retrieval keeps its separate 8192-token retrieval budget and is not the active-prune threshold.
• Do not use OpenCode as the primary A/B baseline.
• Do not compare active-prune-only, stale-prune-only, eager retrieval, and history-search-gated retrieval in the first primary experiment.
• Do not validate stale prior-context pruning in phase 1.
• Do not use verifier feedback, self-check coaching, or autonomous retry feedback for phase-1 benchmark evidence.
• Do not use the current default autonomous retry loop directly for phase-1 evidence, because it runs verifier after each attempt and feeds retry-oriented feedback back into the model.
• Do not claim stale-prune non-inferiority from the phase-1 active-prune result.
• Do not claim active-prune non-inferiority from a cap-dominated run.
• Do not claim absolute losslessness.
• Do not claim 5pp non-inferiority from a first-phase run that is not powered for it.

Open questions

• Should a benchmark-only maxSteps override be added as a secondary control, and if so what fixed value should be declared before formal runs?
• Should the next formal slice be <= 30, <= 45, <= 60, or another declared slice under the fixed continuation policy?
• What exact redaction rules should apply before sharing trace snippets?
• Should stale prior-context pruning get a separate follow-up issue with a benchmark-safe replay design?
• If the first phase passes 10pp non-inferiority, do we want a second-phase 5pp experiment with a larger sample?

[Astro-Han]

Update proposal for phase 1:

The current formal Harbor/Terminal-Bench path is single-turn: one sendMessage can contain many model/tool steps, but it does not create prior runtime context. The smoke runs confirmed that MAKA_CONTEXT_STALE_TOOL_RESULT_PRUNE=on is wired, and that current-turn tool results can exceed the prune threshold, but stale pruning does not activate because stale pruning only applies to prior turns / prior attempts.

To keep the first formal A/B clean, phase 1 should validate archive-backed active tool-result pruning instead of adding a multi-turn/retry harness to force stale pruning.

Revised phase-1 comparison:

• A: Maka prune off (MAKA_CONTEXT_BUDGET=off)
• B: Maka active tool-result prune on (MAKA_CONTEXT_ACTIVE_TOOL_RESULT_PRUNE=on) with archive-backed tool result storage and archive retrieval enabled

The phase-1 conclusion must be scoped to active tool-result pruning in the current single-turn Harbor path. Stale prior-context pruning should move to a follow-up experiment with a benchmark-safe replay design. We should not claim stale-prune non-inferiority from the phase-1 active-prune result.

Report updates:

• Activated subset should include attempts where activePrunedToolResults > 0.
• Diagnostics should include activePrunedToolResults, activeEstimatedTokensSaved, activeArchiveFailures, plus archive retrieval/failure counts where present.
• Existing pass-rate, 10pp non-inferiority, Newcombe/Wilson lower bound, B losses, discordant pairs, token/cost summary, and trace refs still apply.