feat(headless): validate active-prune 2048

[Astro-Han]

Summary

• Set Harbor active tool-result pruning to the measured 2048-token threshold and surface prune diagnostics in cell output and A/B reports.
• Add deterministic Harbor continuation plumbing for tool-step-cap recovery, including continuation failure propagation and max total runtime step accounting.
• Fix provider replay for interleaved parallel tool calls and run A/B arms for the same task pair concurrently.

Why

Closes#293

The active-prune policy needs both a validated 2048-token threshold and benchmark evidence that it saves cost without a large regression detected under the 10pp phase-1 framing. Issue #293 frames this as a first-phase non-inferiority experiment with a 10 percentage-point margin, not as a proof of mathematical losslessness or a 5pp claim.

Evidence

Bottom line: archive-backed active tool-result prune at the 2048-token threshold did not show a regression larger than the declared 10pp non-inferiority margin, while saving 31.6% cost and 41.7% tokens across the combined evidence set. The margin clearance is narrow: the stratified 95% lower bound is -9.78pp against a -10pp margin, clearing by 0.22pp. This supports “no large regression detected” under the phase-1 framing, not proof of equivalence or absolute losslessness.

Design: non-inferiority test where A is prune off and B is active prune on with archive retrieval enabled. The evidence is reported as two strata because the older full runs predate continuation rescue and A/B pair concurrency, while the new rescue run uses the fixed harness.

Stratum	Pairs	A pass	B pass	Delta (B-A)	95% lower bound
Old full runs, 4 slices	89	38	42	+4.49pp	-9.94pp
Continuation rescue	32	10	9	-3.13pp	wide, inconclusive alone
Stratified rollup	121	48	51	+2.48pp	-9.78pp

Combined cost was $2.1600 baseline vs $1.4781 candidate, saving $0.6819 / 31.6%. Combined tokens were 192.0M vs 111.9M, saving 80.1M / 41.7%.

The new 32-task continuation rescue run is inconclusive by itself. Its value is that it exercises the replay fix, benchmark-safe continuation, and A/B pair concurrency on the tasks that previously hit the tool-step cap. The rescue run logs had no APICallError, insufficient tool messages, invalid_request_error, or runtime_error.

Per-run breakdown and local artifacts

Run	Slice	A passed	B passed	Delta	Cost A -> B	Tokens A -> B	Notes
issue293-active-prune-default-le30-005	old full, <=30m	21/34	20/34	-2.94pp	$0.2867 -> $0.1917	21.91M -> 12.49M	2048 snapshot, no continuation
issue293-active-prune-default-30to45-001	old full, 30-45m	3/6	3/6	0pp	$0.0938 -> $0.1067	6.22M -> 7.34M	2048 snapshot, no continuation
issue293-active-prune-default-45to60-001	old full, 45-60m	5/13	5/13	0pp	$0.2429 -> $0.2168	19.69M -> 16.96M	2048 snapshot, no continuation
issue293-active-prune-default-gt60missing-001	old full, >60m/missing	9/36	14/36	+13.89pp	$0.6531 -> $0.4457	60.79M -> 34.67M	2048 snapshot, no continuation
issue293-continuation-canary-002	canary	1/1	1/1	0pp	$0.0496 -> $0.0323	4.69M -> 2.23M	continuation + replay fix smoke
issue293-continuation-rescue-001	rescue, prior tool-step-cap tasks	10/32	9/32	-3.125pp	$0.8836 -> $0.5172	83.38M -> 40.46M	continuation on, A/B pair concurrency on

Local artifacts used for the evidence above:

• maka-eval/runs/issue293-prune-ab/issue293-active-prune-default-le30-005/runtime-policy-ab-result.json
• maka-eval/runs/issue293-prune-ab/issue293-active-prune-default-30to45-001/runtime-policy-ab-result.json
• maka-eval/runs/issue293-prune-ab/issue293-active-prune-default-45to60-001/runtime-policy-ab-result.json
• maka-eval/runs/issue293-prune-ab/issue293-active-prune-default-gt60missing-001/runtime-policy-ab-result.json
• maka-eval/runs/issue293-prune-ab/issue293-continuation-canary-002/runtime-policy-ab-result.json
• maka-eval/runs/issue293-prune-ab/issue293-continuation-rescue-001/runtime-policy-ab-result.json
• Report markdown exists beside each result as runtime-policy-ab-report.md.

Scope

Changed:

• Runtime context-budget defaults and diagnostics for active current-turn tool-result pruning.
• AI SDK replay materialization for interleaved parallel tool_call / tool_result events.
• Harbor continuation env passthrough, continuation summaries, max total runtime step enforcement, per-turn cap diagnostics, and controller/report propagation.
• Continuation error handling so a later-turn sendMessage failure is reported as a failed invocation instead of being hidden behind an earlier step-cap result.
• Continuation step accounting now uses token-usage runtimeSteps diagnostics instead of counting RuntimeEvent rows.
• Runtime-policy A/B execution so both arms for the same task/rep start concurrently.
• Regression tests for replay grouping, continuation behavior, continuation error propagation, max total runtime steps, A/B pair concurrency, reports, and output contracts.

Not included:

• No UI changes.
• No provider secret or benchmark result artifacts are committed.
• No maka-eval/ tracked files; local run outputs stay git-excluded.
• No repo-local one-off Issue 293 rescue runner; the rescue run remains documented by local artifacts and explicit run settings.
• No test-file boundary cleanup for packages/headless/src/__tests__/prompt-ab-run.test.ts; that file should be split in a follow-up pure test-organization PR.

Verification

Local gates:

• npm --workspace @maka/runtime run build
• npm --workspace @maka/headless run build
• node --test packages/runtime/dist/__tests__/ai-sdk-backend.test.js packages/runtime/dist/__tests__/active-tool-result-prune.test.js packages/runtime/dist/__tests__/runtime-runner.test.js packages/headless/dist/__tests__/harbor-cell.test.js packages/headless/dist/__tests__/prompt-ab-run.test.js packages/headless/dist/__tests__/runtime-policy-ab-run.test.js packages/headless/dist/__tests__/fixed-prompt-controller.test.js packages/headless/dist/__tests__/cell-output.test.js packages/headless/dist/__tests__/harbor-adapter.test.js
• git diff --check

Result: 221 related tests passed.

User-facing impact

None for the desktop UI. This affects benchmark/headless Harbor execution and runtime replay robustness. No changelog, docs, breaking changes, or migrations are included.

Reviewer notes

This is intentionally one PR because the 2048 policy evidence depends on the runtime replay fix, Harbor continuation, and A/B pair-concurrency fix being exercised together. The commits are split by concern for review.

The active-prune default of 2048 is intentional. Issue #293 was updated so this is no longer scoped as a non-goal: 8192 was the archive retrieval budget, while active current-turn prune should match the measured 2048 threshold and the existing stale tool-result prune default.

packages/headless/src/__tests__/prompt-ab-run.test.ts is now over 1.4k lines and mixes manifest, generic A/B execution, fingerprinting, summary/rendering, and prompt-specific run tests. That boundary cleanup is worthwhile, but it should be a separate pure test-organization PR so this PR does not mix behavior changes with file-move noise.