feat(headless): report active prune evidence

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.

[Astro-Han]

Summary

Add active tool-result prune diagnostics to Harbor cell output, A/B summaries, and markdown reports, including a B-active paired comparison slice.

Why

This covers part of the #293 phase-1 reporting gap for active tool-result pruning. Runtime already emits active prune diagnostics, but headless A/B reports previously only recognized stale prunedToolResults. That made active-prune runs hard to inspect and could mix untriggered or unpaired attempts into safety conclusions.

Scope

Changed:

• Summarize and validate activePrunedToolResults, activeEstimatedTokensSaved, and activeArchiveFailures in Harbor cell output.
• Include activeToolResultPrune in Harbor context budget policy snapshots.
• Add active-prune reporting keyed by activePrunedToolResults > 0.
• Report the active-prune subset as a B-active paired slice: the B attempts that activated active prune, plus the matching A (taskId, rep) attempts.
• Report active subset task count, attempt count, observed/missing/coverage, pass rate, full token/cost, failure diagnostics, and context/archive diagnostics.
• Keep activated attempt refs active-only so stale-only pruning does not appear as active-prune evidence.
• Render existing archive diagnostics: archive placeholders, archive write failures, retrieved count/tokens, retrieval skipped, retrieval failures, and active archive failures.
• Add/adjust headless tests for active diagnostics, active-only subset reporting, stale-only exclusion, paired active subset reporting, archive metric rendering, missing-pair coverage, and full token/cost rendering.

Not included:

• No stale prior-context prune validation.
• No multi-turn or autonomous continuation harness.
• No archive reason-count schema expansion.
• No field-list-driven context-budget metric refactor; that maintenance cleanup can be a follow-up PR.
• No formal non-inferiority claim.

Verification

• RED: npm --workspace @maka/headless test -- --test-name-pattern "renders active prune subset pair coverage" failed because the active subset line omitted attempts/observed/missing/coverage and full token/cost fields.
• GREEN: npm --workspace @maka/headless test -- --test-name-pattern "active prune subset|context budget activation": 490 pass, 0 fail. The package test script still ran the full @maka/headless suite.
• Re-rendered active-prune smoke report from existing results.jsonl: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001/runtime-policy-ab-report.md
• Active-prune smoke run: /Users/yuhan/.local/maka-eval/runs/issue293-prune-ab/issue293-active-smoke-001
• Tasks: count-dataset-tokens, extract-elf
• A: context budget off
• B: active prune + archive retrieval
• B activation: 2/2 attempts, 2 tasks
• activePrunedToolResults=1091
• activeEstimatedTokensSaved=1400611
• activeArchiveFailures=0
• Infra/plumbing failures: 0
• Clash/Mihomo observed traffic delta: 535.92 MiB

User-facing impact

None. This is headless benchmark reporting only.

Reviewer notes

The smoke run showed active pruning activates and reports correctly, but both B attempts ended with incomplete_tool_calls after hitting the single-turn 50-step cap. That is a harness limitation for longer tasks, not a non-inferiority result. Formal #293 evidence should wait for either shorter stable tasks or a benchmark-safe multi-turn continuation harness.