feat(benchmark): extend economic mechanism A/B testing beyond PR #340 active tool prune

[Astro-Han]

Background

PR #340 (feat(headless): validate active-prune 2048) established A/B testing for active tool prune on Terminal Bench and reported:

• Token: -41.7%
• Cost: -31.6%
• Performance: +2.48pp

This showed the in-repo runtime-policy-ab-run framework can isolate a single mechanism's economic effect.

Goal

Extend that methodology to the remaining economic mechanisms, starting with an inventory of what is toggleable today and what needs wiring.

Current gaps / questions

1. Mechanism inventory is unclear

We need a single reference table that lists every cost-relevant mechanism, its current Harbor env support, and whether it is in scope for the first token-savings benchmark.

2. Some mechanisms are not toggleable for A/B

Mechanisms confirmed toggleable via Harbor arm-local contextEnv (MAKA_CONTEXT_*):

• MAKA_CONTEXT_ACTIVE_TOOL_RESULT_PRUNE — in scope for token-savings
• MAKA_CONTEXT_STALE_TOOL_RESULT_PRUNE — in scope for token-savings
• MAKA_CONTEXT_ARCHIVE_RETRIEVAL — in scope for token-savings (post-prune hydration)

Mechanisms toggleable via Harbor arm-local contextEnv but out of scope for first token-savings slice (capacity mechanisms):

• MAKA_CONTEXT_ACTIVE_FULL_COMPACT
• MAKA_CONTEXT_SEMANTIC_COMPACT

Mechanisms toggleable via Harbor profile env but not arm-local contextEnv (because runtime-policy-ab-run only whitelists MAKA_CONTEXT_*):

• MAKA_ECONOMY_TASK_MODE

Mechanisms not toggleable via Harbor env at all:

• Synthesis cache — ContextBudgetPolicy.synthesisCache has no Harbor switch. Desktop uses it, but headless lacks loadSynthesisCache / writeSynthesisCache callbacks and artifact lifecycle wiring.
• History search — RuntimeEventHistorySearchPolicy is desktop-only; not in HARBOR_CELL_CONTEXT_ENV_KEYS.
• Tool availability — Headless currently always uses economy mode via buildIsolatedHeadlessToolAvailability(). There is no MAKA_CONTEXT_TOOL_AVAILABILITY or similar env override.

3. No living benchmark doc

Results are scattered (Twitter threads, private runs). There is no in-repo reference showing mechanism X on dataset Y saves Z% tokens at cost W% with pass rate change V.

Proposed scope — first slice

Phase 1: Mechanism inventory + toggle feasibility

Create a reference table in this issue (or a linked doc) covering:

Mechanism	Current Harbor support	In scope for token-savings benchmark?	Notes
Active tool-result prune	MAKA_CONTEXT_ACTIVE_TOOL_RESULT_PRUNE	yes	Already validated in PR #340
Stale tool-result prune	MAKA_CONTEXT_STALE_TOOL_RESULT_PRUNE	yes	Same logic as active, but for prior turns
Archive retrieval (post-prune hydration)	MAKA_CONTEXT_ARCHIVE_RETRIEVAL	yes	Runtime-side hydration only; model cannot actively retrieve. Depends on MAKA_CONTEXT_TOOL_RESULT_ARCHIVE_DIR
Economy-task prompt policy	MAKA_ECONOMY_TASK_MODE	maybe	Profile-env toggleable, not arm-local contextEnv; may need framework extension
Tool availability / schema economy	no Harbor env	needs design	Headless always uses economy mode; needs explicit switch
Synthesis cache	desktop env exists, Harbor wiring missing	needs separate slice	Needs loadSynthesisCache / writeSynthesisCache callbacks + artifact lifecycle, not just env parsing
History search	desktop env exists, Harbor wiring missing	needs design	Gates archive_retrieval history_search_gated mode; adds retrieved-history tokens
Active full compact / semantic compact	MAKA_CONTEXT_ACTIVE_FULL_COMPACT, MAKA_CONTEXT_SEMANTIC_COMPACT	no	Capacity mechanisms; toggleable today but out of scope for first token-savings slice
History compact	desktop env only, not in Harbor env keys	no	Desktop-only, like synthesis cache

Phase 2: Expose or verify toggles

For mechanisms already wired (active prune, stale prune, archive retrieval), verify they work through runtime-policy-ab-run and produce diagnostics.

For economy task mode, decide whether runtime-policy-ab-run needs a broader arm-local env allowlist beyond MAKA_CONTEXT_*.

For synthesis cache and tool availability, scope the engineering work in follow-up comments or child issues before env parsing.

Phase 3: Run a smoke benchmark

Before any 30-task or 121-task run, do a single-task smoke run that proves:

• the mechanism activated (visible in diagnostics or output artifacts)
• the baseline and arm configs are actually different
• metrics are extractable from maka-cell-output.json

Phase 4: Living doc (after first verified result)

Once we have at least one new mechanism validated, create docs/economic-mechanisms-benchmark.md with:

• test date, sample size, model, arm env
• baseline vs arm metrics table (input tokens, cache read/miss, output tokens, cost, pass rate, steps)
• interpretation and raw data link

What this issue does NOT cover

• Changing mechanism behavior (measurement only)
• Renaming existing env vars
• Testing capacity mechanisms (active full compact, semantic compact, history compact) for token savings
• 121-task full matrix or leave-one-out (those come after Phase 1 inventory is settled)

Related

• PR feat(headless): validate active-prune 2048 #340 — feat(headless): validate active-prune 2048 (active tool prune A/B baseline)
• runtime-policy-ab-run.ts infrastructure
• terminal-bench-sample-runs.json profile definitions

[ARE404]

I'd like to take the synthesis-cache slice from this issue (the "synthesis cache — needs a separate slice" row). I traced the current state:

• The mechanism is fully implemented in the runtime — AiSdkBackend consumes loadSynthesisCache / writeSynthesisCache and the contextBudget.synthesisCache policy (read_write mode, maxBlocks, maxEstimatedTokens). It's inert if the loader isn't provided (ai-sdk-backend.ts ~L1746).
• Desktop wires it: synthesis-cache-artifacts.ts (loadSynthesisCacheBlocksFromArtifacts / persistSynthesisCacheBlocksToArtifacts) backed by the artifact store, injected in main.ts, with the policy built in context-budget-policy.ts.
• Headless does not: harbor-cell.ts's new AiSdkBackend({…}) omits both callbacks, buildHarborCellContextBudgetBackendOptions builds no synthesisCache policy, there's no MAKA_CONTEXT_SYNTHESIS_CACHE env, and headless has no artifact store for the blocks.

Proposed plan:

1. Persistence — a run-scoped synthesis-cache artifact store for headless (reuse @maka/storage artifacts pointed at the cell's artifact dir, mirroring desktop). This is the "artifact lifecycle" piece.
2. Toggle — build the synthesisCache sub-policy in buildHarborCellContextBudgetBackendOptions gated by a new MAKA_CONTEXT_SYNTHESIS_CACHE, and add it to HARBOR_CELL_CONTEXT_ENV_KEYS + the runtime-policy-ab-run arm-local contextEnv allowlist so it's A/B-toggleable.
3. Callbacks — inject loadSynthesisCache / writeSynthesisCache into the harbor-cell backend, backed by that store.
4. Smoke — a single-task run proving the cache activates (read/write counts + synthesisCacheWriteSkipped* in maka-cell-output.json) and that baseline vs arm configs actually differ.
5. A/B + doc — baseline vs arm on terminal-bench-sample, then the first entry in docs/economic-mechanisms-benchmark.md (Phase 4).

Two questions before I start:

• Preferred location for headless synthesis-cache artifacts — the run-scoped cell artifact dir, or a dedicated subdir?
• Track this as a child issue, or under feat(benchmark): extend economic mechanism A/B testing beyond PR #340 active tool prune #481?

Measurement + wiring only; no changes to mechanism behavior.

[Astro-Han]

@ARE404 Thanks for picking this up. Two answers on your open questions, plus one gap I want called out in the plan before you start.

Artifact location: use the storage root that's already resolved, not a new dir or subdir.

@maka/storage's ArtifactStore layout is fixed: createArtifactStore(workspaceRoot) puts blocks in {workspaceRoot}/artifacts/{sessionId}/{id}-{name} with a metadata.jsonl index. So the choice that matters is what to pass as workspaceRoot, and there's already an answer.

harbor-cell.ts:392-394 resolves both:

outputDir = env.MAKA_OUTPUT_DIR ?? '/logs/agent'
storageRoot = env.MAKA_STORAGE_ROOT ?? join(outputDir, 'maka-storage')

storageRoot is the run-scoped root meant for @maka/storage. Pass it to createArtifactStore and synthesis-cache blocks land under {outputDir}/maka-storage/artifacts/{sessionId}/, run-scoped and session-scoped for free. It also doesn't collide with the tool-result archive, which writes flat files to {outputDir}/tool-result-archives/ (harbor-cell.ts:1359). No new dir env, no dedicated subdir.

One thing step 1 should spell out: loadSynthesisCacheBlocksFromArtifacts and persistSynthesisCacheBlocksToArtifacts live in apps/desktop/src/main/synthesis-cache-artifacts.ts, and headless can't import app-layer code. The runtime pieces they call (buildSynthesisCacheBlocksFromHydratedArchives, estimateTokens, validateSynthesisCacheBlockShape) are already in @maka/runtime. What's missing is lifting the two artifact-store glue functions to a shared package (@maka/storage or @maka/runtime) so desktop and headless share one implementation. That step isn't covered by "mirror desktop," so make it explicit in step 1.

Tracking: child issue, linked back here.

Open a child issue and land it as its own PR. The slice is a full vertical (helper lift, persistence, toggle, callbacks, smoke, doc entry) with its own verification boundary, which is how we split PRs by concern. Keeping the implementation details out keeps #481 as the methodology overview, where it belongs. Link the child issue back to this one.

[Astro-Han]

Status update: #340 attribution confirmed, retrieval wiring gap identified, next slice = stale prune

#340's result attributes to active prune alone. Per #293's phase-1 definition, prune on meant active tool-result pruning only; stale prior-context pruning was explicitly excluded from the phase-1 conclusion. The B arm also enabled MAKA_CONTEXT_ARCHIVE_RETRIEVAL, but that flag was structurally inert there: retrieval candidates are collected only from stale-kind archive placeholders in the replayed events (context-budget.ts isArchivedToolResultPlaceholder), while active-prune placeholders live in transient in-flight messages and never persist to events. With stale prune off, retrieval had nothing it could ever hydrate. Net: the -41.7% tokens / -31.6% cost / +2.48pp result is a clean single-mechanism attribution to active prune.

Archive retrieval and history_search_gated have therefore never been effective in the Harbor A/B path — not by decision, but by wiring: eager retrieval requires stale-kind placeholders that only stale prune produces, and the gated mode requires historySearch, which is desktop-only and absent from HARBOR_CELL_CONTEXT_ENV_KEYS. Both mechanisms are fundamentally model-recovery aids (getting pruned content back within reach). Whether and how to rebuild recovery — including the model-initiated history-search direction — is an architecture question tracked separately, not a measurement slice here.

Next measurement slice: stale prune multi-turn A/B. #293 already listed this as the required follow-up, and it is now more urgent than planned: #621 turned stale prune default-on everywhere on mechanism-coherence grounds, and with no effective recovery path the pass-rate cost of cross-turn pruning needs a number. Design: continuation-enabled runs, A = defaults + MAKA_CONTEXT_STALE_TOOL_RESULT_PRUNE=off, B = defaults. Result seeds docs/economic-mechanisms-benchmark.md (Phase 4) alongside the #340 entry.

Economy task mode is dropped from the measurement plan. Reading economy-task-policy.ts, it is a benchmark prompt device (behavioral constraints for data-transform tasks, auto-triggered by instruction substrings), not a generalizable economy mechanism; A/B-ing it would measure that prompt, not the mechanism family.

Remaining after stale prune: semanticCompact vs activeFullCompact head-to-head (paid LLM vs free deterministic compaction, informs the desktop default; note current semanticCompact is the post-rewrite real LLM implementation). Synthesis cache continues in #578.

[Astro-Han]

Closing this broad tracker now.

The stale-prune slice identified here as the next measurement is no longer pending:

• feat: enable staleToolResultPrune by default on headless, desktop, and CLI #621 enabled stale tool-result pruning by default across Headless, Desktop, and CLI.
• test(headless): retrieval arm activation contract + prune A/B smoke profiles #623 landed the multi-turn A/B profiles and activation contract.
• fix(headless): separate A/B outcomes from evidence #770 records the completed 81-task stale-prune WAL replay: both arms observed 81/81 outcomes, with 80 evaluated pairs, one infrastructure exclusion, and no missing pairs.

The synthesis-cache slice is independently tracked by #578.

The only remaining proposal in this umbrella is a semanticCompact vs activeFullCompact comparison. That experiment is not currently in the decision or execution window. If we decide to run it later, a narrow, pre-registered issue based on the runtime and benchmark state at that time will be cleaner than keeping this broad tracker open.

Thanks to everyone who helped turn the original mechanism inventory into implemented and measurable paths.

中文

关闭这个宽泛的 tracking issue。

这里列为下一项测量工作的 stale prune 已经不再处于待办状态：

• feat: enable staleToolResultPrune by default on headless, desktop, and CLI #621 已在 Headless、Desktop 和 CLI 中默认启用 stale tool-result pruning。
• test(headless): retrieval arm activation contract + prune A/B smoke profiles #623 已加入多轮 A/B profiles 和 activation contract。
• fix(headless): separate A/B outcomes from evidence #770 记录了已完成的 81-task stale-prune WAL 重放：两组均观察到 81/81 个结果，共 80 个有效配对、1 个基础设施排除，没有缺失配对。

Synthesis cache 已由 #578 独立跟踪。

这个 umbrella 中只剩 semanticCompact 与 activeFullCompact 的对比提议，但它目前不在决策或执行窗口。以后如果决定运行，基于届时的 Runtime 和 benchmark 状态新建一个范围明确、预注册的实验 issue，会比长期保留这个宽泛 tracker 更干净。

感谢大家把最初的机制盘点逐步变成已经实现、可以测量的路径。