Skip to content

test(headless): retrieval arm activation contract + prune A/B smoke profiles - #623

Merged
Astro-Han merged 2 commits into
mainfrom
test/481-retrieval-arm-activation-contract
Jul 7, 2026
Merged

test(headless): retrieval arm activation contract + prune A/B smoke profiles#623
Astro-Han merged 2 commits into
mainfrom
test/481-retrieval-arm-activation-contract

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

Two pieces preparing the next #481 measurement slices (stale-prune multi-turn A/B, then eager-retrieval A/B):

  1. A harbor-cell contract test pinning the live shape of an eager archive-retrieval arm: with post-feat: enable staleToolResultPrune by default on headless, desktop, and CLI #621 defaults plus MAKA_CONTEXT_ARCHIVE_RETRIEVAL=on, the built backend options must carry the placeholder producer (staleToolResultPrune.enabled), the eager retrieval policy, and both archive writer/reader callbacks.
  2. Three Terminal-Bench smoke profiles encoding the exact A/B arm shapes: maka-prune-default (defaults + continuation, low per-turn step cap so turns actually roll over), maka-stale-off (A arm for the stale A/B), maka-retrieval-on (B arm for the retrieval A/B).

Why

Refs #481 (see the status-update comment). The #340-era arms enabled MAKA_CONTEXT_ARCHIVE_RETRIEVAL without stale prune; eager retrieval only hydrates stale-kind placeholders, so the flag was structurally inert and nobody noticed. The contract test makes that failure mode impossible to reintroduce silently, and the committed profiles make tomorrow's runs reproducible instead of living in local runner configs.

Verification

  • npm run -w @maka/headless test: 737 pass / 0 fail / 1 skipped (new contract test included)
  • terminal-bench-sample-runs.json parses (python3 json.load); the smoke script resolves profiles dynamically from the manifest, usage text updated

Reviewer notes

  • Two commits: the contract test and the benchmark profiles are independent rollback units.
  • The profiles deliberately set MAKA_MAX_STEPS=50 with continuation max 3 turns / 150 total steps so tasks cross turn boundaries — stale prune and retrieval only act cross-turn, and a single-turn run measures nothing (the feat(headless): validate active-prune 2048 #340 lesson).

@Astro-Han
Astro-Han merged commit dff48fa into mainJul 7, 2026
3 checks passed
@Astro-Han
Astro-Han deleted the test/481-retrieval-arm-activation-contract branch July 7, 2026 17:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han