fix(headless): mount Maka's build outputs into the task container, not the repo root by Astro-Han · Pull Request #2298 · apache/maka

Astro-Han · 2026-08-06T01:18:17Z

Summary

The Maka arm received the repo root at /opt/maka-agent, exempt from the declared-mount rule every competitor arm follows. The exemption's premise was that Maka runs our own runtime — but the model does not know it is ours, and it reads what is reachable.

In the #2245 two-arm run it did exactly that. On extract-elf it ran grep -rl "extract-elf|extract memory values" /opt/maka-agent and read docs/eval/terminal-bench-2.1-*.csv — this benchmark's own per-task pass/fail table from earlier runs. On db-wal-recovery it read packages/headless/src/terminal-bench-adapter.ts, the verifier source. Both cells passed. Neither file is one the runtime executes; both were reachable only because the mount was the root.

The container runs dist, so the arm now gets dependencies, workspace manifests, each workspace's build output, and the single harbor entrypoint the adapter invokes. Sources, docs/, repo history and the harness manifest source are gone from inside it.

Refs #2245

Verification

npm run -w @maka/headless test — 1425 pass, 0 fail. Three existing tests asserted the root mount and now assert the narrowed shape.
npm run lint, npm run format — clean
Ran the container for real, since the contract tests only check that the declared list is self-consistent — they cannot tell whether Maka still starts:

docker run --rm $(mounts from makaRepoPaths()) node:24-slim \
node /opt/maka-agent/packages/headless/dist/cli.js --help

The first attempt failed with Cannot find package '@slack/socket-mode': npm keeps a private node_modules in any workspace whose version pin it could not hoist, and the list only had the root one. That is exactly the class of failure the contract tests cannot see. With those added, the CLI starts and harbor --help resolves.

Reachability from inside the same container, after the change:

path	before	after
`docs/` (per-task results)	readable	blocked
`packages/headless/src/` (verifier source)	readable	blocked
`.git/`	readable	blocked
`maka-eval/`	readable	blocked
`packages/headless/harbor/run-harness-ab.mjs`	readable	blocked
`packages/headless/dist/harness-ab-manifest.js`	readable	readable

Known gap — now closed by #2303

The last row is not fixed here. harness-ab-manifest.ts compiles TERMINAL_BENCH_2_1_REVISION, the upstream repository URL, the task-tree fingerprint and all 89 task ids into packages/headless/dist, which the container needs as a directory. Fixing it means moving benchmark identity out of a compiled runtime package, which is a data-layout change rather than a mount change; #2303 does that, and adds the content-level check this PR's path-level assertions structurally cannot make.

Review round 2

The suite checked that an absent workspace dependency tree is dropped. The incident was the opposite direction: packages/runtime/node_modules exists — npm could not hoist @slack/socket-mode past a version conflict — and omitting it resolved on the host, then failed inside the container on the first import that needed it. Only a live container run caught that.

declares every workspace dependency tree npm could not hoist away now asks the repo directly, and fails on the pre-fix list. The renderer-only exclusion and the workspace list are also derived from the root manifest in one place instead of two.

…t the repo root The Maka arm was exempt from the declared-mount rule on the theory that it runs our own runtime. The model does not know it is ours. In the #2245 two-arm run it grepped the mount and read `docs/eval/terminal-bench-2.1-*.csv` — this benchmark's own per-task pass/fail table from earlier runs — and, on another task, the verifier source at `packages/headless/src/terminal-bench-adapter.ts`. The container executes `dist`, so it now receives dependencies, workspace manifests, build outputs and the one harbor entrypoint the adapter runs. Sources, docs, repo history and the harness manifest are no longer reachable from inside it. Workspace-private `node_modules` are declared but dropped when absent: which ones exist depends on what npm hoisted, and Docker materialises a missing bind source as a new empty directory inside the repo.

The suite checked that an absent workspace dependency tree is dropped. The incident was the opposite: `packages/runtime/node_modules` exists — npm could not hoist `@slack/socket-mode` past a version conflict — and omitting it resolved on the host, then failed in the container on the first import that needed it. Only a live container run caught that. Ask the repo instead.

…obes `install()` tests for both `run-cell.mjs` and `run-host-cell.mjs` before either can run, in every cell-mode branch. Only the first was declared, so the narrowed mount aborted the trial at install rather than degrading it — a regression against the repo-root mount this PR replaces, and one the container smoke check missed because it exercised loading, not installing. The competitor arms already have the check that would have caught it: read the adapter for what it names at a container path and assert the mount declares it. Maka had no equivalent, which is why the omission shipped. It does now — `maka_agent.py` joins path segments onto the mount root instead of writing literals, so the test matches the join. Also asserts the sources-and-records invariant against the mounts actually produced rather than the declared list: a declaration naming no forbidden path still leaks every one of them if the builder mounts the root anyway.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

The list lived twice — once implied by the mount declaration, once spelled in the test that cross-checks it against the root manifest. Two copies of an exclusion drift into a workspace nobody mounts.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

…aded arm mounts (#2303) * refactor(headless): move the benchmark identity out of the build a graded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label. * refactor(headless): hold the identity boundary where it actually is Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy. * test(headless): check every mounted file, and the fingerprint the list forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one. * refactor(headless): drop a benchmark task id from a retry comment

Astro-Han · 2026-08-06T01:18:17Z