refactor(headless): move the benchmark identity out of the build a graded arm mounts by Astro-Han · Pull Request #2303 · apache/maka

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han · 2026-08-06T01:53:37Z

What

A graded arm executes packages/headless/dist, so anything compiled into a shipped module is readable from inside the container being scored. harness-ab-manifest.ts carried each benchmark's pinned revision, task-tree fingerprint and full task list — 89 Terminal-Bench ids and 113 DeepSWE ids — and every one of them shipped.

That is the exact material of the #1970 incident: an arm read the pinned revision out of the mount, fetched the task's reference solution from the public repository at that revision, and recorded a pass that measured retrieval.

#2298 narrowed the mount so docs/eval and src no longer reach the container. It could not close this one, because the mount is a list of paths and the identity was inside a path the container legitimately needs.

How

The data moves to packages/headless/harbor/benchmark-identity.json, loaded by harbor/benchmark-identity.mjs. Neither is mounted by any arm — agent-repo-mount.ts names exactly one file out of harbor/, and it is run-cell.mjs.
harness-ab-manifest.ts keeps the assertions and loses the answers: assertTerminalBench21TaskSet and its three siblings become assertFrozenTaskSet(label, expected, actual) and assertFrozenTaskTreeFingerprint(label, expected, actual). Error messages are unchanged.
Each entry in HARNESS_BENCHMARK_PROFILES now carries its own taskTreeFingerprint. With that, the DeepSWE-full and Terminal-Bench task-source branches were the same code under different names and collapse into one; only the subset-30 profile still differs, because its tree carries more tasks than the frozen set.
No product module was affected — the identity was only ever read by the two host-side harbor/*.mjs entrypoints and by tests.

Verification

A new test in harness-ab-manifest.test.ts reads every shipped dist/*.js and asserts it contains no revision, no upstream URL and none of the 202 task ids. This is the first check on what the mounted files contain rather than on which paths are declared, and it immediately found a real leak: harbor-smoke-config.ts hardcoded '*sqlite-with-gcov' — a Terminal-Bench 2.1 task id — as a fallback the smoke manifest was already supplying. Removed, and it now fails loudly instead.

Red/green confirmed: restoring that literal fails the new test.

Loading the real entrypoint reproduces all three profiles byte-for-byte:

terminal-bench-2.1 d49e28f1 89 sha256:456826a
deep-swe-1.1 6db64a40 30 sha256:508aedc
deep-swe-1.1-full 6db64a40 113 sha256:973091a

npm run -w @maka/headless test — 1422 pass, 0 fail. Lint and format clean.

Known residue

Compiled tests under dist/__tests__ are still mounted and still mention individual task ids in fixtures. Closing that means splitting the test build out of dist, which is a cross-cutting change to every workspace's tsconfig and the test runner; a fixture id is also not the benchmark's identity in the way a pinned revision and a complete task list are. Left for its own change.

dist/terminal-bench-adapter.js remains readable and remains in the graded closure. It is genuinely needed to run, and what it reveals — a verifier grades by exit code — is generic harness mechanics rather than any task's answer.

…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.

Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy.

…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han mentioned this pull request Aug 6, 2026
fix(headless): mount Maka's build outputs into the task container, not the repo root #2298
Merged

Astro-Han force-pushed the refactor/benchmark-identity-out-of-dist branch from 1a563c6 to 2a08231Compare August 6, 2026 02:12

Astro-Han changed the base branch from main to fix/harness-maka-repo-mount-scopeAugust 6, 2026 02:12

Astro-Han force-pushed the refactor/benchmark-identity-out-of-dist branch 2 times, most recently from e79c5ba to 683e1c9Compare August 6, 2026 06:04

Astro-Han changed the base branch from fix/harness-maka-repo-mount-scope to mainAugust 6, 2026 06:04

Astro-Han added 3 commits August 6, 2026 14:10

test(headless): check every mounted file, and the fingerprint the lis…
d0e7cf3
…t forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one.

Astro-Han force-pushed the refactor/benchmark-identity-out-of-dist branch from 683e1c9 to d0e7cf3Compare August 6, 2026 06:11

refactor(headless): drop a benchmark task id from a retry comment
86b0877

Astro-Han marked this pull request as ready for review August 6, 2026 06:25

Astro-Han merged commit 43b6694 into mainAug 6, 2026
11 checks passed