Uh oh!
There was an error while loading. Please reload this page.
feat(headless): run DeepSWE via Pier as a benchmark axis of the harness A/B - #1398
Merged
Conversation
- port codex_agent.py to harness_compat (Pier tree selection) and add the Pier-required install_spec/network_allowlist, sharing the proxy-endpoint parsing with kimi_code_agent via provider_proxy.py - extend createPierTaskRunner with a codex arm: pinned toolchain mount, --ak constructor kwargs, proxy wiring, and resolver-credential proxy routing for the Maka arm (Codex OAuth) - freeze the DeepSWE subset-30 task list + task tree fingerprint in harness-ab-manifest and generalize the manifest benchmark field - add run-deepswe-ab.mjs: Maka vs Codex on openai-codex/gpt-5.6-sol via Pier, reusing the harness A/B engine, manifest, and report pipeline Issue #1343.
…f run-harness-ab A benchmark is a bound pair of frozen task source and executor (Terminal-Bench 2.1 <-> Harbor, DeepSWE <-> Pier), so it becomes HARNESS_BENCHMARK_PROFILES + MAKA_HARNESS_AB_BENCHMARK on the existing driver, orthogonal to the competitor axis, instead of a parallel run script. Task selection, run-id/order-seed derivation, the oracle-evidence hook, and the runner factory dispatch on the profile; the Terminal-Bench default behavior (env names, run ids, order seed, resume fingerprints) is unchanged. Deletes run-deepswe-ab.mjs.
…dentity Review follow-ups on the DeepSWE benchmark axis: - codex_agent.py hydrates codex.txt and the sessions rollouts in run()'s finally, mirroring the Kimi adapter: Pier replaces the default /logs/agent bind mount under --mounts-json while capabilities.mounted stays true, so without this download a real token-burning run reads zero events and every cell is misclassified as failed. - Benchmark profiles now carry their bound executor explicitly, and a Pier benchmark freezes `pier --version` (stdout only) into the toolchain fingerprint so a resume across a Pier upgrade forks instead of mixing cells; the Harbor payload stays byte-identical. - resolveHarnessBenchmarkProfile's no-argument default reads MAKA_HARNESS_AB_BENCHMARK, so defaulted call sites agree with the production entry points. - resolveFrozenBenchmarkTasks passes selectTasksByIds an options object, so missing-task errors keep their DeepSWE subset-30 scope prefix.
…iew polish
- The manifest's benchmark section carries { executor: { id: 'pier', version } }
for Pier benchmarks, so the frozen executor identity is auditable without
trial-log archaeology; the key is absent for Harbor benchmarks and a test
locks the byte-identical Terminal-Bench payload.
- The opencode × Pier guard fires in main() before any run root or lock is
created, so a doomed pairing leaves no run state behind.
- The headless README documents the benchmark axis (MAKA_HARNESS_AB_BENCHMARK,
the DeepSWE subset-30 profile, and the Pier executor pin).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #1343. Makes the DeepSWE (Datacurve) benchmark runnable through the existing harness A/B driver, executed via Pier ≥ 0.3.0, with the Codex CLI as a second proxied competitor arm.
A benchmark is a bound pair of frozen task source and executor (Terminal-Bench 2.1 ↔ plain Harbor 0.13.2, DeepSWE ↔ Pier), so instead of a parallel run script it becomes a second entry in
HARNESS_BENCHMARK_PROFILES, selected byMAKA_HARNESS_AB_BENCHMARKand orthogonal to the existing competitor axis:codex_agent.pymoves ontoharness_compat(Pier selects thepier.*tree; plain Harbor keepsharbor.*) and implements the Pier-requiredinstall_spec()(None — the pinned toolchain must never be reinstalled from the network) andnetwork_allowlist()(host provider proxy only, noapi.openai.comfallback). Proxy endpoint parsing is shared with the Kimi adapter viaprovider_proxy.py.createPierTaskRunnergains thecodexarm: pinned-toolchain mount,--akconstructor kwargs (version,reasoning_effort), Squid-legal fixed-port lock generalized to all container-CLI arms, and resolver-backed credentials (Codex OAuth) now force the proxy path on the Maka arm, matching the Harbor runner's contract.harness-ab-manifest.tsfreezes the DeepSWE identity:DEEP_SWE_REVISION, the 30-task discriminative subset (11×p=0.25 + 17×p=0.50 + 2 anchors from the public v1.1 leaderboard trials), and its task-tree fingerprint.run-harness-ab.mjsdispatches task discovery/assertions, run-id and order-seed derivation, the oracle-evidence hook (Terminal-Bench only), and the runner factory on the benchmark profile. Terminal-Bench default behavior — env names, run ids, order seeds, resume fingerprints — is byte-for-byte unchanged; the k3 order-seed special case collapses into the general template it already equaled.Resulting matrix from one driver:
BENCHMARK=deep-swe-1.1×COMPETITOR=kimi-code|codexcovers both the #1343 K3 target and the GPT-5.6-Sol/Codex-OAuth comparison;opencodeon DeepSWE fails fast (no Pier arm).Verification
packages/headlesssuite: 1287 pass / 0 fail (new: 6 Pier codex-arm cases, DeepSWE benchmark-axis CLI case).tests/test_harness_compat.py: 7/7 pass under the Pier venv, 3 pass + 4 skips under plain Harbor (codex adapter now covered in both trees).pier runend-to-end with the fake backend ondasel-html-document-format: adapter loads on the Pier tree, offline container, verifier grades reward 0, structured verifier outcome and cell output read back correctly.npm run format/npm run lintclean.Review focus
The Terminal-Bench path must be behavior-identical:
resolveHarnessAbRunId, order-seed derivation, and the oracle-evidence flow moved under profile dispatch with default parameters preserving the old signatures.