Skip to content

feat(headless): add OpenCode Pier arm for DeepSeek-metered full DeepSWE - #1731

Merged
Astro-Han merged 3 commits into
mainfrom
feat/headless-opencode-pier-arm
Aug 1, 2026
Merged

feat(headless): add OpenCode Pier arm for DeepSeek-metered full DeepSWE#1731
Astro-Han merged 3 commits into
mainfrom
feat/headless-opencode-pier-arm

Conversation

@Astro-Han

@Astro-HanAstro-Han commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacks on #1730 (the deepseek-v4-flash-max runtime profile). Makes deep-swe-1.1-full | deepseek-v4-flash-max | opencode runnable end to end: an external runner can now execute the full 113-task DeepSWE v1.1 leaderboard set, Maka vs OpenCode, both on DeepSeek V4 Flash at max reasoning with metered official-API billing.

DeepSWE executes under Pier, and OpenCode had no Pier arm — PierAgent was maka | kimi-code | codex, and opencode_agent.py was built on the plain-Harbor tree only. This PR ports the arm following the exact seam #1398 cut for Codex:

  • opencode_agent.py moves onto harness_compat (pier tree under Pier, harbor tree under plain Harbor — the trees are type-incompatible and Pier's TrialResult rejects harbor-tree AgentInfo), and implements the Pier-required install_spec() (None — the pinned toolchain is bind-mounted and checksum-verified, never reinstalled from the network) and network_allowlist() (host provider proxy only, no fallback domain)
  • pier-task-runner.ts gains the opencode arm: pinned-toolchain mount at /opt/maka-opencode-toolchain, adapter version pinned to OPENCODE_TOOLCHAIN_SPEC, toolchain fingerprint and MAKA_OPENCODE_VARIANT (reasoning effort) forwarded via --ae; provider auth flows through the per-attempt host proxy lease (env-file only, never argv) like every competitor arm
  • one composition whitelist entry: deep-swe-1.1-full | deepseek-v4-flash-max | opencode → run id deepseek-v4-flash-maka-vs-opencode-deepswe-full-v1
MAKA_HARNESS_AB_BENCHMARK=deep-swe-1.1-full \
MAKA_HARNESS_AB_RUNTIME=deepseek-v4-flash-max \
MAKA_HARNESS_AB_COMPETITOR=opencode \
MAKA_HARNESS_AB_LIMIT=5 \
node packages/headless/harbor/run-harness-ab.mjs

Verification

  • packages/headless suite: 1453 pass / 0 fail (new: 4 OpenCode Pier-arm runner cases — adapter import path, version pin, toolchain mount requirement, proxy/toolchain/variant wiring; 1 composition case)
  • harbor/tests/test_harness_compat.py: 9 pass / 0 fail under the Pier venv (new: OpenCode network-shape + install_spec case; the tree-contract test now covers all four adapters), 4 pass + 4 skips under plain Harbor — historical shape preserved
  • TS stub-based OpenCode adapter contract still green against the harness_compat import switch
  • real dry-run against the pinned DeepSWE task source: benchmark=deep-swe-1.1-full (113 frozen tasks via pier); runtime=deepseek-v4-flash-max (deepseek/deepseek-v4-flash, max, metered); competitor=opencode@1.17.18
  • npm run format / npm run lint clean
  • Not run: a live metered Pier trial (no new spend launched for this PR)

Review response

External review (reproduced against the real Pier venv) found three defects in the first revision; all fixed in d34f8a1ab:

  • model name format: the Pier runner passed the provider-stripped bare id (deepseek-v4-flash) via -m, but the OpenCode adapter requires provider/model — every competitor cell would have failed before any model call. The runner now reuses the Harbor runner's modelForOpenCode seam for the opencode arm.
  • version provenance: the pinned version was validated but never forwarded; the opencode arm now rides --ak version=… like Codex, so trial/trajectory metadata records 1.17.18 instead of unknown.
  • native-Linux VM proxy host: Pier's default advertised host host.docker.internal does not resolve on native Linux Docker. New MAKA_HARNESS_AB_PROVIDER_PROXY_ADVERTISED_HOST env feeds the existing providerProxyAdvertisedHost seam (e.g. 172.17.0.1), documented in the README.

New test coverage: argv asserts -m deepseek/deepseek-v4-flash and --ak version=… on the opencode wiring case; a hub-options resolver case for the advertised-host env.

Second review round (fresh eyes, verified against the real Pier venv and pier's docker.py):

  • Fixed in 7e828da9b — container-log hydration: Pier's explicit --mounts-json replaces the default /logs bind-mount (pier environments/docker/docker.py), so the OpenCode CLI stream stayed inside the container; _error_messages() and trajectory/cost parsing would have seen nothing and misclassified real provider errors. The adapter now downloads /logs/agent/opencode.txt before the error-event check and again in run()'s finally — the same seam the Kimi and Codex arms already carry. New adapter test pins remote path, host target, and idempotence under both interpreters.
  • Pushed back — systemPromptHash on competitor cells: all three competitor adapters (kimi, codex, opencode) record the controller prompt hash from MAKA_SYSTEM_PROMPT while the CLIs run with their native prompts; that is the established execution-identity contract on main, and the published TB2.1 report (docs: publish DeepSeek Flash Terminal-Bench comparison #1719) was produced under it. Changing the semantic in this PR would make new runs incomparable with published results. If the identity should become model-visible per arm, that is a cross-cutting change for all arms and deserves its own PR.

Review focus

The plain-Harbor OpenCode path (Terminal-Bench 2.1 × DeepSeek, #1730) must stay behavior-identical: the harness_compat switch selects the harbor tree when pier is absent, and the adapter's run logic is untouched — verified by the existing stub contract and the harbor-tree test run.

@Astro-Han
Astro-Hanforce-pushed the feat/headless-opencode-pier-arm branch from 7e828da to 0113e3aCompareAugust 1, 2026 03:57
@Astro-Han
Astro-Han changed the base branch from feat/headless-deepseek-flash-runtime to mainAugust 1, 2026 03:57
@Astro-HanAstro-Han reopened this Aug 1, 2026
@Astro-Han
Astro-Han merged commit 7b90603 into mainAug 1, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han