Skip to content

feat(headless): add DeepSeek Flash harness A/B runtime - #1730

Merged
Astro-Han merged 1 commit into
mainfrom
feat/headless-deepseek-flash-runtime
Aug 1, 2026
Merged

feat(headless): add DeepSeek Flash harness A/B runtime#1730
Astro-Han merged 1 commit into
mainfrom
feat/headless-deepseek-flash-runtime

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

Brings the deepseek-v4-flash-max runtime profile onto main, cherry-picked from the local experiment branch that produced the published Terminal-Bench 2.1 comparison (#1719). Main currently cannot re-run that comparison: the report is merged but the runtime that produced it is not.

  • runtime profile: DeepSeek provider, deepseek-v4-flash at max reasoning, metered billing against the checked-in DEEPSEEK_V4_FLASH_PRICING identity
  • auth: API key file via MAKA_HARNESS_AB_DEEPSEEK_KEY_FILE, defaulting to ~/.maka/secrets/deepseek.key; each runtime ignores other runtimes' key-file env so a stale export cannot misroute credentials
  • composition: terminal-bench-2.1 | deepseek-v4-flash-max | opencode; DeepSeek OpenCode routes through the per-cell host proxy capability, never a raw key

This is the runtime layer only. The DeepSeek-metered DeepSWE composition and the OpenCode Pier arm stack on top in a child PR.

Verification

  • packages/headless suite: 1449 pass / 0 fail (includes the cherry-picked DeepSeek runtime CLI/adapter cases)
  • real dry-run against the pinned Terminal-Bench 2.1 task source: benchmark=terminal-bench-2.1 (89 frozen tasks via harbor); runtime=deepseek-v4-flash-max (deepseek/deepseek-v4-flash, max, metered); competitor=opencode@1.17.18
  • this exact code produced the frozen run behind the published report in docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md (run id deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7)
  • npm run format / npm run lint clean
  • Not run: a live metered run (no new spend launched for this PR)

Review focus

Cherry-pick fidelity: the only conflict against main was the README compositions paragraph (main gained the deep-swe-1.1-full sentence in #1726); resolution keeps main's wording and adds the DeepSeek composition. Everything else applied byte-clean.

@Astro-Han
Astro-Hanforce-pushed the feat/headless-deepseek-flash-runtime branch from 2ef8469 to 5c6f437CompareJuly 31, 2026 19:49
@Astro-Han
Astro-Han merged commit 223a987 into mainAug 1, 2026
6 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han