[codex] docs(eval): publish nine-arm Terminal-Bench 2.1 leaderboard by hqhq1025 · Pull Request #3004 · apache/maka

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

hqhq1025 · 2026-08-14T09:06:07Z

Summary

Docs-only: publish the final nine-harness Terminal-Bench 2.1 comparison around DeepSeek V4 Flash at max reasoning effort, plus an 89-task accepted-outcome CSV.

This extends the four-arm report in #2208 with OpenCode, Kimi Code, ZCode, Pi, and DeepSeek Harness (DSH).

Full report:docs/eval/terminal-bench-2.1-deepseek-v4-flash-nine-arm.md

Leaderboard

Rank	Harness	Passed	Pass@1	Tokens	Avg/task	Cache	Cost
1	Codex	73/89	82.0%	259.62M	2.92M	98.6%	$2.105
2	Maka	69/89	77.5%	185.71M	2.09M	98.8%	$1.815
3	Pi	66/89	74.2%	131.40M	1.48M	98.5%	$1.303
4	DSH	65/89	73.0%	182.84M	2.05M	98.6%	$1.705
5	ZCode	63/89	70.8%	223.46M	2.51M	98.9%	$2.287
6	Reasonix	60/89	67.4%	216.17M	2.43M	99.0%	$1.935
7	OpenCode	58/89	65.2%	166.44M	1.87M	98.6%	$1.745
8	Kimi Code	53/89	59.6%	193.75M	2.18M	98.2%	$2.162
9	Claude Code	49/89	55.1%	247.60M	2.78M	98.9%	$2.633

All nine harnesses pass the same 28/89 tasks. Five tasks fail on all nine.

Scope and accounting

Terminal-Bench 2.1 revision d49e28f1e4ddd13d289e85a5f312a66750951932
89 tasks, one repetition, official verifier, task-native timeout ×1
deepseek-v4-flash, thinking enabled, reasoning effort max
web tools removed from the provider-visible surface; shell networking retained under benchmark-contamination egress filtering
the original eight arms ran as one cohort; DSH is a configuration-aligned later run and is disclosed as descriptive rather than simultaneously paired
recorded token, cache, and cost aggregates are reported directly without extrapolation

DSH recovery disclosure

DSH initially recorded 61/89. Real-machine diagnosis found three Eval-specific defects: a five-minute Bash timeout that interrupted dpkg, interactive tzdata setup, and background-service teardown before verifier execution. Commit 28ebe0949 fixed those paths.

Seven cells were rerun. Four changed from fail to pass (hf-model-inference, merge-diff-arc-agi-task, polyglot-c-py, regex-log), producing the final 65/89 result. Three remained genuine verifier failures.

Verification

CSV contains exactly 89 rows and 89 unique task ids
recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
recomputed agreement: 28 all-pass, 5 all-fail, 56 mixed
CSV SHA-256: c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a
git diff --check passed
secret-shaped token scan passed
docs-only change; repository code tests were not run

Review focus

whether the independent DSH run is disclosed strongly enough relative to the simultaneous eight-arm cohort
whether the accepted-attempt and recovery policy is clear
whether the recorded-economics convention is described without implying billing precision

Astro-Han

Pushed one commit to your branch: the test failure was just the missing ASF header on nine-arm.md. The audit passes now.

Also marked the PR ready for review — these numbers deserve to be published, and I want to cite them in the website positioning discussion (#4307).

简体中文

往你的分支推了一个提交：test 挂掉的唯一原因是 nine-arm.md 缺 ASF header，现在审计通过了。

顺手转成了正式 PR —— 这份数据值得发布，我想在 #4307 的网站定位讨论里引用它。

Astro-Han mentioned this pull request Aug 14, 2026
feat(eval): add the DeepSeek Harness benchmark arm #2971
Merged
4 tasks

M4n5ter force-pushed the codex/tb21-nine-arm-report branch from f5af587 to 40045ffCompare August 26, 2026 08:46

docs(eval): publish nine-arm Terminal-Bench results
63a861c

M4n5ter force-pushed the codex/tb21-nine-arm-report branch from 40045ff to 63a861cCompare August 26, 2026 10:03

github-actionsBot added the effort/M Under 500 readable lines label Aug 27, 2026

Astro-Han marked this pull request as ready for review September 1, 2026 02:08

chore(eval): add the missing ASF header to the nine-arm report
438c2f8

Astro-Han approved these changes Sep 1, 2026
View reviewed changes

Astro-Han merged commit 279c2e8 into apache:mainSep 1, 2026
1 check passed