docs(eval): record the four-arm Terminal-Bench 2.1 comparison by Astro-Han · Pull Request #2208 · apache/maka

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han · 2026-08-05T05:49:24Z

Summary

Records the four-arm Terminal-Bench 2.1 comparison of Maka, Codex, Claude Code, and Reasonix around one DeepSeek V4 Flash model, over all 89 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the harness is the only variable that moves.

Codex leads on pass@1 at 70/89 (78.65%) against Maka's 65/89 (73.03%), but the exact two-sided McNemar test puts that five-task gap at p = 0.359, and an independent earlier run over the same suite flips 19% of Codex's own tasks while moving its net score by 3. Maka records the lowest cost per pass at $0.03215, on 61% of Codex's input tokens, and the fastest median cell at 332.8 s.

The diagnostic decomposition is the part worth reading: on the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 percentage points of each other (Claude Code 90.00%, Codex 88.33%, Maka 86.67%). Claude Code has the highest conditional pass rate and the lowest headline score because it ran out of budget on 26.97% of the suite.

The report follows the structure of the earlier Maka vs OpenCode report: frozen setup, outcome accounting, integrity hashes, and artifact pointers. It adds exclusive outcomes, a positioning section stating what the run does and does not establish about Maka, and a next-steps section.

Refs #1970

Verification

Corrected in this branch: the setup table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, read from each task.toml: 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600, 750, 7200, and 12000 s. Every dependent statement is updated.
Per-task CSV regenerated from the harness cohort report's own passed / budget_exhausted / infra_failed fields rather than from a duration threshold, and cross-checked against the report body: 89 rows, per-arm totals match every number in the prose.
Failure decomposition recomputed per task from the committed CSV: of the 12 tasks Codex won alone, 8 are Maka verification failures and 4 are budget exhaustions; all 7 tasks Maka won alone are Codex verification failures.
Duration statistics computed from the controller WAL (353 task_completed events), and deadline-share ratios computed against each task's own [agent] timeout_sec.
Outcome accounting read from the attempts WAL: 362 admissions against 356 scheduled cells, six infrastructure retries, no cell admitted a third time.
Pairwise McNemar p-values computed as exact two-sided binomial probabilities over discordant pairs.
Integrity hashes recomputed with shasum -a 256 over the archived evidence and the committed CSV.
Docs-only change: no build, lint, or test surface is affected. Biome does not cover Markdown or CSV.

Review focus

Three things are stated in the report rather than resolved:

Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M vs 1.57M tokens) while agreeing for Maka. The cause is unidentified, so the report makes no cross-source total-volume claim.
Maka is the only arm carrying an external system prompt (default-headless) and the only arm running a tool-result prune policy. This is disclosed as a live harness asymmetry; the run has no prompt-off or prune-off control.
Reasonix contributes three unscored cells that survived the retry policy, so its comparisons rest on 86 shared tasks.

Four harnesses around one DeepSeek V4 Flash model over all 89 tasks, every arm inside the task container under the same budget and package mirror. Codex leads on pass@1 at 70/89 against Maka's 65/89, but the exact McNemar test puts that five-task gap at p = 0.359 -- only the Codex advantages over Claude Code and Reasonix reach significance. Maka records the lowest cost per pass. On the 60 tasks where no arm ran out of budget the top three invert and sit within 3.33 points of each other, so the headline spread tracks whether an arm finishes rather than whether it can solve the task.

…ext steps The Experiment table stated a flat 900 s per-cell budget. The run used the task-native agent timeout at x1, which is 900 s on 48 of the 89 tasks and ranges from 600 s to 12000 s elsewhere. Every statement resting on that number is corrected. Adds the frozen setup, outcome accounting, integrity hashes, and artifact pointers the earlier Maka vs OpenCode report carries, plus exclusive outcomes and two analysis sections. The failure decomposition behind them is new: of the 12 tasks Codex won alone, 8 are Maka verification failures and only 4 are deadline losses, so the larger half of the gap is answer correctness rather than unfinished work.

Astro-Han force-pushed the docs/eval-4arm-tbench-2.1-deepseek-v4-flash branch from b576e95 to 0fe3d48Compare August 5, 2026 06:44

Astro-Han force-pushed the docs/eval-4arm-tbench-2.1-deepseek-v4-flash branch from 0fe3d48 to 7b28ae3Compare August 5, 2026 07:00

Astro-Han mentioned this pull request Aug 5, 2026
eval(headless): benchmark Maka, Codex, and Claude Code synchronously on DeepSeek V4 Flash #1970
Closed

Astro-Han mentioned this pull request Aug 5, 2026
test(cli): waitFor's fixed 250 ms wall-clock deadline makes 373 TUI assertions flaky under CI load #2221
Closed

Astro-Han marked this pull request as ready for review August 5, 2026 08:52

Astro-Han merged commit 1aacc8d into mainAug 5, 2026
21 of 23 checks passed

Astro-Han mentioned this pull request Aug 8, 2026
eval(headless): benchmark Maka against Reasonix on DeepSeek V4 Flash #2076
Closed

hqhq1025 mentioned this pull request Aug 14, 2026
[codex] docs(eval): publish nine-arm Terminal-Bench 2.1 leaderboard #3004
Merged

tt-a1i mentioned this pull request Aug 20, 2026
setup 闸门缺一档：允许 agent 提议一轮 setup，由用户按键确认 openpi-dev/openpi#39
Open