docs(eval): report the three-arm edit-contract comparison by Astro-Han · Pull Request #3158 · apache/maka

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

Astro-Han · 2026-08-17T09:11:18Z

Summary

Adds the report and per-task CSV for the three-arm edit-contract experiment on Terminal-Bench 2.1 with deepseek-v4-flash: one harness, one model, one composition, and the file-editing tool contract as the only variable (str_replace_editor baseline vs @deepseek-ai/dsh-tool-fs vs a repo-authored V4A apply_patch plugin).

Headline results, 86 scored tasks per arm:

str_replace_editor 56, apply_patch 56, fs 53 — and the run has no power to separate them. Exact McNemar p = 1.00 / 0.63 / 0.65 with 95% CIs of roughly ±11 pp; the minimum detectable effect is ~13 tasks and power against the observed spread is 7%. The report states this as a failure to detect, not a finding of equivalence.
The arms disagree on 29/86 tasks; with no same-arm repetition the run cannot decompose that into variance vs a real per-task effect.
Costs are computed at DeepSeek's current time-of-day pricing, apportioned peak/off-peak per attempt window: $3.15 / $3.18 / $3.56 per arm (93% off-peak). The harness's own stale flat table would have understated this ~2.4× (fixed separately in fix(eval): update DeepSeek V4 Flash pricing to current published rates #3153).
Four operational findings (glibc floor from native modules, CLI double--- parsing, fs.inotify.max_user_instances as a host-wide budget, verifier bandwidth as a scheduling constraint) with their fixes.

Verification

Docs-only. Numbers are derived from the archived run results (5 run directories, later runs superseding earlier cells); the analysis script cross-checks totals against the per-arm token columns. CSV records per-task per-arm outcome, status, seconds, cost, requests and source run.

AI use

Select exactly one:

No generative tool made a substantive contribution
Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code ran the experiment harness, performed the statistical analysis, and authored the report, CSV and this PR body. The commit carries a Generated-by trailer. A human contributor reviews the final text and owns the merge decision.

Checklist

Tests cover the change and fail without it — docs only
Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

Yes — described under Summary above
No

coderabbitai · 2026-08-17T09:11:37Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cb3df9a2-0c56-4757-aea2-220c1ca8e827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

One harness, one model, one composition, three file-editing plugins. The arms score 56, 56 and 53 of 86 tasks, and the run has no power to separate them: with 17 to 22 discordant pairs per comparison its minimum detectable effect is about 13 tasks, and against the differences actually observed — zero and three — its power is 3% to 7%. The report says that rather than reporting the three non-significant p-values as a finding of equivalence. The 95% intervals are the honest summary: each spans roughly 22 percentage points, so every pair is as consistent with no difference as with one arm being nine or ten tasks better. The arms disagree on 29 of 86 tasks. No arm was run against itself, so this data cannot say how much of that is variance and how much is a real per-task effect; a same-arm repetition is named as the cheapest experiment that would make the rest of it interpretable. Records what the arms differ in, none of it powered: fs nearly doubles the baseline's verification failures while losing fewer cells to the deadline, spends 30% more reasoning, and costs 20% more per pass. Records that the treatment is a tool family rather than a diff format — fs presents four tools and the least instruction text of the three. Four harness failures found and fixed during the run are written up as operational findings, because each is a prerequisite for reproducing it: the toolchain's base image is a glibc floor for the task images, a task instruction beginning with a dash needs two argument separators, inotify instances are a host-wide budget that caps concurrency, and verifier bandwidth is a scheduling constraint for tasks that build an environment. Three tasks are unscored. The two torch tasks are reported as unscored rather than as zeros: an agent that exhausted its budget may still have left a passing state behind, and the verifier that would have said so never ran. Generated-by: Claude Code

M4n5ter force-pushed the docs/eval-edit-contract-arms-report branch from 20f4778 to b8a83deCompare August 26, 2026 10:03

github-actionsBot added the effort/M Under 500 readable lines label Aug 27, 2026