feat(eval): compare three edit contracts on the DeepSeek Harness

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Astro-Han]

Summary

A Terminal-Bench score is attributed to a framework, which leaves the edit contract inside the framework and unmeasured. Comparing Maka's edit/write against the harness's str_replace_editor cannot separate them either — that comparison moves the framework and the contract at once.

These three arms hold everything else and move only the contract:

arm	contract
deepseek-harness	str_replace_editor — one tool, four commands, unique-literal match, all guidance in the tool description
deepseek-harness-fs	the harness's read/write/edit — three tools, snake_case, three prompt sections of their own
deepseek-harness-apply-patch	V4A apply_patch — one envelope, changes located by surrounding context

Same one-shot CLI, same toolchain and fingerprint, same dsh-fs-local provider under every editor, byte-identical package.json and cordis.yml. The variable is the whole tool family and not one dimension of it: tool count, argument names, path conventions and where the guidance lives move together, because each family ships that way. A result is attributable to the family as shipped, not to patch syntax as such.

No arm mounts dsh-fs-observation-policy. It was mounted for the fs arm alone on the reasoning that its tools' guidance mentions the policy; of the four presets in @deepseek-ai/dsh, three mount dsh-tool-fs and none mounts the policy, and its own documentation covers str_replace_editor too. Mounting it in one arm gave that arm two failure modes the others cannot have.

Three compositions in three files can drift apart while still producing numbers, so the control is asserted rather than intended. edit-contract-arms.test.ts fails when the compositions differ by any row outside their editors, when they compose in a different order, when the model, context window, reasoning effort, stream idle timeout, sandbox mode, persona or either Bash deadline drifts in one, when the arms stop sharing one toolchain identity object, when the experiment launches them with anything but a different profile argument, or when arms multiplied by concurrency leaves the 128-trial convention.

The patch arm is ours, and that is the thing to review

The harness ships no patch tool, so plugins/tool-apply-patch/ is this repository's while the other two arms run their vendor's. Fidelity to Codex is therefore itself an experimental variable, and it is asymmetric: a tool stricter than the reference costs its arm turns the reference never spends, and a safer one wins recoveries the reference never gets.

One deviation cannot be removed.apply_patch in Codex is a freeform tool — the API is sent a 108-character description plus a Lark grammar the decoder is constrained to, so a model on that path cannot emit a malformed envelope. The harness registers JSON function tools and has no grammar seam. This arm is therefore the unconstrained case, and syntax errors it can make are errors Codex's decoder would have prevented. That cost is charged to this arm and to no other, and it leads the arm's description in the README.

What is not charged to it is the absence of a read tool. Codex registers no handler that reads file text either — its handler set is long, and the only entry taking a filesystem path to read is view_image, which takes images — so an arm that reads with cat is that contract, and every arm has the same bash at the same 16000-character cap.

What the model is told is upstream's own prose for exactly that case: the ## apply_patch section of prompt_with_apply_patch_instructions.md, carried verbatim as upstream-apply-patch-instructions.md with three substitutions applied over it. A test pins the file's digest, requires each substitution to apply exactly once, and requires the built description to differ from the copy nowhere else. That file is upstream's and it is not delivered from where it sits: at the ported commit it is referenced only from a test whose four model cases all declare expects_apply_patch_description: false. Seven of the eight instruction templates in models-manager/models.json carry no ## apply_patch section and leave the format to the grammar; the eighth, gpt-5.2, carries a shorter one with the same envelope and worked example. So the format is still taught in prose upstream and this is the fuller version of it. Either way no shipped configuration is the one this arm is in — gpt-5.2 gets prose and grammar — so the choice was between upstream's own text and text written here.

Three further divergences are known and each is pinned to bytes the reference produces:

• the tool implements the function-tool path — verify the whole envelope, then re-apply it from the raw text — where the standalone binary applies sequentially with no verification;
• it runs NormalizeToLf, the shipped default, where PreserveLineEndings is Stage::UnderDevelopment, default_enabled: false upstream;
• its diagnostics differ from Codex's, which is the one place this arm may be easier on a model. This one is recorded in prose only; the other two are named with their causes in the test files, which also assert that each still diverges.

A third divergence was recorded here and has since been removed rather than documented: the provider's readText strips a leading byte-order mark and rejects a NUL byte, neither of which String::from_utf8 does. Reading through readBytes and decoding in the tool matches the reference exactly.

The grammar is implemented in full. *** Delete File: and *** Move to: are the only operations ctx.fs has no primitive for, so they run against ctx.fs.processPath(target); under a provider that confines, both refuse up front rather than escape. This arm mounts danger-full-access, where the refusal never fires.

Sources and licences are in the plugins' NOTICE. An earlier revision of this branch vendored the OpenAI Agents SDK's applyDiff as the hunk applier; it is a different V4A implementation, not Codex's, and nothing from it remains.

What the adversarial review rounds changed

Two rounds of four fresh-eye passes each — the port against the binary, the harness integration, the three-arm control, and an audit of every factual claim. The second round reviewed the first round's fixes, which is what caught the two entries below that begin "the previous fix". Every finding was re-verified against upstream source or the reference binary before being acted on.

Nine port defects, all now red-checked. The matcher used JavaScript's trim where upstream uses Rust's, so a file whose first line carried a byte-order mark matched a pattern without one and was written back with the mark deleted, and a line terminated by U+0085 was refused where the reference applies it. The verify pass stated a delete target instead of reading it, applying deletes of non-UTF-8 files that the reference refuses outright. It emitted fs/observed {present} off a bare stat, telling an observation policy a file had been seen that nothing read. The duplicate-path key was the provider's realpath-derived target key rather than upstream's lexical join, so an envelope naming both a symlink and its target was refused where the binary applies both. A move destination ran the fs/write-intent waterfall and discarded the answer, turning a decider's refusal to clobber into a silent overwrite. A failed unlink and an abort between passes escaped the seam's error taxonomy.

Five of those were invisible because the test provider was weaker than dsh-fs-local: it decoded lossily, called every non-directory a file, took three parameters so the signal and sandbox policy could not be observed, and never returned a decider's answer.

The previous fix read patch targets through the wrong seam.ctx.fs.readText is not read_file_text: upstream reads bytes and calls String::from_utf8, while dsh-fs-local additionally rejects a NUL in the first 8192 bytes and strips a leading byte-order mark. NUL is valid UTF-8 and the reference patches and deletes straight through it, so making the verify pass read delete targets — itself a correct fix — took *** Delete File: on such a file with it. Reading through readBytes and decoding in the tool is exactly String::from_utf8, and it removes the byte-order-mark divergence too. Separately, *** End of File as the first line of an update hunk was refused where upstream ignores it.

Twelve more mutations the suite could not see. Five dropped the turn signal from a provider call, which a fake that ignored the signal could not notice; two removed or falsified the version on fs/observed; two swapped a removal's announcement with the removal itself, which leaves the event list identical and the trail a lie; one bypassed the create decider, one renamed its event, one made the sandbox-denial remap a no-op. The fake now honours and records the signal on every call, records what the filesystem held when each observation was published, records the version writeText returned, and can be made to deny a write.

Three uncontrolled differences between the arms, of which one fix was itself wrong. streamIdleTimeoutMs was omitted from all three profiles, leaving a five-minute ceiling on the gap between two streamed tokens behind reasoningEffort: max and a 65-minute bash deadline; it now carries upstream's own 172800000. maxConcurrentTaskGroups: 64 made 192 concurrent trials where the saturating specs here land on 128; it is now 42.

readMaxLineLength was raised from its shipped 2000 to the budget to close a cap the baseline has no counterpart to, and that was worse than the asymmetry. The tool appends a ... (line truncated to N chars) marker and then charges the whole line against readMaxBytes, so a cap near the budget makes a long line not fit at all, drops every line after it, and returns a window no offset recovers — 201 lines at 2000 becomes 0 at 16000. No value equalises the two budgets anyway, one counting UTF-16 code units and the other UTF-8 bytes, so the cap is back where the tool ships it and the guard test asserts the invariant rather than the number.

Twenty-one false claims across the two rounds, listed in the two docs commits. The largest was the provenance one, twice: first that the vendored prose is what Codex sends, then that it is retired. The read-budget script had four transcription errors of its own, all inflating the character ratio, so it no longer asks to be trusted — --self-check <dir> lifts the real renderers out of an installed copy and asserts agreement, and the figures are 1.093x lines and 1.001x characters over 2197 files.

Verification

Lint, format and typecheck clean. 47 TypeScript and 200 plugin tests pass.

Fidelity is measured against the binary rather than argued from the Rust:

• codex-oracle.mjs runs the released codex under argv0 apply_patch over a temporary tree for each of 94 cases and records what it printed and what the tree became; codex-fixtures.test.mjs replays every case through the registered tool and needs no codex to do it.
• __tests__/upstream/scenarios/ carries Codex's own conformance suite verbatim — the 25 scenario directories upstream publishes as "meant to be easily portable to other languages or platforms" — and upstream-scenarios.test.mjs replays them upstream's way: copy the input tree, apply the patch, ignore the exit status, require the resulting tree to match exactly. 22 match outright; the three that do not are named divergences. These are the only inputs here this repository did not choose.
• codex-fuzz.mjs generates envelopes from random line soup, including near-miss context, and runs both. It earned its keep on this branch: it caught a regression introduced here, where rebuilding the file forward let a context-free insertion survive inside a range a later chunk replaces. 2300 envelopes across four seeds diverge on nothing.

Red-checked, not just green:

• Appending a line to the description, renaming the tool, editing one byte of the vendored instructions, or neutering a substitution each fails the contract-surface test.
• Removing any one of the four context-matching passes fails at least one oracle fixture; restoring String.prototype.trim in the loose passes fails the trailing-U+FEFF fixture; disabling the heredoc strip fails both heredoc fixtures; matching the heredoc close with includes instead of endsWith fails the trailing-text fixture.
• Each of the twenty-one port defects and unseen mutations above fails the suite when reintroduced. Seventeen of them did not before these two rounds.
• Both divergence lists assert that each entry still diverges, so an excuse cannot outlive the thing it excused.

End-to-end against the rebuilt linux/amd64 toolchain: loading the plugin from inside the built tree registers exactly one tool named apply_patch, with one required input string and a 2903-byte description carrying no mention of the shell delivery. The instructions file ships, so it is covered by checksums.sha256 and the re-pinned fingerprint.

Not run: an actual benchmark cohort. No dependency moved.

Review focus

One claim this repository does not reproduce, and which should be read as a one-time manual check rather than a standing guarantee: the three profiles were compared against the upstream examples/jsonrpc-agent composition during development — tool names, tool schemas, system prompt and message sequence on the first outbound request — and found identical.

The read budget matched at 16000 equalises the budget's number, not delivered content. scripts/measure-read-budget.mjs makes the residual checkable and --self-check makes the script itself checkable: over 2197 tracked sources the fs arm delivers 1.093x the lines and 1.001x the characters; among the 570 files where a budget binds, the extremes are 1.398x/1.168x and 0.837x/0.720x.

Two things a write-up must carry that this branch cannot fix. The fs arm receives 764 characters of tool-contributed system-prompt guidance where the other two receive none, and two of its three sections reference an fs-observation-policy no arm mounts. And the treatment is the whole tool family plus that guidance, not "the edit contract" — the arm names are shorter than the thing being varied.

experiments/.../edit-contracts.json declares repetitions: 1, so 89 tasks x 3 subjects is 267 cells. Cost, not power, chose that. McNemar at one repetition is weak for the effect sizes this is likely to see: on ~20% discordant pairs a true 5pp difference is about 2 sigma before any correction for three pairwise comparisons, so a write-up has to report discordant counts and intervals rather than three success rates.

Stacked on #2971.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1357f048-8ea6-474e-8887-d512c43407ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}