CodeMode: harden boundaries and measure its TB2.1 impact

[Astro-Han]

Goal

CodeMode is a retained Maka capability. Harden its implementation and measure whether enabling it explains part of the existing Maka–Codex gap on Terminal-Bench 2.1.

Delivery

Ship independent, reviewable PRs:

1. CodeMode evaluator hardening

   • Reject source that escapes the cell wrapper.
   • Enforce result and tool-output limits during traversal, before materialization.
   • Replace boundary tests that currently fail before reaching the behavior they claim to cover.

2. Runtime integration hardening

   • Exclude provider-native tools from nested CodeMode execution.
   • Exclude hidden nested events from provider-context token estimates.
   • Remove ineffective tests.

3. TB2.1 Direct vs CodeMode evaluation

   • Extend the existing frozen 89-task Harbor harness.
   • Run paired, interleaved Maka Direct and Maka CodeMode arms.
   • Hold model, prompt, editing tools, executor, task order, and limits constant.
   • Report verifier pass rate, paired outcomes, provider steps, tool calls, actual tokens, wall time, and CodeMode failure categories.
   • Compare the measured CodeMode delta with the existing Maka–Codex gap; do not add Codex as a third arm unless the prior result is not comparable.

PRs 1 and 2 can proceed in parallel. PR 3 should remain implementation-independent, with final results rerun after the hardening PRs land.

Follow-up decision

CodeMode will not be removed. The evaluation informs its rollout and whether a separate PR should replace only the evaluator with @ai-sdk/code-mode QuickJS.

Any replacement must preserve Maka ToolRuntime permissions, suspension, abort, durability, recovery, hidden replay, and Desktop/CLI/Headless packaging.

[Astro-Han]

I’m working on PR 1: CodeMode evaluator hardening, scoped strictly to packages/code-mode. I’ll cover cell-wrapper escape rejection, early result/tool-output limits during traversal, and replacement or removal of ineffective boundary tests, using TDD. I’ll keep runtime integration, the TB2.1 harness, TypeScript/Acorn changes, and any AI SDK replacement out of scope.

[Astro-Han]

Closing with the original evaluation plan intentionally revised, not as a claim that the proposed 89-task run was completed.

Delivered hardening:

• fix(runtime): harden CodeMode integration #2467 hardened Runtime integration by excluding provider-native tools from nested execution and hidden nested events from provider-context estimates.
• fix(code-mode): harden evaluator boundaries #2468 hardened the original evaluator boundaries.
• feat(code-mode): replace Self with QuickJS #2549 subsequently replaced that evaluator with the resource-bounded QuickJS executor and removed the superseded interpreter.

The completed 10-task, single-repetition diagnostic produced equal verifier outcomes for Direct, Self, and QuickJS: 7/10 each. It did not establish equivalence or non-inferiority. It did establish that automatic rollout was not justified: Direct used 212 provider steps, 239 tool calls, and 6.03M tokens, while QuickJS used 354 steps, 883 tool calls, and 12.30M tokens.

Decision:

• retain CodeMode as a supported, default-off capability;
• do not automatically enable it for any model from this evidence;
• do not run the original 89-task expansion without a narrower rollout hypothesis;
• move the remaining user-facing work to an explicit settings preference under GPT tool-protocol compatibility: ApplyPatch and Code Mode #1383.