feat(cu): make provider function loops AX-first

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.

[hqhq1025]

Summary

Makes the provider-neutral maka_computer function path explicitly Accessibility-first and validates the same bounded semantic loop across supported provider surfaces.

This PR is stacked on #920. Until #920 merges, GitHub shows the cumulative native observation-loop commits plus this PR's final commit. After #920 merges, the review diff reduces to b3a34c8c.

AX-first product contract

The primary mutation path is:

model -> list_apps / observe -> AX element_id -> click_element / set_value
-> strict fresh refetch -> cua-driver AX action -> fresh observation

The model-facing description now tells providers to prefer click_element and set_value from the immediately preceding observation. It no longer advertises coordinate input as an equivalent background-safe path.

Coordinate click, scroll, drag, key, type, and pixel fallback remain in the provider-neutral grammar for compatibility, but the current product policy rejects them with unsupported_action because compatibility event delivery can interfere with the user's physical input. A model can recover by re-observing and choosing a semantic element action.

This matches the recovered Codex architecture: semantic element clicks prefer AXPick/AXPress; synthetic per-PID input is an executor fallback, not the model's default targeting path. OpenAI native computer_call remains the observation-only experimental surface from #920, not the primary mutation route.

Provider coverage

• OpenAI Responses function tools
• Anthropic Messages function tools
• Claude subscription transport
• Kimi Coding Plan Anthropic-compatible SSE
• MiniMax Coding Plan Anthropic-compatible SSE

Kimi and MiniMax coverage is hermetic protocol evidence, not a claim of live credentials on this machine.

Model-loop coverage

• list_apps -> observe -> set_value -> fresh observation -> final
• dynamic observation IDs are consumed rather than guessed
• a coordinate attempt fails closed, then the model recovers through a fresh semantic plan
• exact provider URL/auth/header and SSE tool-use/tool-result contracts
• Runtime path uses getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer

Verification

• focused Runtime model/provider tests: 5/5
• script contract tests: 4/4
• prior full Runtime run: 1535 pass, 7 skip
• prior full build and typecheck pass
• no Computer Use executor files are modified

Depends on #920.

cc @Astro-Han

[Astro-Han]

Approved for the #925 delta. I did not find a P0 or P1. This stacked head still contains #920 and now needs a restack onto current main; the findings below are specific to #925 and are non-blocking:

• P2 — scripts/cu-synthetic-model-scenario.mjs:25-41,64-100: the synthetic fixture never records that an observation was issued or consumed, and every observation reuses obs-fixture. A standalone provider loop can call set_value first with that guessed constant and pass without list_apps or observe, so the harness does not prove that IDs came from the latest observation. Issue a fresh observation ID and validate an active lease.

• P2 — packages/runtime/src/computer-use-tools.ts:1039-1048: the model-facing description says coordinate clicks always fail with unsupported_action, but the retained exact Electron page path still accepts left_click with a coordinate and returns semantic-background before the compatibility fallback guard. The wording hides a supported safe path; say that compatibility/pixel fallback is disabled while the exact Electron semantic path remains available.

• P3 — scripts/cu-real-runtime-model-e2e.mjs:32-73: the backend ignores the app/window selected by the model, substitutes the fixed fixture, replaces the action observation ID with obs-fixture, and captures the same fixed window. The full-Runtime E2E can therefore pass when target forwarding or observation mapping is wrong.

• P3 — packages/runtime/src/__tests__/computer-use-model-loop.test.ts:28-71,340-360: the cross-provider contract test changes providerType but always returns the same MockLanguageModelV3, bypassing getAIModel and every provider adapter. It checks the Runtime tool definition, not provider serialization; name the scope accordingly or exercise each adapter.

• P3 — scripts/cu-real-function-model-e2e.mjs:46-61: the OpenAI harness sends store: true, while docs/computer-use-model-loop-foundation.md:42-46 says these runs did not persist raw provider responses. The fixture is synthetic, but the documented retention boundary and request should agree.

• P3 — scripts/cu-synthetic-model-scenario.mjs:1-5,90-91: click_element is advertised in the function schema but always throws. A model that reasonably focuses the field before setting it makes the OpenAI run exit outside the typed tool-result path; implement the harmless fixture click or omit it from this schema.

[hqhq1025]

Superseded by #985, which restacks the combined provider-loop work on current main and includes the #920 foundation.