feat(cu): add fenced OpenAI computer observation loop

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.

[hqhq1025]

Summary

Adds a provider-native OpenAI Responses computer-call codec, bounded transport, and pure loop as an observation-only experimental surface.

This is not the primary Maka Computer path. The primary product path remains the provider-neutral maka_computer function tool, where the model observes an AX tree and explicitly uses semantic element actions. The native OpenAI loop does not implicitly convert coordinates into Accessibility elements.

Safety boundary

• default native actions: screenshot, wait, and visual-only move;
• click, type, scroll, drag, and keypress are rejected before execution unless a future caller supplies an explicit policy backed by an isolated executor;
• the whole action batch is validated before its first side effect;
• pending safety checks require explicit acknowledgement;
• parallel computer calls fail closed;
• failed/incomplete Responses are not treated as completion;
• continuation uses the exact response and call IDs with a screenshot output.

Protocol hardening

• strict GA actions[] and preview action dialects remain separate;
• stable safety instructions stay outside user input;
• parallel_tool_calls is disabled;
• Responses bodies are capped at 16 MiB;
• HTTP error details are bounded and redacted;
• malformed success JSON does not echo response content;
• OpenAI strict-object adapter emits all properties required+nullable, rejects unknown/accessor keys, and projects known fields per action without weakening the core parser.

Real evidence

Using the local Azure Responses bridge and gpt-5.6-sol, the GA native computer tool returned only screenshot under the default policy. The new transport/codec/loop completed the screenshot continuation in two turns and executed no physical input action.

Verification

• Runtime full suite: 1530 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused action/codec/loop/policy/transport/strict-adapter tests pass.
• No Desktop, Computer Use executor, or provider credential files are modified.

[hqhq1025]

@Astro-Han#920 is clean and all CI checks pass. It is deliberately observation-only by default: real gpt-5.6-sol via the Azure Responses bridge completed a two-turn native computer screenshot continuation with no physical input action. The primary Accessibility-first product model path is isolated in fork stacked PR hqhq1025#9 and will be opened upstream after #920 merges.

[hqhq1025]

Ready for review. This remains an observation-only native OpenAI surface; the primary product mutation path is the AX-first provider-neutral function tool continued in #925. cc @Astro-Han

[Astro-Han]

Approved. I did not find a P0 or P1. The loop is still an explicit experimental API rather than the default product path, so these are non-blocking P2/P3 findings:

• P2 — packages/runtime/src/openai-computer-codec.ts:49-57,150-214 and openai-responses-transport.ts:41-46: requests do not include the Responses API store body field, so desktop screenshots inherit the API's default retention. The transport test's store=false query parameter does not set the request-body option. Make the storage choice explicit, or use a continuation that does not depend on a stored response.

• P2 — packages/runtime/src/openai-computer-loop.ts:84-101: top-level in_progress and per-call in_progress/incomplete states are accepted as actionable. An empty in_progress response is returned as completed, while an incomplete call's actions are executed. Only terminal completed responses and calls should advance the loop.

• P2 — packages/runtime/src/openai-computer-loop.ts:24-48,157-164: once a batch starts, the result type cannot represent partial delivery or an unknown outcome. If action two throws after action one succeeded, the caller receives only the raw exception and can retry the already-delivered action.

• P2 — packages/runtime/src/openai-computer-loop.ts:157-176: the final allowed turn still executes its actions and captures a screenshot, then throws max_turns_exceeded without sending that screenshot or reporting the delivered actions. Check the continuation budget before dispatch, or return the delivery state.

• P2 — packages/runtime/src/openai-computer-codec.ts:73-76 and openai-computer-loop.ts:74-78: maxTurns limits responses, not actions, and it accepts Infinity. One response can contain an unbounded actions[]; even the default-allowed wait action consumes two seconds per item. Add finite turn and per-batch action limits.

• P2 — packages/runtime/src/openai-computer-actions.ts:78-83,108-112: the default policy calls this path observation-only but allows move, which is converted to a raw host coordinate and sent to an executor with no isolated-target or display-identity proof. A normal host executor can therefore move the user's real pointer without allowAction.

• P3 — packages/runtime/src/openai-computer-actions.ts:108-110 and openai-computer-loop.ts:157-164: a provider screenshot action is sent through the executor, then the screenshot provider is called unconditionally. That turns one requested capture into two capture paths while discarding the executor's result.

• P3 — packages/runtime/src/openai-computer-loop.ts:74-81: when no signal is supplied, the loop creates an AbortController that never aborts. A stalled transport, executor, or capture can hang forever despite maxTurns; add a default deadline or make the required caller timeout explicit.

[hqhq1025]

Superseded by #985, which consolidates #920 and #925 on current main and includes the outstanding review fixes.