fix(cu): classify and gate semantic failures

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.

[hqhq1025]

Summary

Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:

• returned Computer Use failures such as { text, error: 'stale_frame' } were projected to text and incorrectly classified as successful tool invocations;
• a model could repeat the same ambiguity-rejected semantic target after every fresh observation because the generic loop signature included the changing observation_id.

Changes

• preserve typed Computer Use failure codes on action results;
• mark top-level returned error values as failed tool results for persistence, events, telemetry, and the generic loop gate;
• label only explicit ambiguous semantic refetch failures as ambiguous_target;
• cache a stable semantic signature of action + element identity/value while ignoring observation_id;
• block the next identical ambiguous semantic target before the backend runs;
• clear the ambiguity gate when the model changes target or action;
• keep user_intervened, target_missing, and ordinary stale/re-observe recovery uncached.

Real evidence

A live gpt-5.6-sol run through AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AX attempted a duplicated stale element. Before this fix, the stale_frame result was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.

Verification

• Runtime full suite: 1499 pass, 7 skip, 0 fail.
• Full repository build and typecheck pass.
• Focused loop-gate and Computer Use tests cover returned-error classification and ambiguity gating across changing observation IDs.
• No native executor files are modified.

[hqhq1025]

@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned target_changed before semantic refetch (zero model dispatch, zero mutation), while an earlier static duplicate-label scenario produced stale-frame behavior. Independently, real Computer Use returned failures were previously counted as telemetry success; this PR fixes that classification. The stable ambiguity gate is covered by focused tests and deliberately excludes user_intervened/target_missing recovery.

[Astro-Han]

Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.

P2

1. Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call observe before retrying. That call has no computerSemanticSignature, which clears lastAmbiguousComputerSignature at packages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction, ambiguous click(obs-1) -> observe -> same click(obs-2) reached the implementation all three times. The added test skips the required observe call.

   The cached key at lines 924-929 also contains only the action, frame-local element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.