fix(cu): bind Maka Computer actions to observed window captures

[hqhq1025]

Host-level Computer Use execution for macOS, keeping cua-driver as the sole native executor and binding every stateful action to the exact observed frame, window, display transform, and Electron page identity that grounded it.

P0 execution contract

A coordinate is never reinterpreted against the current highest-z window at dispatch time.

observe
-> frameId / epoch
-> bind normalized action to screenshot-time target
-> claim canonical fingerprint
-> validate exact live target
-> background dispatch
-> fresh postcondition observation
-> verify effect
-> invalidate old frame / advance epoch

Model-facing coordinate, type, key, scroll, drag, mouse-move, and zoom calls must echo the latest frame_id + frame_epoch. The runtime creates the action fingerprint before executor dispatch; replay is rejected.

Observation identity

Each screenshot produces a one-shot observation containing:

• unique frame ID and epoch;
• capture time and screenshot dimensions;
• display logical bounds, source-pixel atlas bounds, and scale;
• layer-0 window PID/window ID/app/title/bounds/z-order;
• privacy-safe SHA-256 AX content fingerprint;
• Electron cdpPort + pageTargetId + pageUrl + targetUrlContains;
• privacy-safe SHA-256 DOM freshness fingerprint.

Equal-scale multi-display screenshots support negative display origins. Mixed-scale or ambiguous screenshot atlases expose only a dimension-proven primary display; unproven displays are unavailable rather than guessed.

Bound action identity

Before entering cua-driver, runtime resolves:

• source screenshot coordinate;
• display logical coordinate;
• window-local coordinate;
• exact target window identity;
• exact display transform;
• exact Electron page identity when applicable;
• canonical action fingerprint.

Drag and zoom endpoints must bind to the same window.

Dispatch validation

Fresh state is used only to validate the bound target, never to choose another target.

Stable fail-closed errors cover:

• no_active_frame
• stale_frame
• stale_epoch
• target_missing
• target_changed
• target_occluded
• page_target_changed
• duplicate_action
• user_intervened

Validation rejects missing/reused/moved/resized windows, changed display transforms, occlusion, changed AX/DOM content, changed/non-unique page targets, stale epochs, and replay. None of these paths falls back to an unbound pixel or foreground action.

Keyboard ownership

Keyboard ownership is runtime-owned, not inferred from current system focus or backend history. It contains:

session + turn + frame/epoch + bound click target

Only a successful, verified, text-editable bound click followed by a fresh observation establishes ownership. It is revoked by new observation, failed/stale action, changed target/page, turn/session end, clearSession, or abort. Type validates the same bound window/page again and never activates it.

Effect verification and postconditions

Successful mutating actions consume the old frame. A fresh observation is mandatory before a new frame is issued. If a high-impact action cannot produce a fresh postcondition, it becomes capture_failed even if the underlying dispatch reported success.

Semantic success still requires observable effect:

• editable click: DOM focus;
• checkbox: checked-state change;
• button/double/triple click: downstream mutation;
• middle click: handled auxclick plus mutation;
• right click: consumed context menu or mutation;
• range drag: persisted value;
• native text: fresh AX value readback;
• Electron text: fresh DOM value readback.

supported:true, ok:false is terminal. Only side-effect-free supported:false may fall back once.

User concurrency and focus

• User pointer movement and frontmost-app changes are allowed by the monitor.
• If user activity invalidates the bound target, the action fails stale/occluded/user-intervened.
• Agent-caused real pointer movement remains a P0 failure.
• Agent focus stealing, temporary activation, foreground CGEvent fallback, and window-less desktop clicks remain prohibited.

Verification

• Core: 812/812.
• Runtime frame/tool focused tests: 32/32.
• Computer Use: 87/87.
• Script/coverage/sequence tests: 20/20.
• Desktop typecheck: passed.
• cua-driver bundle/provenance gate: passed.
• Added direct tests proving:
  • screenshot-time target dispatch;
  • decoy/occluder zero interaction;
  • missing/reused/moved window rejection;
  • display-transform rejection;
  • AX/DOM user-intervention rejection;
  • CDP target ID/URL change rejection;
  • duplicate action rejection across epoch advancement;
  • negative-origin and 1x/2x display transforms;
  • no pixel/foreground fallback on validation failure.
• A real E2E run proved the first frame-bound Electron click reached the exact captured window. Later reruns under a full-screen ChatGPT window stopped before input because all fixture points were occluded, which is the required fail-closed result.

Scope

This PR is execution-layer only. Provider/model harnesses and realRunEnabled scenario policy remain in the model-loop/UX PR and are not added here.

Remaining release gap

Production Electron packaging, Developer ID signing, notarization, and post-package verification remain separate release-pipeline work. The pinned cua-driver-rs-v0.7.1-maka.2 compatibility Mach-O is universal and ad-hoc signed.

[hqhq1025]

Execution-layer P0 found by the real OpenAI model-loop harness:

• The model correctly grounded the owned Electron fixture around (1240,681).
• While the user was working, ChatGPT covered that coordinate.
• cua-driver-backend.resolveWindowAt() re-resolved by highest z-order at action time, selected pid=94159,title=ChatGPT, and dispatched the pixel path (path=cgevent, verified=false).
• This disturbed the user's focus and repeated because the intended fixture never changed.

This belongs in #699, not the provider/model harness. Required execution contract, matching Codex/Sky window2:

1. Observation returns a stable target window identity plus screenshot/frame id.
2. Pointer actions carry that target identity/frame id.
3. Backend rehydrates the same pid + window_id and maps the coordinate into that window, regardless of occlusion.
4. If the target/window/frame is stale or missing, fail closed. Never reselect the highest-z window under the same global coordinate.
5. Add a regression with an owned target window covered by a second window: zero events to the covering window, no frontmost PID change, no real cursor movement.

Codex/Sky evidence: window2 click/drag/scroll accept Window {app,id} and optional screenshotId; screenshot IDs must remain cached for that target window.

The model-loop branch now has only a harness guard (target_occluded) to prevent further accidental clicks; no permanent backend workaround is being added there.

[hqhq1025]

#699 Window / Frame Binding 验收要求

P0 核心原则

模型基于哪一帧截图产生动作，动作就必须绑定那一帧以及那一帧中的目标窗口。

禁止在 dispatch 时重新按当前最高 z-order 窗口解释裸坐标。之前点击落到 ChatGPT，就是因为 observation 和 dispatch 之间重新选了窗口。

1. Observation 身份

每次截图必须生成不可复用的 observation：

interfaceCuaObservation{frameId: string;epoch: number;capturedAt: number;screenshotWidthPx: number;screenshotHeightPx: number;displays: DisplaySnapshot[];windows: WindowSnapshot[];}

每个窗口至少包含：

interfaceWindowIdentity{pid: number;windowId: number;bundleId?: string;bounds: Rect;zIndex: number;}

Electron/CDP 目标还需要绑定：

{cdpPort;pageTargetId;pageUrl;}

不能只依赖可重复的 window title。

2. Action 身份

模型动作在进入 executor 前必须转成：

interfaceBoundAction{frameId: string;epoch: number;target: WindowIdentity;sourceCoordinate: Point;windowCoordinate: Point;actionFingerprint: string;}

target 必须从生成截图时的 windows snapshot 解析，不能在执行时调用“重新找坐标下最高窗口”。

3. Dispatch 前验证

以下任一情况必须 fail closed：

• frameId 不是当前有效 frame；
• epoch 已变化；
• pid/windowId 不存在；
• windowId 被复用到其他窗口；
• 窗口移动、缩放、换屏导致 transform 变化；
• Retina/display scale 变化；
• 原目标被其他窗口遮挡；
• CDP page target 变化或不再唯一；
• 用户操作改变了目标状态；
• 同一 bound action 被重复提交。

建议稳定错误码：

no_active_frame
stale_frame
stale_epoch
target_missing
target_changed
target_occluded
page_target_changed
duplicate_action
user_intervened

任何错误都不能退回裸 pixel click。

4. 坐标空间

必须保存完整变换链：

model coordinate
-> source screenshot coordinate
-> display logical coordinate
-> window-local coordinate

要求覆盖：

• 1x/2x Retina；
• 负 display origin；
• 多显示器；
• 窗口跨屏；
• 截图裁剪；
• 窗口移动；
• drag 两端必须属于同一 bound window。

执行层接收的是已经绑定的 source/window 坐标，不能为了前端光标动画重新缩放。

5. 用户并发与焦点

用户正常使用电脑不应自动视为 agent 违规。

但需要区分：

• 用户移动鼠标、切换自己的前台窗口：允许；
• 用户行为导致目标 frame/window 失效：当前动作返回 stale，重新观察；
• agent 改变真实鼠标位置：P0 失败；
• agent 抢前台焦点：P0 失败；
• 为了坐标点击临时 activate 窗口：不允许；
• 后台投递失败时，不允许自动降级为 foreground CGEvent。

6. Keyboard ownership

键盘目标只能来源于一次成功并验证过的 bound click：

sessionId + turnId + pid + windowId + frame/epoch

以下情况立即撤销 ownership：

• 新 observation；
• stale/failed click；
• 用户改变目标；
• window/page identity 变化；
• turn/session 结束；
• action abort。

type 不能依赖当前系统焦点，也不能主动激活目标窗口。

7. Action 生命周期

推荐流程：

observe
-> 创建 frameId/epoch
-> 模型输出 action
-> bind action
-> claim 去重
-> validate exact target
-> background dispatch
-> UI settle
-> fresh observation
-> verify effect
-> invalidate old frame/advance epoch

动作成功必须携带：

{
path: 'ax'|'cdp'|'cgevent';
effect: 'confirmed'|'unverifiable'|'suspected_noop';
verified: boolean;}

高影响动作没有 fresh postcondition，不得报告 verified:true。

8. 必须通过的测试

• L1 单窗口精确点击，禁止重复点击。
• L2 type/drag/scroll，全部绑定同一窗口。
• L3 两个相似窗口，decoy 必须零交互。
• L3 截图后替换窗口，旧 action 必须 stale_frame。
• L3 action 前加入 occluder，必须 target_occluded，不得点穿。
• 窗口在 observation 后移动，必须拒绝。
• 1x/2x、负 origin、多显示器坐标测试。
• 用户同时移动鼠标/切焦点，agent 不得移动真实光标或抢焦点。
• 同一 action 重放必须拒绝。
• CDP target 变化或不唯一必须拒绝。
• 失败路径必须证明没有调用 pixel/foreground fallback。

Done 标准

完成后才能把这些场景设置为 realRunEnabled: true：

l1-single-click
l2-multi-control
l3-two-window
l3-stale-window

并用真实 Maka 模型链跑，不只是 mock backend。

现有 CuaFrameState 已经提供 frameId/epoch/duplicate 状态机，可以复用；但关键是必须贯穿：

screenshot -> provider action -> executor -> postcondition

不能只停留在独立类型和单元测试里。

[Astro-Han]

Thanks for putting this together. The overall direction makes sense to me, especially binding actions to the exact observation instead of resolving the target again at dispatch time.

I spent some time tracing the production paths, and I’m finding this difficult to review confidently at the current size. The PR is about 22k added lines across 122 files, and it includes several fairly different changes: the core/runtime contract, cua-driver execution, Electron page targeting, desktop wiring, the cursor overlay, CLI support, provider adapters, OpenAI transport, and multiple E2E harnesses.

I also found a number of concrete issues while reading it. A few examples:

• the observed frame does not currently include or validate the AX/DOM freshness fingerprints described in the PR body, so content inside the same window can change while an old coordinate remains valid;
• semantic element matching can fall back from a changed token to role + label, which may accept a replacement control;
• type, wait, and standalone screenshot do not appear to participate consistently in frame invalidation;
• screenshot capture, Electron page resolution, validation, and final dispatch happen across separate operations, leaving some TOCTOU gaps;
• several overlay assumptions do not hold for semantic actions, secondary displays, or overlay load failures;
• the provider adapters and OpenAI backend appear to be separate work, and some of them are not connected to the current desktop path.

There are more cases in the same areas, but I’m not sure posting a very long list of inline comments on this PR would be useful. Several findings seem to come from the same missing ownership boundaries, so fixing them one at a time here may make the diff even harder to reason about.

Would you consider splitting this along the existing dependency seams?

1. Core types, permissions, runtime frame state, and the backend interface.
2. The macOS cua-driver backend, including window/page identity, freshness checks, dispatch, and keyboard ownership.
3. Desktop integration, binary packaging, lifecycle cleanup, and the required overlay behavior.
4. A small deterministic cross-layer E2E suite.

CLI host control, provider-specific adapters, and the OpenAI Responses loop could then be separate follow-ups. Unit tests should stay with the implementation they cover.

This is a suggestion, not a blocking request from me. Please push back if the dependency boundaries make a different split more practical. My main concern is that the current PR mixes enough independent behavior that it is hard to tell whether the safety contract actually holds across all layers. Smaller PRs would make it much easier to give each part a useful review.

---

中文补充

整体方向我是认可的，尤其是让动作绑定观察时的窗口和页面，而不是执行时再临时寻找目标。这条主线值得继续做。

目前比较实际的困难是，这个 PR 同时改了太多层。两万多行里既有 runtime 的帧状态，也有 cua-driver、Electron page 定位、桌面接线、overlay、CLI、provider adapter 和多套 E2E。每一部分单独看都需要不少上下文，放在一起以后，很难确认从观察、校验、派发到后置验证的安全约束是否真的闭环。

阅读过程中也确实发现了一些具体问题。例如，同一窗口内部内容变化后，旧坐标仍可能有效；AX token 变化后仍可能按角色和标签接受新的控件；type、wait、screenshot 与帧失效的关系不一致；Electron 截图、page identity 校验和最终派发之间也存在时间窗口。overlay 在语义动作、副屏和加载失败等情况下，也不一定能准确反映真正执行的动作。

这些问题分布在 observation、backend validation、dispatch、keyboard ownership 和 overlay 几个环节，逐条修补很容易顾此失彼。继续在当前 PR 里改下去，review 范围可能还会进一步扩大。

所以想建议按依赖边界拆一下：

1. 先提交 core/runtime contract，包括类型、权限、frame/epoch 状态和 backend interface。
2. 再提交 macOS cua-driver backend，把窗口、页面、内容 freshness、派发和 keyboard ownership 放在同一个审查范围内。
3. 然后接 desktop integration，包括 binary、生命周期和 overlay。
4. 最后补一个小而确定的跨层 E2E，验证代表性的 observe、action、fresh observation 流程和关键失败路径。

CLI host control、provider adapter 和 OpenAI Responses loop 可以各自作为后续工作。它们和当前 native execution 主线有关，但不是证明这条主线正确所必需的部分。

这不是阻塞要求，只是基于目前 review 体验给出的建议。如果实际依赖关系更适合另一种拆法，也欢迎直接 push back。主要希望每个 PR 都能形成一个可以单独理解、验证和回滚的完整意图，这样大家更容易给出有效 review。

[hqhq1025]

@astrohan This implementation has been superseded by the rebuilt, review-split Computer Use stack #857 -> #892 -> #893 -> #894 -> #895 -> #896 -> #897 -> #898. Closing this older execution PR to keep review on one canonical implementation.