feat(computer-use): restore focus and cursor around synthetic input; true per-window capture on Windows

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code

[Tryanks]

Summary

Mimics Codex computer-use input hygiene: when the target window is not frontmost, synthetic input actions instantly switch focus to it, act, and switch back; the user's cursor position is saved and restored after pointer actions. Windows screenshots now capture the window's own content instead of the screen region under it.

Behavior

• Focus guard (both platforms): clicks, keyboard input, scroll, and drag briefly activate + raise the target window when needed, then re-activate the previously frontmost app. Best-effort with a ≤500 ms activation poll — a guard failure never turns a working action into an error.
• Cursor guard (both platforms): pointer actions save the user's cursor position and warp it back afterwards. Explicit move_mouse is exempt so hover persists.
• No flicker on accessibility paths: successful AXPress / AXValue (macOS) and UIA pattern actions (Windows) acquire no guards; guards are acquired lazily only on synthetic-input fallbacks, after input validation.

Platform notes

• macOS: FocusGuard reads the frontmost pid via NSWorkspace, activates via NSRunningApplication activateWithOptions:, and raises the specific window via AXRaise + AXMain (reusing existing objc/AX helpers in ax.rs). CursorGuard restores via CGDisplay::warp_mouse_cursor_position (no synthetic events). Screenshots were already true per-window (screencapture -l), unchanged.
• Windows: ForegroundGuard uses SetForegroundWindow with the AttachThreadInput fallback (and SW_RESTORE for minimized windows); CursorGuard uses Get/SetCursorPos. Capture now prefers PrintWindow + PW_RENDERFULLCONTENT (32-bit top-down DIB → PNG via the image crate already in the tree), which renders the window's own content even when occluded or backgrounded, falling back to the existing UIA screen-region grab on failure. All GDI objects are released via RAII.

Known tradeoff

Restoring the previous app right after a click can dismiss transient UI (menus/popovers) the click just opened in the target app — inherent to the switch-back approach; unchanged from the Codex scheme being mirrored.

Testing

• cargo fmt --all --check ✓
• cargo clippy -p computer-use-mcp --all-targets -- -D warnings ✓ (host)
• cargo clippy -p computer-use-mcp --target x86_64-pc-windows-gnu --all-targets -- -D warnings ✓ (cross-compiles the real Windows backend)
• cargo test -p computer-use-mcp ✓ 25/25
• CI's windows-2022 job is the authoritative Windows gate.

🤖 Generated with Claude Code