feat(computer-use): 2.0 background delivery, activation, overlay, harness

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

[Tryanks]

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

• Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
• Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
• Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
• Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
• Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
• Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
• Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
• Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
• Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

• Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
• Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

• macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
• The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.