feat(computer-use): Windows UIAutomation backend by Tryanks · Pull Request #268 · Tryanks/tcode

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks · 2026-08-24T14:24:33Z

Implements the last remaining track of #123 — the Windows (UIAutomation) backend for computer-use-mcp. The macOS backend and raw-image fallback already shipped.

Structure (mirrors backend/macos/): backend/windows/{mod,uia,input,capture,pure}.rs.

uia.rs — RAII COM init (balanced CoUninitialize; RPC_E_CHANGED_MODE accepted as an existing usable apartment), EnumWindows z-order root enumeration → ElementFromHandle, UIA tree walk into the sharedoutline::UiNode builder (ref/diff identity and the tools layer work unchanged), pattern lookup (Invoke/Toggle/ExpandCollapse/SelectionItem/Value/RangeValue/Scroll/ScrollItem/Legacy).
input.rs — SendInput: press/click (UIA Invoke first, physical left-click at target-frame center as fallback), set_text (focus + Ctrl+A + unicode), type_text/keypress (UTF-16 KEYEVENTF_UNICODE, surrogate pairs), scroll/drag/move.
capture.rs — PrintWindow(PW_RENDERFULLCONTENT) + BitBlt fallback → RGBA → PNG (already-resolved png 0.18.1), same text-sparse seam as macOS.
pure.rs — control-type→role, pattern→action, root-kind, VK map, rect→Frame: pure fns tested on every platform (mounted via #[path] so macOS/Linux CI runs them).
Filter matching hoisted to a shared matches_root_filters (deletes the macOS-private copy).

Dispatch gains Windows arms; macOS is untouched; Linux keeps unsupported. Every HRESULT/Win32 failure is a typed BackendError — no panics in the user's session. bundle_id = process executable filename (no Windows analog; documented).

tcode --cu-smoke — new diagnostic entry (parallels --preview-smoke): enumerates roots + observes the first, prints a summary. Verified live on macOS (enumerated 4 real windows, PASS). This is how the backend gets validated on real hardware.

Verification I ran — native fmt / clippy -D warnings / full workspace tests; clippy/build/test --no-run on x86_64-pc-windows-gnu -D warnings; 24 pure tests; live macOS --cu-smoke.

Real-Windows smoke pending before merge — I'll run --cu-smoke on physical Windows 11 (root enumeration, live UIA observe, a click/type action) via a dev-build artifact and post results here, same as #149.

Part of #123

🤖 Generated with Claude Code

Implements the Windows track of #123. backend/windows/ mirrors the macOS backend: uia.rs (COM-init RAII, EnumWindows z-order root enumeration, UIA tree walk into the shared outline::UiNode, pattern lookup), input.rs (SendInput keyboard/unicode/mouse/wheel/drag), capture.rs (PrintWindow+BitBlt → PNG for the raw-image fallback), pure.rs (control-type→role, pattern→action, VK, rect mappings — pure fns tested on every platform via #[path]). Filter matching is hoisted to a shared matches_root_filters used by both backends. Dispatch adds Windows arms; macOS untouched; Linux keeps unsupported. HRESULT/Win32 failures become typed BackendErrors — no panics in the user session. Adds : enumerates roots and observes the first, prints a summary — the on-device validation entry (parallels --preview-smoke / --cu-permissions). Verified live on macOS. Part of #123 (CDP roots remain the sole tool-surface deviation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observing only the first root landed on a game window (sparse tree by design); observing all roots surfaces the rich-tree cases too. Still performs no action, so it cannot disturb a live desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

STA without a message pump made cross-process child navigation return nothing, collapsing every observed tree to just its root element (real hardware showed 1 node per window). UIA clients are meant to run in an MTA; switch CoInitializeEx to COINIT_MULTITHREADED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 80635d8.

…on crate Replaces ~1,900 lines of hand-rolled UIAutomation COM (which collapsed every observed tree to its root element on real hardware — an STA threading pitfall) with a thin adapter over uiautomation 0.25. The crate owns COM init (MTA), tree walking, control patterns, and input/screenshot. Kept: the platform-neutral outline vocabulary, depth/node caps, is_text_sparse + CapturePolicy seam, shared matches_root_filters, and the pure mapping tables (map.rs, tested on every platform). A minimal direct windows dep remains only for exe-name process lookup. Net -227 LOC; no unwrap/expect/panic in the adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

semantic:false is visual mode — it returns a synthetic single root node by design, so the harness never exercised the tree walk (every window showed 1 node, on macOS and Windows alike). semantic:true walks the real accessibility tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Tryanksand others added 8 commits August 24, 2026 22:24

tmp: cu-diag instrumentation (will revert)
80635d8

Revert "tmp: cu-diag instrumentation (will revert)"
25d67cb
This reverts commit 80635d8.

fix(computer-use): allow the shared root-filter helpers to be dead co…
0688765
…de on Linux matches_root_filters/contains_case_insensitive are only called by the macOS and Windows backends; Linux compiles neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Tryanks enabled auto-merge (squash) August 24, 2026 18:13

Tryanks merged commit 3d31b3a into mainAug 24, 2026
3 checks passed

Tryanks deleted the feat/123-windows-uia branch August 24, 2026 18:20

Tryanks mentioned this pull request Aug 24, 2026
Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123
Closed

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tryanks commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects