FlaUI-MCP

An MCP (Model Context Protocol) server that enables AI agents to automate Windows desktop applications using accessibility APIs - the same way Playwright automates browsers.

Why This Exists

When Playwright's MCP server automates browsers, it provides:

• browser_snapshot → Structured accessibility tree with element refs
• browser_click ref="..." → Click by ref, not coordinates

FlaUI-MCP brings the same pattern to Windows desktop apps:

• windows_snapshot → Accessibility tree with refs like w1e5
• windows_click ref="w1e5" → Click element by ref

No screenshot parsing. No coordinate guessing. Just semantic element references.

Quick Demo

Agent: Calculate 3 × 3
1. windows_launch { "app": "calc.exe" }
→ Window handle: w1
2. windows_snapshot { "handle": "w1" }
→ - window "Calculator" [ref=w1]
- button "Three" [ref=w1e43]
- button "Multiply by" [ref=w1e35]
- button "Equals" [ref=w1e38]
- text "Display is 0" [ref=w1e15]
3. windows_batch { "actions": [
{"action": "click", "ref": "w1e43"},
{"action": "click", "ref": "w1e35"},
{"action": "click", "ref": "w1e43"},
{"action": "click", "ref": "w1e38"},
{"action": "snapshot", "handle": "w1"}
]}
→ 1. click: Invoked Three
2. click: Invoked Multiply by
3. click: Invoked Three
4. click: Invoked Equals
5. snapshot: ... "Display is 9" ...

Installation

Prerequisites

• Windows 10/11
• .NET 8.0 Runtime

Download Release

Download the latest release from Releases and extract to a folder.

Choose the ZIP that matches your machine:

Asset	Use when
FlaUI-MCP-win-x64-*-self-contained.zip	64-bit Windows, no .NET runtime required
FlaUI-MCP-win-x64-*.zip	64-bit Windows with .NET 8 Runtime already installed
FlaUI-MCP-win-arm64-*-self-contained.zip	Windows on ARM64, no .NET runtime required
FlaUI-MCP-win-arm64-*.zip	Windows on ARM64 with .NET 8 Runtime already installed

Configure MCP Client

Add to your MCP configuration (e.g., ~/.copilot/mcp-config.json):

{
"mcpServers": {
"windows": {
"type": "local",
"command": "C:\\path\\to\\FlaUI-MCP.exe",
"tools": ["*"]
}
}
}

Or using dotnet run:

{
"mcpServers": {
"windows": {
"type": "local",
"command": "dotnet",
"args": ["run", "--project", "C:\\path\\to\\src\\FlaUI.Mcp"]
}
}
}

Available Tools

Tool	Description
windows_launch	Launch a Windows application
windows_snapshot	Get accessibility tree with element refs
windows_click	Click an element by ref
windows_type	Type text into an element
windows_send_keys	Send key presses or key chords (for example Ctrl+A)
windows_fill	Clear and fill a text field
windows_get_text	Get text content of an element
windows_screenshot	Capture window/element as PNG
windows_list_windows	List all open windows
windows_focus	Bring a window to foreground
windows_close	Close a window
windows_batch	Execute multiple actions in one call

windows_screenshot supports an optional background: true argument when a window handle is provided. This uses native background capture when available and falls back to the normal screenshot path if Windows returns a blank frame. It can also save screenshots with savePath, which must be an absolute local .png path. Existing files are not replaced unless overwrite: true is set.

Tool calls have a 30-second timeout so a blocked UI Automation provider or modal dialog returns an actionable error instead of hanging the MCP server forever.

Keeping the Screen Awake

Long automation runs generate no keyboard or mouse input, so Windows would normally turn off the display and show the lock screen mid-run — which breaks screenshots and can freeze rendering. While tools are actively being called, FlaUI-MCP holds a Windows power availability request (the same signal video players and conferencing apps send) that keeps the display on and suppresses the idle lock. The request appears in powercfg /requests (run as admin) with the reason "FlaUI-MCP is driving Windows UI automation".

The request is released after 5 minutes without a tool call, so an idle MCP server does not keep your screen on. Configure via the FLAUI_MCP_KEEP_AWAKE_SECONDS environment variable: a positive value changes the idle period, 0 disables keep-awake entirely.

Note: this covers the common idle-lock paths (display timeout, screensaver, sleep). A domain group policy that enforces a hard machine inactivity limit ("Interactive logon: Machine inactivity limit") locks based on input idle time and is not suppressed by availability requests — no application can override that policy.

Restricting Which Apps Can Be Automated

By default, FlaUI-MCP can automate any application on the desktop — including File Explorer, browsers, and terminals. If the agent driving it is ever misled (for example by prompt injection), that is a large attack surface. Set the FLAUI_MCP_ALLOWED_APPS environment variable to restrict automation to specific apps:

{
"mcpServers": {
"windows": {
"type": "local",
"command": "C:\\path\\to\\FlaUI.Mcp.exe",
"env": {
"FLAUI_MCP_ALLOWED_APPS": "TabularEditor3"
}
}
}
}

The value is a semicolon- or comma-separated list of process names, matched case-insensitively, with or without .exe (full paths are reduced to their file name). When the variable is unset or empty, everything is allowed.

While the allowlist is active:

• windows_launch refuses to start non-allowed executables.
• Window handles are only ever issued for allowed processes, so every ref-based tool (snapshot, click, type, fill, get_text, screenshot by handle/ref) is automatically scoped to allowed apps. windows_list_windows lists only allowed apps' windows.
• Ref-less keyboard input (windows_send_keys / windows_type without a ref) verifies the foreground window belongs to an allowed process first, and the Windows key is rejected outright (it opens system UI like the Start menu and Win+R outside any allowlist).
• windows_screenshot refuses fullScreen capture and foreground-window capture of non-allowed apps, so other windows' content is not disclosed.

Scope honestly stated: matching is by process name, so this is a guard against a misdirected or prompt-injected agent driving unintended apps through this server — not a sandbox against a local attacker, and it does not restrict anything the agent can do through other tools (like a shell). Dialogs owned by the allowed process (including common file dialogs, which run in-process) keep working; apps it launches as separate processes (e.g. a browser for OAuth) are blocked unless also listed.

Tool Examples

Send a keyboard chord to a target element:

{
"ref": "w1e5",
"chord": "Ctrl+A"
}

Send a sequence of key presses or chords:

{
"keys": ["Ctrl+A", "Delete", "Enter"]
}

Capture a window using opt-in background capture:

{
"handle": "w1",
"background": true
}

Save a screenshot to disk without replacing existing files:

{
"handle": "w1",
"savePath": "C:\\Temp\\capture.png"
}

Replace an existing screenshot file explicitly:

{
"handle": "w1",
"savePath": "C:\\Temp\\capture.png",
"overwrite": true
}

Safety and Limitations

• Keyboard input is focus-dependent. When you use windows_send_keys, the tool focuses the supplied ref first when possible, but Windows still sends keys to the active keyboard focus.
• windows_screenshotbackground: true is only valid with a window handle. If native background capture returns a blank frame, FlaUI-MCP falls back to the normal capture path.
• savePath accepts absolute local .png paths only. UNC paths, device paths, non-PNG extensions, and existing files without overwrite: true are rejected.
• Desktop integration tests require an interactive Windows session because they launch real WinForms and WPF windows.
• A timeout error means the MCP request returned, but a blocked Windows UI Automation provider or modal dialog may still need to be dismissed before retrying the operation.

How It Works

The Accessibility Snapshot

When you call windows_snapshot, you get a structured text tree:

- window "Calculator" [ref=w1e1]
- group "Number pad" [ref=w1e39]
- button "Seven" [ref=w1e47]
- button "Eight" [ref=w1e48]
- button "Nine" [ref=w1e49]
- text "Display is 0" [ref=w1e15]

This comes from Windows UI Automation - the same API screen readers use. Each element has:

• Role (button, text, group, textbox)
• Name ("Seven", "Display is 0")
• Ref (w1e47) - a handle for interaction
• State ([disabled], [readonly], [checked])

Why Not Screenshots?

Approach	Pros	Cons
Accessibility Tree	Semantic, precise, fast, works at any resolution	Requires UI Automation support
Screenshot + Vision	Works with any app	Slow, expensive, imprecise, resolution-dependent

FlaUI-MCP uses accessibility because it's what screen readers use - it's designed for programmatic UI interaction.

Modal Dialogs

UI Automation pattern calls (Invoke, Toggle, Select) are synchronous cross-process calls. When a click handler opens a modal dialog (ShowDialog() in WinForms/WPF), the handler — and therefore the pattern call and the app's entire UIA provider — stays blocked until the dialog closes.

FlaUI-MCP handles this instead of hanging:

• windows_click runs the pattern call on a background thread and watches the app's top-level windows via non-blocking Win32 APIs. If a modal appears, the tool returns immediately with the dialog's title.
• While the call is pending, UIA-based tools targeting that app (windows_snapshot, windows_get_text, ref-based typing/clicking) fail fast with guidance instead of timing out.
• Tools that don't need UIA keep working throughout: windows_screenshot, windows_send_keys / windows_typewithout a ref (pure keyboard input), windows_list_windows, windows_focus, and windows_close.

Typical flow: click a button → "modal dialog opened" → screenshot to see it → send keys (e.g. Enter or Tab+Enter) to dismiss it → snapshot works again.

Building from Source

# Clone
git clone https://github.com/shanselman/FlaUI-MCP.git
cd FlaUI-MCP
# Build
dotnet build src/FlaUI.Mcp
# Run
dotnet run --project src/FlaUI.Mcp

Testing

# Unit tests
dotnet test tests\FlaUI.Mcp.Tests
# Desktop integration tests; requires an interactive Windows session
dotnet test tests\FlaUI.Mcp.IntegrationTests

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ AI Agent (GitHub Copilot, Claude, etc.) │
│ - Calls MCP tools: windows_snapshot, windows_click, etc. │
└─────────────────────────────────────────────────────────────────┘
│ MCP Protocol (JSON-RPC over stdio)
▼
┌─────────────────────────────────────────────────────────────────┐
│ FlaUI-MCP Server (.NET 8) │
│ - Implements MCP tool handlers │
│ - Builds agent-friendly accessibility snapshots │
│ - Maps element refs ↔ AutomationElements │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ FlaUI Library (github.com/FlaUI/FlaUI) │
│ - UIA3Automation for modern apps (WPF, UWP, Win32) │
│ - Control patterns: Invoke, Value, Toggle, Selection │
│ - Tree walking and element discovery │
└─────────────────────────────────────────────────────────────────┘

Supported Applications

Works with any Windows application that supports UI Automation:

• ✅ Win32 apps (Notepad, Explorer, etc.)
• ✅ WPF applications
• ✅ WinForms applications
• ✅ UWP/Store apps (Calculator, Settings, etc.)
• ⚠️ Electron apps (partial - depends on accessibility implementation)
• ❌ Games (typically no UI Automation support)

Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Acknowledgments

• FlaUI - The excellent .NET UI Automation library this project is built on
• Playwright - Inspiration for the snapshot/ref interaction model
• Model Context Protocol - The protocol that makes this possible