A macOS computer-use agent: multimodal screen understanding, structured LLM planning, and native UI actuation — with a real-time chat UI, safety guardrails, and optional hardware relay for cross-machine control research.
Chat UI → Orchestrator → ActionPlanner (LLM) → Executor
↑ ↑
ScreenPerception SafetyGuard + kill switch
(capture, OCR, AX tree)
- Eyes → Brain → Hands architecture with an explicit FSM (
IDLE → PLANNING → EXECUTING → VERIFYING) - Structured action plans (typed actions, not freeform shell)
- WebSocket chat UI at
http://127.0.0.1:8420 - Safety: kill switch, confirmation gates, coordinate bounds, rate limits
- 69 tests across foundation, perception, relay protocol, and comms
- Optional hardware relay: Pico 2W BLE HID bridge — advertises to the target machine as Apple Wireless Keyboard (combined mouse + keyboard) for standard HID pairing (see
usb-ble-mouse/)
Requires Python 3.11+ and macOS 14+.
python3 -m venv venv &&source venv/bin/activate
pip install -e .[dev]
# API key (Keychain — never commit keys)
security add-generic-password -s computer-control-agent -a gemini-api-key -w 'YOUR_KEY'# Or Vertex AI: set project_id in configs/default.yaml and use gcloud ADC# Permissions: Accessibility + Screen Recording for Terminal
bash scripts/setup_permissions.sh
# Run (local macOS mode — edit configs/default.yaml: relay_mode: local, capture.method: screen)
agent-ctl
open http://127.0.0.1:8420Kill switch: Cmd+Shift+Escape
Copy the example config and customize:
cp configs/default.yaml.example configs/default.yamlSee ARCHITECTURE.md for subsystem details and CLAUDE.md for developer commands.
Gemini (default), Claude, and OpenAI are supported. Keys are read from macOS Keychain (agent/config.py).
| Path | Role |
|---|---|
agent/eyes/ | Screen capture, OCR, accessibility tree |
agent/brain/ | Orchestrator, LLM client, action planner, safety |
agent/hands/ | Mouse, keyboard, apps, executor |
agent/chat/ | FastAPI + WebSocket UI |
web/ | Chat frontend |
usb-ble-mouse/ | Pico 2W BLE HID firmware + relay client |
pytest
ruff check- API keys in Keychain only
- Web UI binds to 127.0.0.1
- Screen data is sent to the configured cloud LLM for planning — review
ARCHITECTURE.mdbefore use on sensitive screens - Destructive actions require user confirmation in the chat UI
MIT — see LICENSE.