desktop — a computer-use environment for agents

Most agents live in a chat box. desktop gives them an entire computer instead: a lightweight Linux desktop that runs entirely inside Docker — XFCE on Xvfb, served through noVNC — with an HTTP daemon + TypeScript SDK so any agent can drive it: shell, GUI apps, browser, keyboard, mouse, screen. Small, fast, disposable.

An agent, eve, ships in this repo as the proof. Give her a task in the web UI, then watch the desktop live in your browser while she works — every click, keystroke and window visible in real time. She narrates her work into the chat and can post desktop screenshots and a session recording right into the conversation.

🎬 Demo video: [add link] · Jump to Architecture or Quick start

Why

Building computer-use agents means choosing between bad options:

• Full VMs (proprietary sandboxes, cloud browsers) — heavy, slow, locked in.
• Automating your own machine — fragile and dangerous; one bad rm and it's your actual filesystem.
• API-only agents — never touch the GUIs where real work actually happens.

desktop is the middle path: a real Debian 13 + XFCE + Chromium desktop in a container capped at 2 CPU / 3 GB, booted with one command, reset with docker compose down, and driven over plain HTTP by anything that can call an API — Claude, GPT, Gemini, GLM, or a bash script. No vendor lock-in, no heavyweight runtime, nothing installed on your host but Docker.

Architecture

┌──────────────────── your machine ────────────────────┐
│ │
│ web UI (Next.js) ────── eve agent │
│ (chat + live (tools + loop) │
│ desktop pane) │
│ │ │
│ computer-use-sdk │
│ │ HTTP │
│ ▼ │
│ ┌──────────── Docker container ─────────────┐ │
│ │ daemon :8095 every call = 1 round-trip │ │
│ │ │ │ │
│ │ XFCE on Xvfb :99 · xdotool │ │
│ │ x11vnc → websockify → noVNC :6080 │ │
│ │ Chromium · xfce4-terminal · ImageMagick │ │
│ └───────────────────────────────────────────┘ │
│ │ │
│ ./workspace persists on host │
└──────────────────────────────────────────────────────┘

Three layers, each replaceable:

Layer	What it does	Swap it with
Container	Real GUI desktop: Xvfb → XFCE → x11vnc → noVNC	Any X11 desktop
Daemon (daemon/daemon.py)	Python-stdlib HTTP API over xdotool/ImageMagick/bash — shell quoting and process cleanup handled for you	Anything that speaks HTTP
SDK (sdk/)	Thin TS client: actions, cmd, create, kill, screenshot, pointer, live, frames	Raw fetch() calls

Because every control surface is one HTTP round-trip to a process already inside the container, calls cost milliseconds — versus ~150 ms+ for spawning docker exec per action. And each computer is just a port: run several containers on 8095, 8096, … and the SDK addresses them independently.

Quick start

docker compose up -d --build

Wait for healthy, then open http://localhost:6080/vnc.html for the desktop.

• 6080 — noVNC desktop · 8095 — HTTP daemon
• ./workspace persists; docker compose down && docker compose up -d resets the rest

Drive it from code

cd sdk && npm install && npm run build && node smoke.mjs

import{computer}from"computer-use-sdk";awaitcomputer.cmd("pgrep -f xfce4-session");// shell — the escape hatchawaitcomputer.create("chromium https://example.com",{title: "web-1"});// a whole input sequence in one round-trip, run inside the containerawaitcomputer.actions([{do: "wait_for",window: "Chrome for Testing",timeoutMs: 45_000},{do: "focus",window: "Chrome for Testing"},{do: "key",keys: "ctrl+l"},{do: "paste",text: "https://news.ycombinator.com"},{do: "key",keys: "Return"},]);awaitcomputer.screenshot("state.png");// → workspace/state.pngconstfeed=awaitcomputer.live();// MJPEG stream, humans + agentsforawait(constframeofcomputer.frames({fps: 4})){/* vision input */}awaitcomputer.kill("web-1");

Full API in sdk/README.md.

eve — the agent that proves it

web/ contains a complete operator agent built on this stack:

• 8 tools, not 20: computer (a whole batched input sequence per call), observe, app, cmd, share, plus todo / ask_question / load_skill. Situational knowledge lives in load-on-demand skills (web/agent/skills/) rather than in the system prompt, so a browser task pays for browser tips and a git task doesn't.

• Works without vision: observe returns window/focus/pointer state as text and, for vision-capable models, the screen pixels — so text-only models stay effective and AI_MODEL is the only switch.

• Watchable by design: the web UI shows the live desktop next to the chat, with the agent's own pointer drawn as an amber ring (Xvfb has no hardware cursor, so the overlay polls /api/pointer). A recording hook captures frames through the whole session.

• Quiet transcript: tool calls are not rendered at all — the live desktop is the progress indicator. What lands in the chat is the agent's reasoning, any media it deliberately shared, and one closing summary.

• Media in the chat: share drops evidence straight into the conversation — an inline screenshot of the current desktop, or the captured frames compiled into a timelapse you can watch (or download) right there. The timelapse also compiles automatically at the end of every turn. Recordings are animated GIFs built with the ImageMagick already in the image, so the container stays ffmpeg-free.

  fix the login bug on github.com/me/myrepo
  → clone → reproduce → fix → test → commit → open PR
  → summary + PR link + screenshots + session recording, all in the chat
  

Setup: no extra credentials needed — everything runs locally.

Layout

├── Dockerfile / docker-compose.yml / entrypoint.sh
├── daemon/daemon.py # in-container HTTP API (port 8095)
├── sdk/ # TypeScript SDK (computer-use-sdk)
├── web/ # Next.js app + eve agent + chat media (screenshots, recordings)
├── guide.md # xdotool automation playbook
└── workspace/ # shared with the container, persists between runs

Tips

• Resolution: RESOLUTION=1600x900 in docker-compose.yml
• Chromium is the memory hog — raise mem_limit if needed
• DISPLAY=:99 for manual docker exec GUI commands
• docker compose logs -f to watch boot (Xvfb → XFCE → x11vnc → websockify)