feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(desktop): add cross-platform window capture - #8103

Open
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture
Open

feat(desktop): add cross-platform window capture#8103
Bil0000 wants to merge 196 commits into
pingdotgg:mainfrom
Bil0000:feat/window-capture

Conversation

@Bil0000

@Bil0000Bil0000 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Adds opt-in window capture for macOS, Windows, and Linux Wayland.
  • Uses Shift + Shift by default on macOS and Windows. Wayland requires a key chord because its portal does not support modifier-pair shortcuts.
  • Checks T3 Code keybindings, common system and app chords, and live OS registration before a normal key chord can be saved on supported desktop sessions.
  • Requests macOS Accessibility and Screen Recording only when the user turns Window Capture on and the permission is not already granted. Startup only reads permission state.
  • Attaches the image with app name and window title, plus the app icon and accessibility data when they are available.
  • Runs accessibility extraction in a short-lived helper, so a native provider crash or timeout falls back to screenshot-only instead of crashing or blocking T3 Code.
  • Adds a dedicated settings page, command palette entry, onboarding, platform-aware shortcut key caps, inline sound play controls, capture sound, high-density app icons, an early and gentler visual cue, compact image cards, and one-shot attachment animation.
  • Stores pending captures durably and preserves their metadata through drafts, uploads, timeline rendering, and provider input.

Why

Users can add visual and text context from another app without leaving their T3 Code workflow or managing screenshot files. Native behavior stays in the desktop service, while the existing settings, IPC, draft, attachment, and provider paths carry the result.

Platform Details

  • macOS and Windows capture the active window. Linux supports Wayland sessions; X11 sessions are unavailable.
  • macOS permission prompts only run on the off-to-on settings transition. Already-granted permissions do not prompt again.
  • Linux Wayland uses the system portal or the optional GNOME extension and requires a key chord. Portal captures omit accessibility data because the portal does not identify the selected window.
  • Outside Wayland, normal key chords are probed through the OS global-shortcut registry. Wayland submits key chords to the desktop portal without claiming that the desktop accepted them. Modifier-only shortcuts cannot be reserved exclusively by any operating system.
  • Modifier-pair listeners run only while Window Capture is enabled, require both physical keys to be held together, and stop on disable or app shutdown. macOS uses a permission-free polling helper; Windows and Linux use the isolated Node-mode hook.

Demo

CleanShot.2026-08-31.at.3.17.42.mp4

Testing on macOS

  1. Run pnpm dev:desktop, then open Settings > Window Capture.
  2. Turn Window Capture on. Permission requests should appear during this off-to-on change, not at startup or on the first capture.
  3. In System Settings > Privacy & Security, allow both T3 Code (Dev).app and the terminal app used to run pnpm (such as Terminal, iTerm, or Warp) in:
    • Accessibility — lets T3 Code identify the active window and attach available text.
    • Screen & System Audio Recording (called Screen Recording on older macOS) — lets T3 Code capture the window image.
      If T3 Code is missing, use the + button to select <repo>/apps/desktop/.electron-runtime/T3 Code (Dev).app.
  4. Fully restart dev mode after changing either permission: stop pnpm dev:desktop with Ctrl+C, then run it again. If macOS offers Quit & Reopen, still restart the terminal command so the dev server and app start together.
  5. Open another app and press both Shift keys together. Confirm T3 Code attaches the image, app name, window title, and available icon and text to the current draft.
  6. Turn Window Capture off and on again. Already-granted permissions must not prompt again.
  7. Check a custom shortcut, each inline sound play button, Capture flash, Capture animations, and Capture window from the command palette.

Verification

  • Focused tests passed for contracts, desktop capture, modifier-pair state, shortcut conflicts, typed IPC, durable delivery, settings support, draft bridge behavior, and timeline rendering.
  • Strict focused lint, formatting, and whitespace checks passed.
  • Contracts, web, server, and desktop type checks passed.
  • Production web, server, and desktop builds passed.
  • Electron desktop smoke test passed with isolated state under Xvfb.
  • Desktop startup was verified with Window Capture and Capture Flash already enabled.
  • Modifier-pair observation and accessibility extraction run outside the Electron main process. Native accessibility crashes, early exits, and timeouts fall back to screenshot-only; source thumbnails remain memory-bounded.
  • The capture flash uses renderer-free native windows on macOS and Windows. Linux uses a short-lived renderer fallback. Every flash timer and window is destroyed after playback.
  • The desktop bundle keeps uiohook-napi as a packaged runtime dependency with prebuilt macOS, Windows, and Linux binaries for the supported CPU architectures.

Checklist

  • One user workflow
  • Cross-platform behavior is explicit
  • Permission timing is explicit
  • UI evidence is included
  • Focused tests and builds pass

Generated by GPT-5.6-sol in T3 Code using the Codex harness.

Note

Add cross-platform window capture with native Linux, macOS, and Windows backends

  • Desktop process gets a DesktopWindowCapture coordinator that manages pending captures, compositor-specific backends, accessibility extraction, transparent transition overlays, and PNG persistence
  • Native capture backends added for Hyprland (Wayland protocol), KDE (D-Bus/KWin scripts), GNOME (Shell extension), macOS (screencapture), and Windows (xa11y), with shortcut registration via D-Bus portals, compositor configs, and native modifier-pair polling
  • Web layer adds WindowCaptureCoordinator.tsx, composer attachment frames with arrival animation, a two-step setup wizard, shortcut recorder, sound playback, and a settings panel with backend-specific status
  • New IPC contracts in orchestration.ts and ipc.ts define capture state, config preview/apply, shortcut schemas, accessibility element trees (bounded to 10,000 nodes / 32,000 serialized chars), and DesktopBridge methods; captured accessibility metadata is compacted and appended to provider input as untrusted JSON
  • Risk: Electron upgraded to 44.1.0 in desktop/package.json; composerDraftStore.addImage now returns boolean and rejects duplicate image IDs; keybindings event-key resolution uses physical key codes for punctuation shortcuts; client settings writes are now serialized through a promise chain; window-capture IPC handlers validate sender identity against the current main window's web-contents ID

Macroscope summarized 2fad302.

Summary by CodeRabbit

  • New Features
    • Added desktop window capture with configurable global shortcuts across macOS, Windows, Linux, and Wayland.
    • Captures include previews, application details, icons, accessible text, configurable sounds, and visual animations.
    • Added onboarding, command palette access, animated composer attachments, and a dedicated Window Capture settings page.
    • Added permission guidance, shortcut conflict validation, status reporting, and pending-capture recovery.
  • Bug Fixes
    • Improved attachment metadata preservation, settings persistence, desktop window focusing, and menu-action delivery.
  • Documentation
    • Added guidance for configuring and using window capture.

@coderabbitai

coderabbitaiBot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3f127620-45e6-49c1-a4b4-463c86a598df

📥 Commits

Reviewing files that changed from the base of the PR and between 297e448 and 293c41d.

📒 Files selected for processing (3)
  • apps/desktop/src/windowCapture/DesktopWindowCapture.test.ts
  • apps/desktop/src/windowCapture/DesktopWindowCapture.ts
  • apps/desktop/src/windowCapture/WindowCaptureTransition.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds desktop window capture across contracts, native capture, IPC, web chat, settings, persistence, provider serialization, packaging, and documentation. It supports platform-specific shortcuts, capture metadata, animations, and pending-capture recovery.

Changes

Window Capture Feature

Layer / File(s)Summary
Contracts and desktop runtime
packages/contracts/..., apps/desktop/src/windowCapture/..., apps/desktop/src/electron/...
Adds capture schemas, platform capture paths, shortcut workers, accessibility metadata, persistence, animations, and desktop integration.
IPC and web capture flow
apps/desktop/src/ipc/..., apps/desktop/src/app/..., apps/desktop/src/main.ts, apps/desktop/src/preload.ts, apps/web/src/components/desktop/..., apps/web/src/components/chat/..., apps/web/src/lib/...
Connects trusted IPC handlers, preload methods, capture coordination, animation destinations, composer focus, attachment previews, and command-palette capture.
Settings and delivery
apps/web/src/components/settings/..., apps/web/src/hooks/..., apps/web/src/composerDraftStore.ts, apps/server/src/...
Adds capture settings and onboarding, serialized settings writes, source metadata persistence, provider-safe window data encoding, and attachment size enforcement.
Build and documentation support
apps/desktop/scripts/..., apps/desktop/package.json, apps/desktop/vite.config.ts, pnpm-workspace.yaml, scripts/..., docs/...
Adds native runtime dependencies, worker packaging, macOS signing and usage metadata, development launcher updates, tests, and user documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk:🟠 High · up to 293c4

Window capture now imports text and metadata from other applications, listens for global shortcuts, and drains captures into drafts. At the current head, captured text is not isolated from provider instructions, the default shortcut can fail on Windows/Linux, timed-out accessibility work can accumulate, and captures can target stale or different drafts; these create security, reliability, and data-integrity risks, so the PR is not merge-ready without fixes or explicit acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 102 functions across 70 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly and concisely identifies the main change: adding cross-platform desktop window capture.
Description check✅ PassedThe description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not…
Full details: Description check

Explanation

The description includes What Changed, Why, platform details, testing evidence, UI evidence through a demo, and a checklist. It is mostly complete, although it uses custom checklist items and does not retain the exact UI Changes heading.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 24, 2026

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service (apps/desktop/src/windowCapture/DesktopWindowCapture.ts), its IPC methods, layer wiring, and contract additions. Module shape (namespace imports, Context.Service with inline interface, exported make/layer, dependencies acquired via yield*) follows the conventions; the Effect.runPromiseWith bridge for the Electron globalShortcut callback matches the existing desktop native-callback pattern. Two error-modeling issues below.

Posted via Macroscope — Effect Service Conventions

Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadpackages/contracts/src/ipc.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review

Two findings, both in the new window-capture UI.

  1. apps/web/src/index.css introduces a global rule keyed on the class trio .animate-in.fade-in.zoom-in-95, which does not exist as a utility in this project. The repo's established pattern (see ComposerStashBadge/prompt-stash-count-enter) is keyframes in index.css plus an animate-[…] utility at the owner. As written, the unlayered global rule wins over Tailwind's layered animate-none, so the call-site motion-reduce:animate-none is inert (reduced motion only works because of the duplicated media query), duration-200 never affects the animation, and any future element combining those class names silently inherits this animation.

  2. WindowCaptureSettings records a shortcut on a Button without the data-keybinding-capture opt-out that app-level shortcut handlers check, so global capture-phase handlers (e.g. sidebar.toggle in AppSidebarLayout) consume the keypress before the recorder sees it.

Details inline.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
Comment threadapps/web/src/index.css Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
@macroscopeapp

macroscopeappBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a broad cross-platform capture workflow with native compositor integrations, permission and shortcut handling, persistent screenshots, accessibility data, and provider prompt changes. Its production and sensitive-data blast radius, plus new static-analysis overrides and product defaults, require human review.

Not approved because:

  • Per-review cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Window capture UI review: one new finding on WindowCaptureOnboardingDialog (dialog body bypasses the DialogPanel contract). The three findings from the previous run (index.css.animate-in.fade-in.zoom-in-95 global override, the matching ChatComposer class string, and the missing data-keybinding-capture on the shortcut recorder) are unchanged in this head commit and are not re-posted.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the web-side window capture surfaces. Four findings, all on newly added lines; the three items flagged in the previous run (global .animate-in override, call-site animation ownership, and the missing data-keybinding-capture marker) are resolved in this revision.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureOnboardingDialog.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI consistency review of the window-capture web surfaces. Two findings on the new settings page; the composer/timeline attachment cards, the onboarding dialog, the keyframe ownership, and the command palette action all follow the existing primitives and conventions.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the web UI surface of this feature (composer/timeline capture cards, capture settings page, onboarding dialog, command palette entry, keyframe ownership). The earlier findings on the window-capture-card-enter keyframe ownership, data-keybinding-capture, DialogPanel composition, capability gating of the onboarding dialog, and the disabled state of the sound/flash/animation switches are all resolved. Two remaining items below.

Still open from earlier reviews (not re-posted): the shortcut recorder still preventDefault()s Tab so focus cannot leave the button, and WindowCaptureAttachmentDetails still anchors its title to text-foreground while rendering on bg-background/70 inside the text-message-foreground bubble.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture web code. Previously flagged items (recorder Tab handling, data-keybinding-capture, disabled-state parity for the feedback switches, capability gating on the onboarding dialog, DialogPanel composition, and the named window-capture-card-enter keyframes) all look resolved at this head.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureCoordinator.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changed web UI surfaces (window-capture settings page, onboarding dialog, composer/timeline attachment cards, command palette entry, index.css keyframe). Earlier rounds' items (shared focus-event constant, DialogPanel composition, data-keybinding-capture, Tab handling, named keyframe animation, uniformly gated switches) all look resolved. Two remaining consistency points on the new settings page.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings on the settings surface introduced/changed here. The composer and timeline capture cards, the extracted SettingsUnavailableGroup, the keyframe ownership, and the onboarding dialog composition all look consistent with the existing primitives now.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/settings/settingsLayout.tsx
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated

@macroscopeappmacroscopeappBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One consistency finding on the new window-capture attachment card; everything else (settings layout group, dialog composition, keyframe ownership, palette action) matches the existing contracts.

Posted via Macroscope — UI Consistency

Comment threadapps/web/src/components/chat/ChatComposer.tsx Outdated
When a capture and the T3 window sit on different displays, macOS runs one
overlay per display. A failure on either surface rejected the shared flight
promise, so landing resolved early and the overlay was destroyed mid-animation:
the card froze half-scaled with no border or app details before disappearing.
Settle each overlay's flight independently so one display failing no longer
cuts the animation short on the others.
Acknowledging a capture tears down the desktop overlay, and the composer only
revealed the attachment tile afterwards. The tile was therefore missing for
every frame between the flight landing and its first paint, which reads as a
blank gap on slower handoffs such as a capture from a second display.
Wait for the landing, reveal the tile, then acknowledge.
Comment threadapps/web/src/components/chat/WindowCaptureAttachmentDetails.tsx Outdated
Comment threadapps/web/src/components/chat/ExpandedImageDialog.tsx Outdated
Comment threadapps/web/src/components/chat/ChatComposer.tsx
Bil0000and others added 5 commits September 1, 2026 22:07
Add opt-in capture setup, bundled GNOME installation, Niri shortcuts, and the native KDE capture helper with motion and accessibility support.
Use direct portal shortcut registration and remove obsolete Electron migration, launch onboarding, shortcut-test state, and duplicate backend probing.
Verified with 488 focused TypeScript tests, 11 native KDE tests, scoped lint/format checks, and desktop, web, and build-script typechecks.
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/settings/WindowCaptureSettings.tsx Outdated
get-windows reports the active window in physical pixels, but xa11y reports
UI Automation bounds in logical (DIP) coordinates. On any display above 100%
scaling the exact-bounds match in findAccessibleWindow never succeeded, so
Windows captures shipped without accessible text or element trees.
Convert the active window bounds to DIP before the accessibility lookup,
matching the region the screenshot already uses. Element coordinates keep
mapping onto the physical screenshot because the image size is still derived
from the captured PNG.
Generated with Claude Fable 5.1 in Claude Code.
Comment threadapps/web/src/components/settings/settingsSearch.ts
Comment threadapps/desktop/src/windowCapture/DesktopWindowCapture.ts Outdated
Comment threadapps/web/src/components/desktop/WindowCaptureShortcutKeys.tsx Outdated
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@Bil0000@SunkenInTime@t3dotgg@nmggithub@juliusmarminge@StiensWout