fix(preview): make in-app browser automation work on real app pages by gbarros-dev · Pull Request #7127 · pingdotgg/t3code

gbarros-dev · 2026-08-15T21:05:04Z

Agents inspecting a real app in the in-app browser were drowning in accessibility trees, matching sidebar chrome on wait, treating hidden buttons as missing, and timing out on resize when the preview panel was not visible.

Snapshots are slim by default (screenshot, main-landmark text, visible clickable elements). Wait defaults to main. Click reports hidden, disabled, or ambiguous targets without leaking locators. Local environment ports resolve to localhost so Vite on ::1 or 127.0.0.1 is reachable. Resize applies through CDP even when the panel is hidden. Console and network stay behind preview_snapshotinclude.

The preview tools assumed a simple, visible page. Real product UIs have sidebars, hidden dialog triggers, and dual-stack local servers, so agents spent turns fighting the harness instead of the app.

Implemented with Grok 4.6 through Grok CLI.

Note

Medium Risk
Touches desktop CDP automation, IPC contracts, and navigation URL resolution; behavior changes for agents using snapshot, wait, click, and resize, but scoped to preview automation with rollback and backward-compatible optional APIs.

Overview
Preview automation is tuned for real product UIs: slimmer defaults, clearer click failures, and guest viewport control when the panel is hidden.

Snapshots default to URL, main-landmark visible text, filtered interactive elements (with stable e1 ids), screenshot, and action timeline. Heavier accessibility, console, and network data load only when include requests them; AX tree fetch and optional slices are skipped otherwise, with screenshot fallback if capturePage fails.

Click resolution distinguishes missing, hidden, disabled, and ambiguous (strict-mode) targets via new typed errors, mapped on the web host to execution messages without leaking locators. Wait-for gains optional scope (main vs document), searches text/locators in that root, requires visibility, and ignores dialog trigger slots.

Resize persists server viewport state and applies CDP device metrics on the guest (setViewport / clear for fill), including rollback on failure and manual chrome resize—so automation size changes work when the preview is not visible.

Local dev URLs map loopback environment hosts (127.0.0.1, ::1) to localhost so dual-stack guests reach Vite bound on either stack.

Contracts, IPC/preload, MCP tool descriptions, and tests cover the new inputs and error shapes.

^{Reviewed by Cursor Bugbot for commit f7d8c91. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Fix in-app browser automation to work on real app pages by adding viewport override and richer snapshot support

Adds a setViewport IPC channel and bridge method that applies or clears a CDP Emulation.setDeviceMetricsOverride per tab, so viewport changes take effect immediately in the guest webview.
Extends automationSnapshot to accept optional include flags (ax, console, network), making snapshots slimmer by default and returning diagnostics only on demand.
Improves click failure reporting with typed errors (hidden, disabled, ambiguous) instead of a generic not-found error, and maps these through the host-side error layer.
Normalizes loopback addresses (::1, 127.0.0.1) to localhost in resolveEnvironmentPortTarget for dual-stack server compatibility.
PreviewAutomationWaitFor gains an optional scope field (main | document) to constrain text/selector matching to the main landmark.

^{Macroscope summarized f7d8c91.}

coderabbitai · 2026-08-15T21:05:13Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7f4b4792-75db-4d03-844b-619d0055471b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

macroscopeapp

Reviewed the Effect service conventions for the changed preview automation code. Two error-modeling violations: a new failureKind discriminator selects the user-facing message in both the desktop and web target-not-found errors, and the new host error drops the underlying failure instead of preserving it as cause. Imports, service definition/layer shape, dependency acquisition, and runtime boundaries in the touched code all look fine.

^{Posted via Macroscope — Effect Service Conventions}

cursor

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 0f2d5d3. Configure here.}

macroscopeapp · 2026-08-15T21:10:29Z

Approvability

Verdict: Needs human review

This PR introduces significant new capabilities (viewport control, scoped element search) and changes runtime behavior (element detection, main-landmark preference for visible text) beyond what the 'fix' label suggests. The scope of changes across preview automation warrants human review.

^{You can customize Macroscope's approvability policy. Learn more.}

gbarros-dev · 2026-08-17T07:40:18Z

Closing to split this into smaller PRs. The problem is still real. The change was four independent fixes sharing one review and then conflicting with #7082.

Replacement PRs, in this order:

Loopback environment ports resolve to localhost
Click reports hidden / disabled / ambiguous without leaking locators
Slim preview_snapshot defaults and wait scope main
Hidden-panel viewport via CDP (after fix(preview): failed tabs no longer sit as a blank floating panel #7236)

This branch is not being merged.

gbarros-dev · 2026-08-17T07:46:13Z

Replacement PRs:

fix(preview): open local environment ports on localhost #7300 loopback → localhost
fix(preview): name hidden, disabled, and ambiguous click failures #7301 click hidden / disabled / ambiguous
fix(preview): keep snapshots slim unless the agent asks #7302 slim snapshot + wait scope main
fix(preview): apply viewport changes when the panel is hidden #7303 hidden-panel viewport via CDP

gbarros-dev · 2026-08-15T21:05:04Z