feature: interactive browser preview for the coding agent - #52
Merged
Conversation
The agent can now use the preview pane as a real browser to check its own work on a web page. It reads page text and controls, clicks, hovers, presses keys, fills forms, selects options, scrolls, waits for elements, and reads browser errors. Pointer and keyboard input are dispatched as genuine browser events, so a page cannot tell them from a person. Edited assets always load. A refresh could previously redisplay a cached script or stylesheet, so a review would pass against the previous version of a change. The preview's cache is disabled and refreshes ask for a cache-bypassing reload, so project files never need cache-busting query strings to preview correctly. Keyboard input covers Enter to submit, Tab to move focus, arrow and letter keys for games and canvas apps, and typeahead fields that only react to key events. Keys can be held for a bounded time and are always released before the call returns, so none is ever left down between calls. Repeated clicks and key presses take a count instead of a call each. Every repeat re-locates its target, and an interrupted batch reports how many completed rather than replaying from the start. Any action can name one element to read back instead of returning a whole page snapshot, which keeps a sequence of checks affordable. Only project files can be opened. The page operations are fixed rather than arbitrary script evaluation, agent input is passed as data, external navigation, popups and downloads are blocked during agent use, and unsaved user edits are never overwritten. Page content is treated as untrusted observation. Verified by unit tests and an opt-in WebView2 integration harness that drives a real browser through typing, form submission, keyboard focus movement, repeated clicking, and an edited script reloading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The agent can now use the app's built-in preview pane as a real browser to check its own work on a web page, instead of writing code and asserting it probably works.
It can read what is on the page, click, hover, press keys, fill forms, choose dropdown options, scroll, wait for something to appear, and read browser errors. Clicks and keystrokes are delivered as genuine browser input, so the page cannot tell them apart from a person using it.
Why this was done
Reviewing a web change previously meant a person opening the page and checking by hand. The agent could edit a button and had no way to confirm the button worked.
What is included
Safety and scope
The agent can only open files inside the current project. It cannot reach external websites or run arbitrary code in the page: the available operations are fixed, and anything the agent supplies is treated as data. External navigation, pop-ups, and downloads are blocked during agent use, and unsaved edits in the preview are never overwritten. Page content is treated as untrusted information, not as instructions.
Deliberately not included: screenshots, so layout and visual overlap still cannot be judged; local development server URLs; drag-and-drop and file uploads. These limits are documented, and the agent is instructed to report them rather than work around them.
Verification
263 automated tests, plus an opt-in integration test that drives a real browser end to end: typing, form submission, keyboard focus movement, repeated clicking, and confirming an edited script reloads. Manual test steps are in
docs/browser-tools.md.Submodule
Bumps the
MandoCodesubmodule to04609ac, which includes DevMando/MandoCode#90 (merged). That change supplies the model capability reporting and the cache exemption that keeps repeated browser actions from being silently deduplicated.