feature: preview screenshots for vision models and local development servers - #53
Merged
Merged
Conversation
Screenshots close the one gap DOM inspection cannot: layout, overlapping or clipped elements, spacing, and canvas rendering. screenshot_desktop_preview captures the visible viewport, or one element when given a selector, and hands the bytes to the model as real image input. Capability is checked before capturing, so a text-only model is told plainly that visual layout could not be checked rather than being handed bytes it will silently drop. The image never enters the model's text context: the tool result carries only metadata while the bytes travel as image content. Development server previews let the agent exercise a running app rather than a static file. Only http or https on loopback with an explicit port is accepted; external hosts, LAN addresses, other schemes, and URLs carrying credentials are refused. That widening is why the origin check moved from a hardcoded host to the one origin the preview was actually opened on, passed to the page as data and compared there. Every script call and every navigation is checked against it, so allowing a dev server never allows the network. Adds the seam test between this app and the pinned harness: an unwired IAiService reports unknown vision and refuses images, and the sink turns every vision state into an honest answer. A pin bump that changed either side would otherwise break delivery silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Two things the browser preview could not do.
Screenshots. The agent can capture what the preview actually looks like and give it to the model as a real image. This covers the one class of question reading the page cannot answer: layout, elements overlapping or getting cut off, spacing, and anything drawn on a canvas. It captures the visible pane, or a single element when asked.
Development servers. The preview can now open an app already running on this machine, instead of only static files. That means the agent can check a real running app, not just a page on disk.
Why this was done
Until now the agent could confirm that a button existed, was enabled, and responded to a click, while remaining completely unable to see that it was sitting on top of another button. Reading a page and looking at a page are different checks, and only one of them was possible.
Static files were also a real limit: most web work runs behind a development server, and none of it could be checked.
Honesty about what the model can see
A screenshot is only useful to a model that accepts images. That is checked before anything is captured, so a text-only model is told plainly that visual layout could not be checked, rather than being handed an image it quietly ignores and then describing something it never saw.
The image is given to the model as image content, never pasted into its text as data. It is treated as evidence for the question that prompted it and is dropped afterward, so a screenshot does not get re-sent with every later message.
Safety
Allowing development servers is a deliberate widening, so the boundary was tightened rather than loosened. Only
httporhttpsonlocalhostor127.0.0.1with an explicit port is accepted. External sites, addresses on the local network, other schemes, and URLs carrying a username and password are all refused.Previously the preview trusted a single hardcoded address. It now records the one address it actually opened and checks every page reading and every navigation against it, so a development server page that tries to redirect somewhere else is blocked exactly as a project file would be.
Verification
279 automated tests, up from 263. The integration test that drives a real browser now also confirms screenshots come back as genuine images, that capturing one element captures less than the whole page, and that a lookalike address is rejected.
This also adds a test across the boundary between this app and the shared engine it pins: an unwired implementation must report unknown image support and refuse images. Without it, updating that pin could break screenshots silently.
Submodule
Bumps the
MandoCodesubmodule tofd4458d, which includes DevMando/MandoCode#91 (merged). That change supplies image delivery: without it a captured screenshot would have nowhere to go.