feature: model vision detection and live-result handling for browser tools - #90
Merged
Merged
Conversation
Model validation now reads the capability list Ollama reports and records whether the selected model accepts image input. The status is shown in the CLI and Desktop and written into the agent's system instructions, so a text-only model is told not to claim visual inspection and a vision model is told a file path is not an image. Detection is keyed to model and endpoint, so it resets when either changes and a late response about a previous model cannot overwrite newer state. Detecting image support does not deliver images. No screenshots are captured or attached; this is capability reporting only. Live preview tool results are also exempted from the agent result cache. Repeating an identical call is normally deduplicated, which is right for reading a file and wrong for clicking a button, where the second click has to actually happen. Failed preview results now register as failures rather than successful steps, and a fresh DOM observation counts as browser evidence for plan checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Two capabilities the desktop browser tools depend on.
Model image-capability detection. The app now detects whether the selected local model can accept images, by reading the capability list the model server reports. The result is shown in the conversation ("Vision supported" / "Text-only model" / "Vision capability unknown") and is written into the agent's instructions. A text-only model is explicitly told not to claim it looked at anything; a vision-capable model is told that a file path is not the same as being handed an image. Detection resets whenever the model or server changes, and a slow response about a previous model cannot overwrite newer information.
Live browser results bypass the result cache. Repeating an identical action is normally deduplicated to save time. That is correct for reading a file and wrong for clicking a button — the second click has to actually happen. Failed browser results are now recognized as failures rather than counted as successful steps, and a fresh page reading counts as real evidence when a plan is checked.
Why this was done
Without capability detection, a text-only model could claim to have visually reviewed a page it never saw. Without the cache exemption, repeated browser actions would silently not occur.
Scope
Detecting that a model accepts images does not give the app any way to send one. No screenshots are captured or attached. This is groundwork plus honest reporting, not a vision feature.
Older model servers that do not report capabilities are handled as "unknown" and still validate successfully. No guessing from model names.
Verification
710 automated tests pass on .NET 8 and .NET 10, including coverage for capability parsing, invalidation when the model or endpoint changes, and out-of-order responses from the model server.
Note for reviewers
MandoCode.Desktophas a companion change that depends on this one. That repository cannot build until this merges and its submodule pointer is updated.