feature: deliver host-captured images to vision-capable models - #91
Merged
Conversation
Model image support was already detected but there was no way to send an image, so a screenshot tool could not exist. Hosts can now call TryAttachImage, which refuses unless the model reports vision support, the payload is an image, and it fits the size cap. A tool result is text, so an image cannot ride back inside one. A queued image is added to history as a real user message once the current turn finishes, and the turn is extended so the model actually looks at it, bounded by MaxImageDeliveriesPerTurn. Delivered images are retracted when the user turn ends. An image is evidence for the turn that captured it; leaving it in history would re-upload megabytes on every later request and crowd out the context the model needs. The model's written conclusion persists instead. Registers screenshot_desktop_preview as a live preview tool so its results are never served from the result cache, and adds a transport seam so delivery is asserted end to end without a live daemon. A probe test pins the wire contract that image content reaches Ollama's images array, which the whole feature rests on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The app can already tell whether the selected model accepts images. It had no way to actually send one, so a screenshot feature could not be built on top of it. This adds the delivery half.
A host captures an image and hands it over. It is refused unless the model genuinely reports image support, the content really is an image, and it fits a size limit. A text-only model is told plainly that it cannot receive images rather than being handed something it will quietly ignore.
Why this was done
Without this, the agent could take a screenshot and still have no way to look at it. This is the missing piece that makes visual checking possible at all.
How delivery works, and why it is not simpler
A tool result is text. An image cannot be returned inside one. So a captured image is added to the conversation as a normal message once the current step finishes, and the model is given another step to actually examine it. The number of such extensions per request is capped, so a model cannot loop taking pictures of itself.
Delivered images are removed from the conversation once the request finishes. Keeping them would re-upload several megabytes on every later message and crowd out the room the model needs to think. What persists is the model's written conclusion about what it saw.
Verification
716 automated tests pass on .NET 8 and .NET 10. This includes a test that asserts an attached image travels all the way to the model server in the format it expects, and a test that the image is delivered and then cleared from the conversation. Refusal cases for text-only models, unknown capability, oversized payloads, and non-image content are covered.
Scope
This delivers images. It does not capture them. The screenshot feature that uses this is a companion change in
MandoCode.Desktop, which also registers its tool here so repeated captures are never served from a cached result.