Uh oh!
There was an error while loading. Please reload this page.
feat: modality negotiation for ui/message and ui/update-model-context - #119
feat: modality negotiation for ui/message and ui/update-model-context#119idosal wants to merge 10 commits into
Conversation
@modelcontextprotocol/ext-apps@modelcontextprotocol/server-basic-react@modelcontextprotocol/server-basic-vanillajs@modelcontextprotocol/server-budget-allocator@modelcontextprotocol/server-cohort-heatmap@modelcontextprotocol/server-customer-segmentation@modelcontextprotocol/server-map@modelcontextprotocol/server-pdf@modelcontextprotocol/server-scenario-modeler@modelcontextprotocol/server-shadertoy@modelcontextprotocol/server-sheet-music@modelcontextprotocol/server-system-monitor@modelcontextprotocol/server-threejs@modelcontextprotocol/server-transcript@modelcontextprotocol/server-video-resource@modelcontextprotocol/server-wiki-explorercommit: |
There was a problem hiding this comment.
Pull request overview
This PR aligns the ui/message specification with the existing implementation in the ext-apps SDK by changing the content parameter from a single object to an array of content blocks. This enables multi-modal messages such as combining text and images in a single message.
Key changes:
- Updated
ui/messagecontent parameter to accept an array of content blocks instead of a single object - Maintains consistency with the existing TypeScript implementation which already uses
ContentBlock[]from the MCP SDK - Enables the use case mentioned in issue #48 for multi-modal messages
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| content: [ | ||
| { | ||
| type: "text", | ||
| text: string | ||
| } | ||
| ] |
There was a problem hiding this comment.
The specification should document what content types are supported in the content array. While the PR description mentions supporting images (e.g., "prompting the agent with a text message and an image"), the spec only shows a single text content block.
Consider adding:
- A reference to the MCP SDK's
ContentBlocktype definition - Documentation of supported content types (text, image, etc.)
- An example showing multiple content items in the array (e.g., text + image)
This would help implementers understand the full capabilities of this API and align with the stated goal of supporting additional use cases like multi-modal messages.
There was a problem hiding this comment.
@ochafik I see that the SDK supports ContentBlock as-is. There are types that are extremely cool and open up really interesting use cases, but I'm not sure hosts would know how to support and may require negotiation (e.g., audio, embedded resource). Should we limit it to text (and perhaps image if we add negotiation) for now?
There was a problem hiding this comment.
@ochafik sounds good. Let's merge this one and I'll open a new one for the negotiation
The MCP Apps draft spec (SEP-1865) has been in flux regarding the
ui/message params.content type. After discussion with the spec authors,
the agreed format is ContentBlock[] (array of content blocks) rather
than a single ContentBlock object.
This change:
- Updates McpMethodParams['ui/message'] to expect content as ContentBlock[]
- Adds ContentBlock type supporting text, image, and resource blocks
- Updates the handler to extract text from the first text block in the array
- Returns empty object {} on success per the spec
Related spec discussions:
- modelcontextprotocol/ext-apps#48
- modelcontextprotocol/ext-apps#119
- MCP-UI-Org/mcp-ui#166Uh oh!
There was an error while loading. Please reload this page.
Fixes npm ci failures in CI by updating lock file to match current dependency tree. Updates hono from 4.11.4 to 4.11.5 and @modelcontextprotocol/sdk to 1.25.3.
| /** Host supports resource link content blocks. */ | ||
| resourceLink?: {}; | ||
| /** Host supports structured content (updateModelContext only). */ | ||
| structuredContent?: {}; |
There was a problem hiding this comment.
Why is structuredContent here? Looking at this thru the lens of a tool result, structued content is supposed to be a sibling to the unstructured content blocks.
Based on my understanding,updateModelContext would be able to set Content[] and StructuredContent. Please correct me if I'm wrong!
There was a problem hiding this comment.
Just Opus doing Opus things. Thanks, great catch!
There was a problem hiding this comment.
No it's actually intentional, not all hosts might support structuredContent, it's a modality of its own (JSON vs. text)
There was a problem hiding this comment.
Yeah, I understand the support rationale... but the point I'm attempting to make is "structured content (JSON) vs unstructured content (text, image, resource, etc)" .... as opposed to just "JSON vs text"
There was a problem hiding this comment.
@aharvard you're right, there's a disambguity here. Perhaps we need a separate SupportedModelContextModalities type for the updateModelContext capability:
interfaceSupportedModelContextModalities{content?: SupportedContentBlockModalities,structuredContent?: {}}And to remove structuredContent from SupportedContentBlockModalities.
This will make it clearer, closer to the impolementation, and will also remove the need for the (updateModelContext only) comment clarification.
wdyt?
89c5058 to
5bf54d2Compare
The spec defines the
ui/message's content as a single message. Passing an array would support additional use cases (e.g., prompting the agent with a text message and an image). This is already the implementation in theext-appsSDK.Since
ContentBlocksupports multi-modal inputs, I've added a modality capability negotiation to allow apps and hosts to pass non-textual inputs to messages and context updates.See #48