.NET: Add `ConversationStoreFactory` API for custom DevUI storage implementations

[svnscha]

When using DevUI with a custom ChatMessageStoreFactory for persistent thread storage, conversations are not persisted across application restarts because there's no equivalent factory API for the conversation listing functionality.

The framework provides an elegant factory pattern for customizing thread message storage: However, DevUI's conversation listing (GET /v1/conversations?agent_id=X) relies on separate internal services IConversationStorage, IAgentConversationIndex that only have in-memory implementations and no equivalent factory API for customization.

UPDATE: The problem is more complex than initially expected.

Sub-Issue 1: Conversation ID
Is conversation preservation actually supported? When messages are saved to a custom store, there's no way for the store to know the associated Conversation ID. The store serialization is not used, nor is there another way to know the current thread from within the message store.

Sub-Issue 2: Application Restarts
Even if messages are persisted, there's no public or supported way to hook into the IConversationStorage interfaces - they're internal. This creates a disconnect: a message store can save messages, but when DevUI requests chat history, there's no API endpoint to attach to, so nothing responds.

I'd appreciate input here cause now I am questioning if this is even intended or just not yet implemented. It kind of feels like that attaching a message store is probably not the use-case at all for DevUI.

[svnscha]

While exploring the agent-framework codebase, it seems that persistent messages aren't yet a first-class use case. The WithThreadStore API exists but doesn't appear to be used in the Hosted Agent setup, especially with the DevUI. Message history also doesn't seem to be preserved.

Now it's unclear whether this is a bug, an intentional limitation, or simply not implemented yet for the DevUI. I'd expect runtime message history to be preserved, potentially with custom (non-in-memory) storage. However, when debugging the examples, only the latest user message is sent to the chat completion API.

Overall it turns out the issue is more complex than initially thought.

It looks like AG-UI likely addresses these gaps, as it provides full thread (conversation) control.

[svnscha]

Most likely could be solved with #2520 .

[westey-m]

Sven Scharmentke (@svnscha), thanks for logging an issue.

Some background: ChatMessageStore is an alternative to conversation id and they are mutually exclusive. Conversation id is only used when the underlying AI or Interference service stores the chat history. When the service does not support this, and a ChatMessageStore is used instead, there is no conversation id. It is up to the ChatMessageStore to generate ids to store the chat history under (if it is indeed storing it in an external store at all).

AgentThreads can be serialized and deserialized so that they can be persisted in storage across incoming service calls when an agent is hosted in a service. The ChatMessageStore on the AgentThread (if available) is serialized with the AgentThread, and any ids that the ChatMessageStore requires should be stored in this serialized payload. See e.g. the CosmosDBChatMessageStore implementation

InMemoryChatMessageStore is the default ChatMessageStore implementation used if the AI Service does not require in-service message storage and if no custom ChatMessageStore is supplied via the ChatMessageStoreFactory. On Serialization, this stores the full chat history in the serialized state, and therefore when serializing an AgentThreads with an InMemoryChatMessageStore, the serialized AgentThread contains the full chat history. Very simple to use without needing to implement a custom ChatMessageStore.

Note that AgentThread objects may also contain other state from e.g. AIContextProviders that need to be preserved long term.

Therefore, it's important for AgentThreads to be serialized and stored between different service calls. This closes the loop around storage of either the chat history directly, or the id of the chat history in the underlying ai service (conversation id) or the id of the chat history in 3rd party storage (e.g. CosmosDBChatMessageStore), plus any other state in the AgentThread.

See the A2A Hosting implementation, which is a good example of how hosting should work. A persistent AgentThreadStore is required to ensure agent threads are stored long term.

Note that not all hosting code in the repo has fully implemented this pattern yet, and this is work in progress.

Also note that the protocol over which the Agent is exposed will determine the ids that are publicly returned to callers of the service, i.e. is it a static context id for A2A or a response id that changes with each request when using an OpenAI responses compatible protocol? These ids may also not be mappable to the chat history storage ids used by the agent, if the agent has ids at all (InMemoryMessageStore uses none). The host also doesn't know what type of chat history storage the agent uses. Therefore it is safest to consider the public ids independent of the internal ids and generate them in the host like on line 44 of the A2A AIAgentExtensions.

Also see more here:

• https://learn.microsoft.com/en-us/agent-framework/user-guide/agents/agent-memory
• https://learn.microsoft.com/en-us/agent-framework/user-guide/agents/multi-turn-conversation#agentthread-storage

[markwallace-microsoft]

Sven Scharmentke (@svnscha), friendly reminder — this issue is waiting on your response. Please share any updates when you get a chance. (This is an automated message.)

[ZeeLyn]

Is there any progress?