Implement elicitation support in gopher-mcp-python

[bettercallsaulj]

Implement elicitation support in gopher-mcp-python

Goal

Implement MCP server-to-client elicitation in gopher-mcp-python so provider OAuth requested during a tool call works automatically, matching gopher-mcp-js.

The target behavior is:

• Existing examples such as examples/api/create_by_url.py should not need explicit elicitation configuration.
• When a tool call returns a provider OAuth authorization URL, the SDK opens or prints the URL, waits for the user to complete the browser flow, accepts the elicitation, and lets the native agent retry the tool call.
• First-step MCP server OAuth remains separate from second-step provider OAuth elicitation. Setting SDK OAuth to disabled should not accidentally disable provider elicitation unless the caller explicitly disables/cancels elicitation.

gopher-mcp-js reference points

• src/elicitation.ts: public request/response/action/options types.
• src/elicitationRuntime.ts: default URL-mode handler, browser/manual fallback, sync handler enforcement, timeout handling, action normalization, URL redaction.
• src/ffi/library.ts: native request struct decoding, callback construction, action int mapping, callback lifetime retention, feature support checks.
• src/agent.ts: create-options merge behavior that defaults omitted elicitation to {}.
• third_party/gopher-orch commit aed9849e in gopher-mcp-js, also cherry-picked as 2d4e60ca in /Users/james/Desktop/dev/gopher-orch: streamable HTTP OAuth elicitation retry fix.

Native prerequisite

• Update the Python native dependency/package to a gopher-orch build that includes the elicitation C API and the streamable HTTP retry fix.
• Verify the native headers expose:
  • gopher_orch_elicitation_request_t
  • gopher_orch_elicitation_callback_t
  • action constants for accept, decline, and cancel
  • gopher_orch_agent_options_supports_elicitation
  • gopher_orch_agent_options_t fields for elicitation_callback, elicitation_user_data, and elicitation_timeout_ms
• Confirm the bundled/platform native packages no longer carry the older libgopher-orch.0.1.32 behavior.
• Keep a clear error for native mismatch: if the caller uses elicitation options but the loaded native library does not support them, raise AgentError with an upgrade/rebuild message.

Public Python API

• Add gopher_mcp_python/elicitation.py.
• Define:
  • GopherAgentElicitationAction: accept, decline, cancel
  • GopherAgentElicitationRequest: elicitation_id, mode, message, url, request_id_json, raw_json, raw_params_json
  • GopherAgentElicitationResponse: action
  • GopherAgentElicitationHandler: synchronous callable protocol
  • GopherAgentElicitationOptions: handler, timeout_ms, open_browser
• Export the new types from gopher_mcp_python/__init__.py.
• Extend GopherAgentCreateOptions in gopher_mcp_python/runtime_options.py with elicitation.
• Accept both snake_case and JS-style camelCase mapping keys where the Python SDK already does this for OAuth, especially timeout_ms/timeoutMs and open_browser/openBrowser.
• Normalize empty/omitted elicitation to a default enabled options object, not None, for agent creation.
• Preserve explicit caller choices. A custom handler or timeout should flow unchanged to the FFI layer.

Runtime behavior

• Add gopher_mcp_python/elicitation_runtime.py or keep the implementation beside runtime option normalization if that better fits the repo.
• Convert native callback input into GopherAgentElicitationRequest with decoded UTF-8 strings and optional fields.
• Implement the default handler:
  • Handle only mode == "url" with a non-empty url.
  • Use the existing oauth_browser.open_authorization_url.
  • If browser opening is disabled or fails to open, print the authorization URL to stderr.
  • Prompt: Complete the OAuth flow in the browser, then press Enter to continue. Type "cancel" and press Enter to cancel.
  • Return accept on Enter, cancel on cancel, and decline for unsupported modes.
  • Respect timeout_ms when waiting for user input if practical on the supported platforms.
• Reject async handlers/coroutines. The native callback is synchronous, so returning an awaitable should raise a clear error or resolve to native cancel.
• Catch handler exceptions inside the FFI callback and return native cancel.
• Redact sensitive URL query parameters in debug logs, including code, token, access_token, refresh_token, id_token, client_secret, and state.

FFI work

• Extend gopher_mcp_python/ffi/library.py ctypes definitions:
  • Add GopherOrchElicitationRequest.
  • Add a CFUNCTYPE for the native elicitation callback.
  • Extend GopherOrchAgentOptions in native field order.
  • Bind optional gopher_orch_agent_options_supports_elicitation.
• Update _AgentOptionsStorage:
  • Own encoded strings and header arrays as it does today.
  • Own the Python callback object for as long as native may call it.
  • Store callback/user data/timeout on the options struct.
• Ensure callback lifetime is at least the agent lifetime, not only the create-call lifetime, if native stores the callback in the agent.
• Release retained callback storage when GopherAgent.dispose() releases the native handle, and clean it up on create failure.
• Keep current no-options behavior for older native symbols when no runtime/elicitation options are needed.
• Update missing-symbol messages to distinguish runtime headers support from elicitation support.

Agent integration

• Update every factory path in gopher_mcp_python/agent.py:
  • create
  • create_with_api_key
  • create_with_server_config
  • create_with_server_id
  • create_with_server_name
  • create_with_gateway_id
  • create_with_gateway_name
  • create_with_url
• The create-options merge should mirror JS: when the caller omits elicitation, inject default enabled elicitation options before converting to native runtime options.
• Do not require changes in examples/api/create_by_url.py or its runner. Default behavior should cover the common OAuth path.
• Preserve first-step OAuth behavior:
  • If runtime Authorization is present, skip SDK OAuth resolver as today.
  • If oauth.mode == "disabled", skip SDK OAuth resolver as today.
  • Still pass default elicitation options to native unless elicitation is explicitly disabled/cancel-only.
• Decide whether Python needs a JS-style agent.run() textual fallback:
  • Prefer the native callback path as the primary implementation.
  • Add a one-retry textual fallback only if the Python native/library combination can still surface provider authorization URLs as final text instead of invoking the callback.

Tests

• Add unit tests for elicitation option normalization:
  • omitted options default to enabled elicitation
  • explicit handler/timeout/open_browser is preserved
  • invalid action/mode/timeout inputs raise ValueError
  • oauth.mode == "disabled" does not remove default elicitation
• Add runtime tests:
  • URL-mode default handler opens browser and returns accept after Enter
  • manual fallback prints the URL
  • typed cancel returns cancel
  • unsupported mode returns decline
  • async handler is rejected
  • handler exception maps to cancel
  • URL redaction hides sensitive query values
• Add FFI tests in tests/test_ffi_runtime_options.py or a new tests/test_ffi_elicitation.py:
  • GopherOrchAgentOptions field order matches native
  • callback fields are populated when elicitation is enabled
  • no elicitation options does not require new symbols on old native
  • explicit elicitation requires native support
  • callback decodes request fields and maps action constants
  • callback storage is retained and released correctly
• Add factory tests:
  • all create paths pass default elicitation options
  • explicit runtime access token/header still passes elicitation
  • SDK OAuth resolver output does not drop elicitation
• Add regression coverage for the second-OAuth hang:
  • Use a mocked native callback path if live OAuth credentials are unavailable.
  • When credentials are available, run a live create-by-url profile query against the test gateway and verify the flow completes after provider OAuth.

Examples and docs

• Keep examples/api/create_by_url.py unchanged unless it has Python-specific bugs unrelated to elicitation.
• Document that elicitation is automatic by default.
• Document custom handler usage for non-interactive hosts.
• Document how to disable browser opening while still printing the URL.
• Add changelog/release notes describing provider OAuth elicitation support and the required native version.

Verification commands

• python -m pytest tests/test_agent_runtime_options.py tests/test_ffi_runtime_options.py
• python -m pytest tests/test_agent_create_with_oauth.py tests/test_oauth_create_with_url_integration.py
• python -m pytest tests/test_oauth_auto_custom_idp.py
• Full suite: python -m pytest
• Live smoke test, when env credentials are available:
  • examples/api/create_by_url_run.sh "get mail profile, using get_gmail_users_get_profile"

Acceptance criteria

• No example-level elicitation config is needed for provider OAuth.
• The second OAuth URL is surfaced to the user, browser/manual flow is supported, and Enter resumes the tool call.
• The run completes instead of blocking after provider authorization.
• Native session reuse does not use the discovery streamable HTTP session when elicitation is active.
• Async handlers cannot silently hang the native callback.
• Secrets and OAuth codes are not printed in debug output.
• Older native libraries fail clearly only when elicitation support is actually required.