Skip to content

feat(mcp): conductor mcp serve — expose workflows as MCP tools (#432) - #491

Draft
Jason Robert (jrob5756) wants to merge 18 commits into
mainfrom
docs/432-mcp-serve-design
Draft

feat(mcp): conductor mcp serve — expose workflows as MCP tools (#432)#491
Jason Robert (jrob5756) wants to merge 18 commits into
mainfrom
docs/432-mcp-serve-design

Conversation

@jrob5756

@jrob5756Jason Robert (jrob5756) commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Closes#432.

Adds conductor mcp serve: a stdio MCP server that exposes registered Conductor workflows as MCP tools to any MCP-compatible host (Claude Code, VS Code, Cursor, your own agent), with run introspection and diagnostics.

What this does

Zero-edit exposure. Every workflow in every configured registry is exposed by default, with a typed inputSchema derived from its own input: block. No workflow needs editing to be reachable — a workflow with no mcp: block is exposed identically to one declaring the defaults explicitly.

Invocation is always a real detached run. A tool call forks conductor run via the existing launch_background path — the server never executes a workflow in-process. It waits up to a bounded per-call timeout (--max-wait-seconds) and then returns either a handle (run_id + dashboard url) or the completed run's output inline, spilling to a resource_link once serialized output exceeds 50 KB. Human gates are never auto-skipped: a run that reaches one parks and reports its dashboard approval URL until a person resolves it.

Run lifecycle tools (conductor_run_status / await_run / cancel_run / list_runs) answer for a run_id before, during, at a gate, and after a run has finished. Answering after it finished is what motivated the terminal run record (below).

Optional toolsets, off by default:introspect (event query, per-step detail, plan tree) and diagnose (doctor/validate equivalents, links — never contents — to raw logs). Raw tool-call payloads are withheld unless --introspect-full.

Degrades rather than overflows. A registry above --max-direct-tools (default 25) publishes a two-tool discovery pair (conductor_find_workflow / conductor_run_workflow) instead of failing or blowing past a host's tool-count limit. The mode is decided once at startup and never varies within or across a connection.

Notable side-effects on existing surfaces

  • conductor status and conductor fleet list now also list recently-completed runs — a deliberate contract change. Previously both meant "alive right now" and a finished run vanished the instant its process exited, which left the MCP run tools with nothing to answer from. Every run now writes a terminal record on exit; --live restores the exact previous scope, and status --json --live drops the additive completed key entirely so existing scripted consumers of payload["running"] are unaffected.
  • conductor doctor gains an mcp section showing what mcp serve would expose (tools, collisions, rejected workflows, failed registries) without starting a server.
  • The mcp SDK dependency is now bounded below 2.0 (mcp>=1.28.1,<2). mcp 2.0.0 renamed the camelCase attributes the existing MCP client reads (Tool.inputSchema → input_schema), so any install whose lock had floated to 2.x had a client that connected and then raised AttributeError on every tool listing — MCP tools were silently non-functional.

New config

workflow.mcp: — per-workflow expose / mode / read_only / destructive / estimated_minutes. Every field defaults to current behavior; an unknown key inside the block is a conductor validate schema error rather than a silently ignored typo. See examples/mcp-serve.yaml.

Shape of the change

70 files, ~19k insertions. New package src/conductor/mcp/serve/ (catalogue, naming, pinning, toolgen, invoke, runs, introspect, diagnose, discovery, server, options, sanitize) plus cli/mcp.py. Roughly half the diff is tests (tests/test_mcp/test_serve_*.py, tests/test_registry/, tests/test_fleet/test_terminal_records.py, tests/test_cli/test_doctor.py).

Design and implementation plan are committed alongside the code:

  • docs/projects/mcp-server/conductor-mcp.design.md
  • docs/projects/mcp-server/conductor-mcp.plan.md (epics E1–E14, all marked done)

User-facing docs: docs/mcp-server.md (with an explicit Limits section for what v1 deliberately does not do — no outputSchema, stdio only, no Streamable HTTP), cross-linked with the existing client-side docs/mcp-tools.md.

Status

Draft. make check and make validate-examples are green; make test is green except one pre-existing, environment-dependent failure unrelated to this work.

Not yet rebased — the branch is currently ~28 commits behind main. I'll bring it up to date before marking ready for review.

Jason Robertand others added 18 commits August 14, 2026 13:02
Jason Robert
docs(mcp): add solution design forconductor mcp serve(#432)
b6c5b11
Adds the engineering/architecture solution design for exposing a user's
registry workflows as MCP tools, so any MCP host (Claude Code, VS Code,
Cursor) can invoke a governed, routed, budget-capped, checkpointed
workflow as a typed tool call instead of driving the CLI through a
hand-written skill.
Covers problem statement, goals/non-goals, requirements, the proposed
design, alternatives considered, dependencies, impact, security, risks,
and open questions. Absorbs and supersedes #135.
Every claim about Conductor's own code is cited by file and symbol
against `main` at 0554517; MCP spec and Python SDK claims are checked
against primary sources, with several inherited from the source issue
corrected inline. Two load-bearing SDK behaviours were verified
empirically across both major versions (DD0).
Filed under docs/projects/<project>/ to match the existing design docs
(aca, fleet-manager, web-ui, agent-sdk).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
docs(mcp): add implementation plan forconductor mcp serve(#432)
b8f4854
Consumes the already-reviewed solution design and breaks it into 14
epics (E1-E14): bounding the `mcp` SDK dependency, the terminal run
record and its retention, surfacing completed runs in `status` /
`fleet list` / History, registry index fields and parse cache, the
`mcp:` workflow block, the catalogue builder, the stdio server,
detached invocation with a bounded wait, run lifecycle tools,
`introspect` / `diagnose`, discovery above the tool cap, the
`doctor` MCP section, and docs.
Every path, symbol, and line reference is grounded against the tree at
b6c5b11, and the pinned mcp 1.28.1 SDK surface was exercised in this
repo's own venv rather than taken from the design's report. Four gaps
the design left open were put to a stakeholder and are recorded as
plan-level decisions R1-R4.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E1: Bound the mcp SDK dependency (DD0)
0f38f31
Bound the mcp SDK dependency to >=1.28.1,<2 in pyproject.toml to
prevent a lock refresh from silently pulling mcp 2.x, which renamed
the camelCase Tool.inputSchema/CallToolResult.isError attributes read
by the existing MCP client (mcp/manager.py:207) — a runtime
AttributeError the module's except ImportError guard cannot catch.
- Re-resolved uv.lock (mcp stays pinned at 1.28.1)
- Added tests/test_mcp/test_sdk_bound.py regression suite asserting
the SDK attribute surface and the declared specifier bound
- Documented the fix in CHANGELOG.md under Unreleased/Fixed
- Marked all five E1 tasks (E1-T1..T5) and acceptance criteria as
DONE in docs/projects/mcp-server/conductor-mcp.plan.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E2: The terminal run record (DD1, P3, G5)
90c359a
Add TerminalRunRecord (frozen dataclass, fully absence-tolerant
from_dict) and its read/write/remove API in fleet/records.py, stored
under run_records_dir()/terminal/ — a subdirectory deliberately
invisible to the three functions that non-recursively glob
run_records_dir(). Wire the write into both run_workflow_async and
resume_workflow_async in cli/run.py: each captures its terminal
status/output/error on every exit path (clean success, explicit
WorkflowTerminated, or an unexpected exception) and writes the
tombstone in the existing finally block, immediately before the live
record is removed, using a never-raising helper. Token/cost/unpriced-
agent totals are read unconditionally from
engine.get_execution_summary()['usage']. A resumed run replaces its
predecessor's terminal record rather than duplicating it. A process
that is kill -9'd (or otherwise dies before the finally runs) is
documented and tested to leave no terminal record.
Mark E2 DONE in the MCP server plan (epic status, all seven task
rows, acceptance criteria).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E3: Terminal-record retention (DD13, FR12)
0a4ac46
Bound terminal run records by [fleet.retention].keep_last, pruned in the
same sweep as the event log they point at, so a run_id resolves
completely or not at all.
- Terminal run records are treated as a fourth companion of their events
log (alongside .bg.stderr.log/.bg.stdout.log), matched by the run_id
embedded in the events log's filename
- Added an orphan-only sweep for terminal records whose events log has
already disappeared, sorted newest-first by ended_at, sharing keep_last
and the keep_last < 1 guard
- Liveness sourced from the same _live_event_log_paths() call already
made by the main sweep (no second read_run_records() call)
- Fixed run_id extraction regex to correctly handle the full
alphanumeric/hyphen/underscore run_id contract instead of only hex
characters
Files: src/conductor/fleet/retention.py, tests/test_fleet/test_retention.py,
docs/configuration.md, docs/fleet.md, docs/projects/mcp-server/conductor-mcp.plan.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E4: Surface completed runs in status, fleet list, and History
94aefdf
- conductor status and conductor fleet list now surface recently-completed
runs alongside live ones, bounded by [fleet.retention].keep_last, with a
new --live flag on both to restore the pre-change 'live only' scope
- HistoryEntry gained output/error_type/error_message fields, enriched by
joining a matching TerminalRunRecord by run_id after the existing
single-pass log scan
- Fleet Manager TUI History screen surfaces failure reason / rendered
output via row selection, extending the existing replay-command
notification
- Fixed: completed rows in conductor fleet list's table now show
started_at (not ended_at) in the Started column
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E5: Registry index fields, offline ref pointer, and parse cache
1128f97
- T1: optional input/mcp fields on WorkflowInfo (config/schema.py, registry/index.py)
- T2: SHA-keyed parse cache (_meta/<sha>/tools.json + tools.complete sentinel)
via save_parsed_tools/load_parsed_tools (registry/cache.py)
- T3: offline ref->SHA pointer (_meta/_refs/<slug>.json) modelled on
plugins/fetch.py (registry/cache.py)
- T4: allow_network seam on fetch_workflow/fetch_workflow_adhoc/resolve_and_fetch
resolving from cache and raising typed RegistryError on a cold pointer
- T5: comprehensive tests including the load-bearing "every function in
registry/github.py patched to raise" test (tests/test_registry/test_cache.py)
- Fix: corrected test_a_build_that_ignores_the_fields_still_loads to exercise
a genuine legacy (pre-E5) model ignoring unknown fields, per review feedback
Includes a minimal McpConfig prerequisite type in config/schema.py for E5-T1;
E6 itself remains unimplemented.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E6: Implement the mcp: workflow block (DD4, FR11)
0de2964
- Wire WorkflowDef.mcp onto the existing McpConfig model in config/schema.py
- Add validator cross-checks for _wait_seconds reserved-input collision and
unslugifiable workflow names (_validate_mcp_exposure, slugify_workflow_name)
- Add _report_mcp(...) CLI reporting modelled on _report_plugins
- Document the mcp: block in docs/workflow-syntax.md
- Add examples/mcp-serve.yaml
- Add tests/test_config/test_mcp_block.py and extend tests/test_cli/test_validate.py
- Mark E6 DONE in docs/projects/mcp-server/conductor-mcp.plan.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E7: Implement catalogue builder — exposure, schema ladder, naming, pi…
4b74960
…nning
Turns registry/CLI configuration into a frozen, immutable list of
mcp.types.Tool objects at startup with zero process launching and zero
network I/O on a warm cache.
- options.py: frozen ServeOptions dataclass holding every startup
argument (registries, workflow_dirs, allow, deny, toolsets,
max_direct_tools, max_wait_seconds, tool_prefix, max_concurrent_runs,
introspect_full).
- naming.py: slugify() delegates to config.validator.slugify_workflow_name;
build_tool_names() computes base slugs, qualifies all colliding
identities with their registry (never only the loser, DD10), and
applies --tool-prefix last.
- sanitize.py: sanitize_description() strips control chars, invisible/
bidi-override Unicode, and instruction-marker shapes, and hard-caps
length at 500 chars (NFR4).
- toolgen.py: maps all 5 InputDef types to JSON Schema, injects the
reserved _wait_seconds parameter, rejects workflows that declare
_wait_seconds themselves, and publishes no outputSchema (DD5).
- pinning.py: Pin dataclass (sha for GitHub registries, content hash for
path registries and --workflow-dir); recheck helpers report drift
without mutating catalogue state (DD6, DD3).
- catalogue.py: build_catalogue() wires the four-rung exposure ladder
and three-tier schema ladder together, degrading unparseable
workflows to a permissive schema instead of dropping them (NFR2).
Marks E7 and all E7-T1..T10 tasks/acceptance-criteria DONE in
docs/projects/mcp-server/conductor-mcp.plan.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E8: Implement MCP stdio server (CLI, transport, tools/list)
b45d394
Adds `conductor mcp serve` Typer sub-app wired to the low-level
mcp.server.lowlevel.Server, publishing a byte-identical tools/list
response built from the frozen E7 catalogue over stdio (DD3). Keeps
stdout protocol-pure by routing all server-side messages, including
the FR10 startup summary (exposed tool/workflow counts, collisions,
degraded schemas), through a dedicated stderr console (DD9).
- New src/conductor/cli/mcp.py: `mcp serve` sub-app with --registry,
--allow, --deny, --workflow-dir, --toolsets, --max-direct-tools,
--max-wait-seconds, --tool-prefix, --max-concurrent-runs, and
--introspect-full flags
- New src/conductor/mcp/serve/server.py: catalogue -> lowlevel Server
wiring, stdio transport, startup summary
- Server reports Conductor's own package version in serverInfo.version
- FR10 collision summary names every distinct registry/workflow pair,
including same-registry collisions
- Registered mcp sub-app on the root app (Environment panel)
- Tests: in-memory stream pair drive of initialize/tools/list with
cross-connection identity check, CLI flag/help/stdout-purity tests
Marks epic E8 DONE in docs/projects/mcp-server/conductor-mcp.plan.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E9: Invocation: always detached, bounded wait, bounded fleet (FR4, FR…
70cc15b
…5, G3, G4, DD2, R3)
- Add build_typed_launch_inputs to fleet/launch.py for JSON-typed MCP inputs
- Extend max_concurrent_runs docstring in mcp/serve/options.py
- Add src/conductor/mcp/serve/invoke.py: tool dispatch, launch_background
invocation (web_port=0, hardcoded skip_gates=False per DD11), FR5 wait
resolution, bounded polling loop, result shaping (structuredContent +
text, resource_link spill per NFR6), and LaunchTracker for R3 concurrency
bounding
- Add tests/test_mcp/test_serve_invoke.py
- Mark E9 DONE in docs/projects/mcp-server/conductor-mcp.plan.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E10: Implement run lifecycle MCP tools (FR6, FR7, DD11)
999a88d
Add conductor_run_status, conductor_await_run, conductor_cancel_run,
and conductor_list_runs plus the shared resolve_run/RunLookup
three-source resolver in src/conductor/mcp/serve/runs.py.
- resolve_run(run_id) tries a live run (read_run_record +
derive_run_summary), then read_terminal_record, then
find_event_log_for_run as a crash fallback, naming which source
answered.
- conductor_run_status shapes each source into a uniform status dict,
including gate prompt/options/option_details and approval URL when
at a gate.
- conductor_await_run bounds polling at 2s cadence, returns early on
terminal or at-gate status, and emits progress notifications when a
token/sender are supplied.
- conductor_cancel_run reuses cli/app.py::stop_records for the
graceful stop ladder, reporting stopped/failed/already_terminal
honestly.
- conductor_list_runs unions live and terminal run records, dedupes by
run_id (live wins), and filters by status/workflow.
Updates docs/projects/mcp-server/conductor-mcp.plan.md: E10 and
E10-T1..T6 marked DONE, acceptance criteria checked.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E11: Implement introspect and diagnose MCP toolsets (FR8, DD12, NFR6,…
d87d3dd
… R4)
Add two opt-in MCP toolsets, both off by default (DD3):
- introspect (conductor_run_events, conductor_node_detail,
conductor_plan_tree) in src/conductor/mcp/serve/introspect.py
- diagnose (conductor_doctor, conductor_validate_workflow,
conductor_run_logs) in src/conductor/mcp/serve/diagnose.py
All six are thin adapters over existing Fleet Manager / diagnostics /
validate machinery. R4's redaction posture is applied specifically where
raw tool-call payloads live: conductor_run_events replaces
agent_tool_start.arguments and agent_tool_complete.result with
{name, status, byte_size} unless --introspect-full is set, while
conductor_node_detail returns prompt/output in full and is proven (not
assumed) to never carry a tool payload. conductor_run_logs follows DD12:
ResourceLink content blocks plus bounded per-file metadata, never file
bytes.
- --toolsets validation added at ServeOptions construction, rejecting
unknown toolset names at startup and surfacing the enabled set in the
startup stderr summary (E11-T1).
- Wired tools/list and tools/call for both toolsets in server.py, gated
on options.toolsets.
- Rejected tool-name collisions between generated workflow tools and the
introspect/diagnose tool set at build_server time.
- Fixed invoke.py's workflow-dir resolution to fail closed
(UnknownToolError) rather than ambiguously rescanning when a recorded
source path disappears.
- Fixed conductor_run_events' event_types argument handling to
distinguish an explicit empty filter from an absent one.
Updates docs/projects/mcp-server/conductor-mcp.plan.md: E11 and
E11-T1..T6 marked DONE, acceptance criteria checked.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E12: Implement discovery fallback above the tool cap (FR9, DD3, G7)
2d24c04
- discovery.py: conductor_find_workflow(query) searches the frozen
catalogue's own name/description/registry fields; conductor_run_workflow
dispatches through invoke_workflow_tool, the same invocation layer a
generated per-workflow tool uses, so a path-shaped/URL-shaped/registry-
source-shaped `name` is refused the same way any unrecognized tool name
is (NFR3) -- there is no separate shape check to bypass.
- server.py: build_server now acts on catalogue.mode (decided once, at
startup, by E7's build_catalogue): direct mode publishes the catalogue's
per-workflow tools as before; discovery mode publishes the fixed
conductor_find_workflow/conductor_run_workflow pair instead, never both.
Both tools/list and tools/call are gated on the mode captured in
build_server's closure, so it can never vary within or across a
connection (DD3). One LaunchTracker is shared per server process (R3).
- tests/test_mcp/test_serve_discovery.py: search/dispatch behavior, path-
shaped-name refusal, above/below-cap tool-list and tool-call gating,
mode stability across repeated calls/connections, and the startup log
naming the exposed count and --max-direct-tools threshold.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E12: Address review feedback on discovery fallback (FR9, DD3, G7)
c446dc9
- conductor_find_workflow: bound result count (_MAX_RESULTS=25) and
report total count/truncation
- conductor_run_workflow: validate flattened inputs against the
resolved catalogue entry's inputSchema before dispatch, raising
LaunchError on mismatch (parity with direct-mode tools)
- build_server: scope collision checks to tools actually published
together, so discovery mode no longer checks against hidden
catalogue names
- build_server/_call_tool/_dispatch_discovery_tool: thread the
request's progressToken and send_progress_notification into the
discovery pair's dispatch for end-to-end progress reporting
Resolves all four round-one review blockers. Focused discovery suite
(31 tests) and full MCP suite (327 tests) pass; ruff and ty clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E13: Add conductor doctor MCP section (Impact Analysis → Operational)
61a6c8b
Adds a new `mcp` section to `conductor doctor` that shows what
`conductor mcp serve` would expose as MCP tools, without starting a
server or attaching a host.
- Added `McpServeDiagnostic` + `gather_mcp_serve()` in
providers/diagnostics.py, wrapping the existing E7
mcp.serve.catalogue.build_catalogue() pipeline offline
(allow_network=False), wired into gather()/ALL_SECTIONS/DoctorReport.
- Added McpServeToolInfo, McpServeCollision, McpServeRejectedWorkflow,
and McpServeFailedRegistry dataclasses with to_dict() serialization.
- Added a thin Rich-table renderer `_render_mcp_serve()` in
cli/doctor.py following the existing `_render_registries` convention,
covering tools, collisions, rejected workflows, and failed
registries.
- Updated `conductor doctor --help` text/examples in cli/app.py to
mention the mcp section.
- Added FailedRegistry dataclass and Catalogue.failed_registries field
in mcp/serve/catalogue.py; threaded a failed_registries accumulator
through build_catalogue/_collect_registry_candidates so a
whole-registry resolution failure is captured structurally instead of
only logged.
- Marked E13 DONE in the MCP server plan document.
Targeted suite: 382 tests passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
E14: Documentation and release for conductor mcp serve
2f9f266
Documents the previously-undocumented conductor mcp serve feature
(epics E1-E13): a new user-facing guide (docs/mcp-server.md) with
host configuration snippets, the exposure ladder, toolsets, the mcp:
workflow block, run lifecycle, and an explicit Limits section; a
disambiguation/cross-link with the existing MCP client docs
(docs/mcp-tools.md); a CLI reference entry for conductor mcp serve
(docs/cli-reference.md); AGENTS.md architecture updates; and
CHANGELOG entries.
Also fixes two bugs surfaced during doc verification:
- direct-mode generated workflow tools are now dispatchable through
tools/call (previously listed but uncallable)
- the default runs toolset (conductor_run_status/await_run/cancel_run/
list_runs) is now both listed in tools/list and dispatched in
tools/call, matching documented default behavior
make check and make validate-examples are green; make test is green
except one pre-existing, environment-dependent failure unrelated to
this epic's changes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert
docs: restore web/ top-level bullet in AGENTS.md
a54ab17
E14-T4 replaced the `- **web/**: Real-time web dashboard for workflow
visualization` bullet with the new `- **mcp/**:` bullet instead of
adding it, orphaning auth.py/server.py/frontend/static (all of which
live in src/conductor/web/) as children of the mcp/ section and
leaving web/ undocumented as a top-level package. Restore the web/
bullet and drop the stray doubled blank line left by the insertion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 25, 2026
Open
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feature: conductor mcp serve — expose workflows as MCP tools, with run introspection and diagnostics

1 participant

@jrob5756