Uh oh!
There was an error while loading. Please reload this page.
feat(fleet): the Fleet Manager — run records, conductor fleet, and an interactive TUI - #431
Merged
Merged
Conversation
Every run path now writes a run_id-keyed JSON run record describing its mode/PID/workflow/port, so foreground, --web and --web-bg runs are all discoverable the same way. This fixes the bug where a plain `conductor run` was invisible to `conductor stop` and to discovery, which only ever saw port-keyed .pid files written by --web-bg. - fleet/records.py: RunRecord + atomic write/read/prune primitives, tolerant of corrupt JSON and legacy port-keyed .pid files. - fleet/summary.py: RunSummary derived from a bounded event-log tail (status, current step, elapsed, tokens, cost, open gate). Status comes from explicit event markers, never inferred from timing. - fleet/history.py, fleet/retention.py, fleet/launch.py: History enumeration, $TMPDIR/conductor event-log retention, and New Run launch reusing cli/bg_runner.launch_background. - fleet/tui/: Textual app with Runs, Run detail, Providers, Registries, New Run and History screens, plus shared stop/kill/gate actions and debounced terminal notifications. Behind the optional `tui` extra. - cli/fleet.py: `conductor fleet` (bare launches the TUI), `fleet list`, `fleet prune`; `conductor stop` now operates over run records. - settings.py: read-only ~/.conductor/config.toml ([fleet.retention]). - cli/pid.py is retained only so pre-upgrade background runs stay discoverable; parent-side PID writing is gone. Squashed from the E1-E15 epic series; that granular history is preserved at tag backup/fleet-manager-pre-rebase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stry The New-run screen had no CSS at all, so Textual laid every widget out `width: auto`: a field's label carried the workflow's entire description on one unwrappable line and ran off the right edge, an unlabelled Checkbox collapsed to a truncated "X…" with nothing to say which field it belonged to, and the Launch button sat below a full-height scroller rather than where it could be seen. - Split each field into a heading (name, type, required), a wrapping muted description line, and the widget itself. - Carry the field name as the Checkbox's own label. - Add DEFAULT_CSS: `width: 100%` is what makes a description wrap, and a `1fr` field list keeps the launch bar on screen. - Focus the first field once a workflow resolves. Also bind `n` on both Registries drill-down screens (the workflows list acts on the highlighted row, matching `k`/`g` on Runs) to open New Run with `<workflow>@<registry>` pre-filled and resolved -- launching the workflow you are looking at no longer means escaping back out and retyping a reference you just navigated through. That makes New Run reachable three levels deep, where popping a single screen on a successful launch would land back on a workflow's inputs with the just-started run nowhere in sight, so `FleetApp.return_to_runs()` unwinds to Runs instead -- where the run appears on the next poll tick. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Five of the six screens carried no CSS at all, so Textual laid them out with raw defaults: tables flush against column 0, no spacing, and every cell the same visual weight as its neighbours. The home screen showed three rows of runs above seventeen blank lines, and the footer silently truncated mid-word, dropping the History binding entirely. Design system: - tui/theme.py is now the single status vocabulary. runs.py, history.py and run_detail.py each defined their own overlapping glyph map, none of which carried colour, so a failure and an unknown outcome landed with identical weight. Colours are Rich style names, not Textual CSS variables -- these render inside Text objects, where $success would be printed literally. - App-level CSS plus a default theme, applied in on_mount rather than as a `theme = ...` class attribute: App.theme is a reactive, and assigning a plain string in the class body replaces the descriptor, which leaves the app looking themed while silently breaking theme switching. - Screens inherit $background, not $surface. Surface is the *elevated* colour (nord #3B4252 vs #2E3440), so painting screens with it produced a flat haze with dim text on top. - `unknown` is left unstyled rather than dim grey: it is History's most common row, and dark-grey-on-dark was near invisible. Run preview: - A summary bar (counts by status, token and cost totals) and a preview pane showing what the one-line row has no column for: PID, port, an open gate's prompt and options, and the workflow's step chain with the current step marked. - The table is height:auto capped at 60% rather than 1fr, which is what removed the dead band between it and the pane. - The trailing column is Mode, not Port: a blank port was the only signal that a run was foreground, and blank reads as "no data yet". New-run form: the Resolve and Launch buttons are gone. Enter already resolved the reference, and a form whose submit control sits below its own scroll region is worse than one submitted from anywhere; both are keystrokes now (ctrl+r / ctrl+s), with a hint line that says why a launch is not yet possible instead of grey ing out a control. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t log Two failures seen against a real background run. Dashboard would not open. On WSL there is no X display and usually no Linux browser, so `webbrowser.open` falls through to `gio`, which answers "Operation not supported" and opens nothing. WSL is now detected and the URL handed to the Windows side -- `wslview` when the `wslu` package is installed, otherwise PowerShell's Start-Process. `explorer.exe` also works but exits non-zero on success, so it cannot report whether it opened anything, which is exactly what the caller needs. The caller made that invisible: it discarded `webbrowser.open`'s return value and notified "Opened dashboard" unconditionally, so a failed open still reported success. It now surfaces the failure and prints the URL, which is the part the user actually needs when no browser will open. No run details. A pre-Fleet-Manager `.pid` file records no event-log path, so current step, tokens, cost and topology all came out blank -- while the log sat on disk beside it. The record does carry the run id and the log's filename ends in it, so the pairing is recoverable. Only an unambiguous match is adopted: run ids are 8 hex chars and a resumed run reuses its predecessor's, so one id can legitimately name several logs (a test suite pinning one produces dozens), and showing a run's details against another run's log is worse than showing none. The lookup does not go through `retention.event_log_root()` -- that creates the directory because it is about to delete inside it, and this is a read-only path that may not exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ll-down Builds the Fleet Manager TUI out from a bare table into something you can actually read a running fleet from. Presentation is factored into leaf modules rather than inlined in the screens that use them: `anim.py` drives every moving glyph from one screen-level frame counter (so they stay in phase and there is one timer, not one per widget), `art.py` holds the whitespace-significant banner art that cannot be edited safely inline, and `dag.py` renders where a run is in its workflow -- shared by the Runs preview and the run-detail header, which both had to answer that question. Two new screens: a splash that covers the run-record scan and per-run bounded log read the first Runs render already blocked on, and a step drill-down (`enter` on a run-detail row) answering what a single step actually did -- its input, output and activity stream -- which neither table above it could show. `summary.py` grows `derive_step_detail` and a bounded head read to feed it, and `records.py` grows `find_event_log_for_run` so a run's log can be located from its id and start time. Runs-screen keys are split into two blocks -- row-scoped (`enter`, `w`, `k`, `g`) then fleet-scoped (`n`, `p`, `r`, `h`, `K`, `q`) -- with `BlockFooter` drawing the rule between them, since ordering alone renders as one undifferentiated run of keys. `enter` needs `priority` to appear at all: DataTable binds it first, hidden, and silently shadowed it. Row-scoped keys retire while the fleet is empty. `K` (kill *all*) moves into the fleet block, away from the single-run `k` it sat one stray Shift from. Also fixes a real engine bug the fleet view exposed: a `questions` node emitted `gate_presented` and never the matching `gate_resolved`, so every consumer tracking "is this run waiting on a human" -- the fleet summary and the dashboard both -- showed the run parked at a question it had already answered, for the rest of the run. `workflow_started` now also records the inputs a run was launched with, so two runs of one workflow are distinguishable in the log by something other than their ids. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`bg_runner` forwards workflow inputs to the child via `--input-json`, but
`conductor run` never declared that option -- so every `--web-bg` launch
carrying an input died immediately with exit code 2 ("No such option:
--input-json"). That took the Fleet Manager's New Run screen with it: its
form is generated from the workflow's declared `input:` block, so it
essentially always passes some.
The flag is load-bearing rather than incidental. The New Run form coerces
each field to its declared type (`fleet/launch.py::_coerce_input` returns a
`string`-typed value verbatim) and nothing downstream restores that typing
-- the engine's `_apply_input_defaults` only fills in *missing* inputs, so
whatever the child's CLI parse produces is final. Routing these through the
public `--input` flag would hand them to `coerce_value`, whose command-line
heuristic re-guesses a bare `true`/`42`/`[1,2]` into a bool/int/list and
silently discards the declared type.
`_serialize_input_value`'s docstring claimed `coerce_value` tries
`json.loads` first, so a JSON-quoted string would decode back to a string.
It does not -- it only tries JSON for values starting with `[` or `{`, so
every string arrives with its quotes baked in. That false comment reads as
though `--input` already handled this, which is the likeliest reason the
option was never wired up; it now describes the real mechanism.
The existing tests missed all of this because they patch out
`subprocess.Popen` and assert on the argv the launcher *builds*, which only
proves it built what the test expected. `TestLaunchedArgvIsAcceptedByTheChildCLI`
checks every flag the launcher emits against the child command's real,
declared Click options, catching argv/CLI divergence without paying for a
subprocess.
Also promotes the Fleet Manager to a headline README section alongside the
web dashboard -- screenshot, key features, and the breadth-vs-depth split
between the two surfaces -- and trims the CLI-reference entry to a pointer
rather than a second copy.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>Nine blocking defects from the review of this PR, plus the highest-value recommendations. Data loss and correctness: - An unrecognised run-record `mode` no longer raises. A raise reached `_read_and_prune` as `corrupt`, which deletes **without consulting liveness** -- so a record written by a newer Conductor made an older one delete a live run's record and orphan that process from `stop`, `status` and the fleet. It now normalises prefix-wise (`fg*` to `fg`, else `bg`), which also keeps D1's stop confirmation armed rather than silently dropping it. `mode` is a `RunMode` Literal so the single write site is checked by `ty`. - `stop` no longer clears a legacy PID file by port alone. The port-keyed fallback fires on the *normal* path (a cooperating child removes its own record), so a freed-and-rebound port meant deleting a different, live run's file -- the hazard issue #344 exists to prevent, under a comment claiming the removal was identity-checked. Adds `remove_pid_file_for_pid`, which matches on port *and* pid and delegates to `remove_pid_file_at`. Reporting a success that did not happen: - `conductor fleet prune` reported a wholesale sweep failure -- including the symlink-tamper refusal -- as "Nothing to prune." with exit 0. `PruneResult.error` carries it; `prune_event_logs` still never raises, since the opportunistic startup sweep depends on that. - Per-file deletion failures had nowhere to go, so a read-only log directory reported a working sweep. `_safe_unlink` now distinguishes "already absent" from a refusal, and `PruneResult.failed` is reported. - The TUI kill action announced "Killed 0 run(s)." at *informational* severity while discarding every reason -- including the identity-mismatch refusal, a safety stop -- into a dropped buffer. `StopOutcome.failed` carries them to the user. Rich markup (AGENTS.md, Console Output): - The kill-confirmation dialog markup-parsed the workflow names it was listing for deletion; two gate notifications and two more sites did the same; `fleet prune --help` dropped the `[fleet.retention]` section name it existed to communicate. - `test_markup_guards.py` gains rules I and J for the surfaces that let those through: Textual content sinks (`Static`/`Label`/`notify`, and every `str` cell of a `DataTable`) and Typer command docstrings. Both were verified against the pre-fix tree. Rule I found a live instance of the same defect in the run-detail table, fixed here. Test isolation: - `read_run_records()` is a pruning reader, and `cli.pid.pid_dir()` ignores `CONDUCTOR_HOME`, so the suite could delete a developer's live run records. An autouse fixture redirects both directories, mirroring the existing `_isolate_event_log_root`. Also: `stop --json` takes D1's refusal branch instead of skipping the gate; foreground self-exclusion works off Linux via a dedicated `CONDUCTOR_SELF_RUN_ID` (a separate name because `event_log.py` reads `CONDUCTOR_RUN_ID` as "adopt this run id"); an inherited `SIG_IGN` is honoured; the event-log head read is lazy; `textual>=8.0` to match the tested floor; dead code and two inert parameters removed; docs, CHANGELOG and AGENTS.md corrected where they described code that no longer exists. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolves the overlap between the Fleet Manager's run-record launch gate (D2) and the two-stage readiness contract main added in #410/#417. Both survive; they are complementary rather than alternatives: - Stage one (`_wait_for_server`) is unchanged. - The parent-side `write_pid_file` main wrote between the stages is replaced by this branch's poll for the *child's* run record. That is a strictly stronger signal -- the child only writes the record once it is executing -- and preserves D2's single-writer invariant, which a reinstated parent-side write would break. - Stage two (`_wait_for_workflow_start`, `/api/info`) is retained, so `workflow_started` / `still_running` still distinguish "listening" from "actually started" and from "already exited". - The stage-two failure cleanup moves from `remove_pid_file_at` to a new `_remove_dead_child_record`, identity-checked on `pid` for the same reason (issue #344): a resumed launch can carry a checkpoint's original `run_id`, so the record under that key may belong to a live process. Test resolution: both sides' new test classes are kept. Main's stage-two tests are re-pointed from the PID file to the run record. Nineteen with-blocks that predate stage two now skip it explicitly -- without that they ran the real 30s `/api/info` wait, taking `test_bg_runner.py` from 2s to 4m33s. Docs: AGENTS.md's `bg_runner.py` bullet describes the merged three-stage sequence; the env-var table keeps main's rewritten `CONDUCTOR_GATE_TOKEN` row (#424) and `CONDUCTOR_WEB_ALLOW_ORIGINS` alongside this branch's `CONDUCTOR_HOME` and `CONDUCTOR_FLEET_NO_ANIM`; CHANGELOG keeps both sides' entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert (jrob5756)
marked this pull request as ready for review
August 14, 2026 13:08
The Windows CI job added in #390 exercised this code for the first time and found five failures. One is a real defect. `write_run_record`'s "atomic write" relied on `os.replace`, which is atomic and reader-proof on POSIX but raises `PermissionError` on Windows while another process holds the destination open. This record is read constantly -- `conductor status`, `fleet list`, the TUI's ~2s poll, the `--web-bg` launch gate -- so the write failed, `cli/run.py` swallowed it, and the run silently became undiscoverable and unstoppable: exactly the defect the run record exists to prevent, reachable only on Windows. `_replace_with_retry` retries briefly there; a persistent permission error still surfaces rather than stalling. Predicted as R13 in review and deferred as unlikely -- CI reproduced it deterministically. The remaining four were tests asserting POSIX-only behaviour: - `TestStopProcessUnexpectedOSError` and `test_kill_already_dead_pid_does_not_raise` simulate an errno from the kill primitive by patching `os.kill`. `terminate_process` dispatches to `_terminate_process_windows` (ctypes `TerminateProcess`) there, so the patch is never reached and the assertions describe a path Windows cannot take. Skipped with that reason. Note the sibling `..._removed_when_process_confirmed_dead_after_oserror` was passing on Windows for the same wrong reason, and is covered by the class-level skip. - `test_concurrent_writers_never_yield_torn_read`'s reader now tolerates `PermissionError` alongside `FileNotFoundError`: both are races with `os.replace` itself, not the *torn* read the test exists to rule out. - `_rendered_footer` assumed the footer is the compositor's last strip. It is on Linux; on Windows the trailing strip is blank, so every footer assertion read an empty line. It now finds the row carrying a binding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`_rendered_footer` read the compositor after a single `pause()`. That is enough on Linux; on Windows the first frame can land before the footer is composited, so the assertion read a blank line -- the last remaining Windows CI failure, and unrelated to the binding grouping these tests cover. It now polls until the footer row appears. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Aug 14, 2026
Jason Robert (jrob5756) added a commit
that referenced
this pull request
Aug 14, 2026
Relocates the six changelog entries from #431, which a stale-branch merge placed inside the already-released 0.1.28 section, into the new 0.1.30 section under the correct Keep a Changelog headings. Previously released sections are restored byte-for-byte. Co-authored-by: Jason Robert <jasonrobert@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert (jrob5756) added a commit
that referenced
this pull request
Aug 14, 2026
The Windows job has been failing on main since #431 landed. Not a test failure -- 7,164 passed, 0 failed -- the step hit its 10-minute limit at 9m56s and was killed. #431 added ~600 tests, most of them Textual `App.run_test()` pilots, and Windows runs the suite at roughly twice the Linux wall time: the step went from ~5m to ~10m and landed right on the boundary. It passed at 9m42s on the PR and failed at 9m56s on main, which is the same run either side of a coin flip. There is no single slow test to fix -- `tests/test_fleet` is 522 tests in ~90s locally, ~0.17s each, which is ordinary for pilot tests. So this raises the limit rather than pretending the suite did not grow. 20 minutes is ~2x headroom on Windows and ~3x on Linux: still short enough to catch a genuine hang, instead of the current setting failing a green run. Also caches the AST parse in `test_markup_guards.py`. Its ten source scans were each re-parsing all 116 files under `src/conductor`, sixteen full passes in total (14.6s -> 11.6s). The trees are only read, so one parse serves every rule. Co-authored-by: Jason Robert <jasonrobert@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug this starts from
Only
--web-bgruns wrote a discoverable record (a port-keyed.pidfile), so a plainconductor run— or evenconductor run --web— was invisible toconductor stopand had to be killed by hand.Every run path now writes a
run_id-keyed JSON record to~/.conductor/runs/<run_id>.jsondescribing its mode (fg/fg-web/bg), PID, workflow path and dashboard port.stop,fleet listand the TUI all read that one store. The legacy.pidfile is still read (and cleaned up) so a pre-upgrade process stays killable, but nothing writes it any more.What's here
conductor fleet list/prune— no optional dependency required.conductor fleet— an interactive Textual TUI behind thetuiextra (pip install 'conductor-cli[tui]'): Runs (home, ~2s poll), Run detail, Step detail, Providers, Registries, New Run, History.~/.conductor/config.toml— machine-wide, read-only in v1, currently[fleet.retention], which bounds the previously unbounded$TMPDIR/conductor/*.events.jsonlgrowth.g) for any run with a dashboard port via the existingconductor gate respondHTTP path. A plain foreground run's gate is display-only — its blocking prompt thread can't be reached remotely — so the PID is shown instead.Behaviour changes worth review attention
SIGTERMdiscards in-flight progress unless periodic checkpoints are on, sofg/fg-webrequires confirmation.--yes/-yskips it; non-interactive stdin without--yesrefuses rather than defaulting to yes. Background-only fleets are unaffected.stopgained--run-id— the only selector that can target a foreground run, which has no port to match on.conductor fleetdeliberately breaks the sub-app pattern.checkpoint/gate/registryuseno_args_is_help=True;fleetusesinvoke_without_command=Truebecause the bare command is the TUI. Noted in AGENTS.md so it doesn't get "fixed".Design constraints held throughout
gate_presented/gate_resolved/workflow_completed/workflow_failed) — measurement during design found timing inference gave 228 false positives and 0 true positives.cli/bg_runner.launch_background, gate resolution reusescli/gate.py, provider diagnostics reuseproviders/diagnostics.py, replay delegates toconductor replay.Engine fix included
A
questionsnode emittedgate_presentedand never the matchinggate_resolved, so every consumer tracking "is this run waiting on a human" — the fleet summary and the web dashboard — showed the run parked at a question it had already answered, for the rest of the run.human_gatehad always paired the two. Also,workflow_startednow records the inputs a run was launched with, so two runs of one workflow are distinguishable by something other than their ids (no new exposure — the rendered prompts already in that log interpolate the same values).Latest commit
Presentation is factored into leaf modules rather than inlined:
anim.pydrives every moving glyph from one screen-level frame counter (one timer, all glyphs in phase),art.pyisolates whitespace-significant banner art,dag.pyrenders where a run sits in its workflow. Plus a splash covering the scan the first Runs render already blocked on, and a step drill-down answering what a single step actually did.Runs-screen keys are split into row-scoped (
⏎ w k g) and fleet-scoped (n p r h K q) blocks, withBlockFooterdrawing the rule between them — ordering alone rendered as one undifferentiated run of keys. Two notes for reviewers:⏎needspriority=Trueto appear at all:DataTablebindsenterfirst withshow=Falseand silently shadowed it.K(kill all) moved into the fleet block, away from the single-runkit sat one stray Shift from.Textual 8.x's native binding groups were the obvious tool here, but its
Footerrenders a group by dropping every per-key description in favour of one group label — turningw Dash k Killinto a barew k. The keys are exactly what a newcomer can't guess, so this keeps the labels and draws the divider instead, reusing thevkeyborder Textual's own stylesheet uses for the docked palette key.Testing
Full suite: 6692 passed, 47 skipped. Lint, format and
tytypecheck all clean.Fleet tests cover record read/write/prune tolerance, retention, status derivation, History enumeration, launch, run-record wiring, notification debouncing, and per-screen Textual
App.run_test()pilot tests. The footer work is pinned by tests asserting the compositor-rendered line rather than the marker class — an earlier attempt applied the class correctly while painting nothing, which only a rendering assertion catches.test_footer_fits_without_truncationguards the single non-wrapping footer line, which has previously dropped a key off the edge unnoticed.