Uh oh!
There was an error while loading. Please reload this page.
fix(cli): make conductor stop confirm the process actually stopped - #383
Conversation
`stop` sent one best-effort signal, printed `Stopped`, and deleted the PID file unconditionally. On Windows that signal reliably fails -- `CTRL_BREAK_EVENT` needs a shared console, which a separate `stop` invocation does not have -- so the usual outcome was: workflow still running, user told it stopped, PID file gone. The run became an untracked orphan still consuming tokens, with no supported way to find it again. Replace the single signal with an escalating ladder, each rung followed by a bounded wait: graceful cancel via `POST /api/kill` (the only rung that lets the run checkpoint), then the platform signal, then forceful termination. A PID file is now removed only once its process is confirmed dead, and the command exits 2 when anything survived. Forceful termination is gated on confirming identity against the run's own dashboard. Terminating by PID with no identity check would be strictly worse than the old bug: the old broken signal killed nothing, whereas `TerminateProcess` on a recycled PID kills a bystander. `run_id` already existed in the PID-file schema and in `/api/info` but the launcher never populated it, so it was always empty -- wire it through `_spawn_bg_child`. Also: - `Liveness` tristate replaces the alive/dead bool internally, so 'probe failed' can no longer be mistaken for 'confirmed running'. The bool wrappers are retained, so the microsoft#166 listing behaviour is unchanged. - Removal is by file path with a re-read identity check, not by port. Port matching could unlink a *newer* run's registration if it took over the port. - `--json` for automation, `--force` for the unidentifiable case. - On Windows the forceful path terminates and waits on the *same* handle, so the confirmation cannot be fooled by PID recycling mid-call. Tests: new `test_stop_ladder.py` covers the ladder, the identity gate and the removal rules; both central guards were verified to fail on a simulated revert. Four tests in `test_stop.py` asserted the old contract (that `stop` reports success merely because `os.kill` did not raise) and were rewritten to assert the new one. Refs microsoft#344
…atch Post-implementation review found the identity check could not succeed for `conductor resume --web-bg`, which is the launch path most likely to need stopping. The launcher generates a fresh run id for the PID file, but a resumed child reuses the checkpoint's original run id -- `event_log.py` returns early on `existing_run_id` before ever consulting `CONDUCTOR_RUN_ID`. `/api/info` therefore reported an id that could never match, so every resumed run was treated as somebody else's process: the graceful rung was skipped, a misleading 'a different run took over this port' warning was printed, and on Windows the forceful rung was refused, leaving the run unstoppable without `--force`. POSIX masked it because SIGTERM terminates the child before the gate is reached. Identify by PID instead. The dashboard runs in the same process as the workflow, so `/api/info` now reports `os.getpid()` unconditionally. That is direct proof of ownership, is identical for run and resume, and is available immediately rather than only after `workflow_started` -- which also fixes identification of runs killed during startup, and of PID files written by older versions that have no run id. Run-id comparison is retained as a fallback for dashboards that predate this change. Also from review: - A positive identity *mismatch* now blocks the polite signal too, not just forceful termination. Refusing only the forceful rung was incoherent: on POSIX the polite rung is SIGTERM, which can kill the very bystander the gate exists to protect. `_confirm_identity` returns a tristate so that 'unconfirmable' (no evidence) stays distinct from 'mismatched' (positive evidence) -- unconfirmable still signals, as the previous implementation always did, so legacy PID files do not regress. - Identity is re-confirmed immediately before the irreversible rung, since seconds of waiting elapse after the first check and the PID could be recycled in that window. - `terminate_process` returning UNKNOWN is reported as `unconfirmed`, not `survived`. A failed probe is not evidence the process lived. - Ambiguous target (multiple runs, no --port/--all) now exits 1. It stopped nothing, so it must not report success. - `test_pid_file_is_removed_when_process_confirmed_gone` patched `app.os.kill`, which mutates the shared `os` module and would also break `pid.py`'s probe on POSIX -- it passed on Windows and would have failed on Linux CI. It now patches `process_liveness` directly. - The `--all` survivor test keyed outcomes off call order, which depends on `Path.glob` ordering; it is now keyed off the pid. Guards re-verified against three separate simulated reverts (unconditional unlink, no identity gate, run-id-only identity); each fails, and the fixed tree passes. Refs microsoft#344
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #383 +/- ##
=======================================
Coverage ? 91.71% =======================================
Files ? 109 Lines ? 17806 Branches ? 0 =======================================
Hits ? 16330 Misses ? 1476 Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Codecov flagged 70% patch coverage, with 51 uncovered lines in pid.py. The gap was exactly the code that does the killing: the ladder tests patch the primitives out to keep control flow readable, so `terminate_process`, `wait_for_exit` and the whole Win32 path had no direct tests at all. For code whose job is to terminate processes, that is the wrong place to be thin — a mistake there either reports a live run as stopped or kills a process that was never ours. New `test_pid_termination.py` covers both platform implementations and every Windows branch through a mocked kernel32, following the pattern already used for the liveness probe. Notably it pins: - terminate and wait use the *same* handle, which is the property that makes the confirmation immune to PID recycling; - OpenProcess requests PROCESS_TERMINATE|SYNCHRONIZE, since without SYNCHRONIZE the wait fails and confirmation silently degrades; - an unexpected WaitForSingleObject result is UNKNOWN, not 'survived'; - a negative timeout clamps to 0 rather than wrapping into INFINITE and hanging conductor stop; - the handle is closed on every failure path; - _TERMINATION_EXIT_CODE != STILL_ACTIVE, or a later probe would read a process we just killed as running forever. The POSIX cases run everywhere rather than skipping on Windows: signal.SIGKILL is injected when absent, so the branch logic is verified on both platforms instead of only on Linux CI. That also let me verify them locally, which the skip had prevented. Also filled the app.py gaps: the three --json branches (nothing running, unknown port, ambiguous target), _request_graceful_kill in full, _signal_process on both platforms including the OSError it must swallow, and the non-dict /api/info guard. pid.py 72% -> 95%. Every line of the patch in app.py is now covered; what remains uncovered there is pre-existing code the patch does not touch.
The coverage tests passed on Windows and failed on Linux CI, which is the exact failure mode I described in microsoft#385 -- and I walked straight into it. Three causes: - `ctypes.FormatError` does not exist on Linux, so the OpenProcess-denied and TerminateProcess-failure branches raised AttributeError from inside the logging call. Patched with `create=True`, matching what test_pid.py already does for `get_last_error`. - `signal.CTRL_BREAK_EVENT` does not exist on Linux, and the assertion referencing it sat *outside* the `with` block, so the injected value was already gone by the time it ran. - Three tests exercised `_signal_process` without pinning `sys.platform`, so they took the CTRL_BREAK branch on Windows and the SIGTERM branch on Linux. Whichever machine you were on decided what was covered. All now pin the platform explicitly, so both branches are verified on both platforms rather than each OS silently testing half of it. Verified by simulating a Linux runner locally -- deleting `FormatError` and `CTRL_BREAK_EVENT` before collection, and adding `SIGKILL`. Guessing was what produced the first red run. Under that simulation the suite matches the Windows result exactly: 48 failed / 685 passed on both, identical to the pre-existing baseline.
Jason Robert (jrob5756)
left a comment
There was a problem hiding this comment.
Reviewed the ladder, the identity gate, the pid primitives, and the tests. The approach is right, and the Windows termination path that keeps one handle open for both the terminate and the wait is the correct way to do it. Most of my notes are small. One I think should be fixed first.
The blocker is the MISMATCHED gate. With --force, a PID that was positively identified as belonging to a different process still gets signalled and force-terminated. The command prints "Refusing to act on it" and then "Stopped" on the next line, returns outcome stopped, removes the PID file, and exits 0. I reproduced it locally. That is the same shape as the bug this PR fixes, so it seems worth closing before this lands.
Two things I could not attach to a diff line:
write_pid_file (pid.py:167) uses write_text, which truncates and then streams, while read_pid_files (pid.py:189) unlinks anything it cannot parse and says nothing. A stop that reads a PID file during that write window gets a JSONDecodeError and silently deregisters a live run, which is the orphaning failure arriving through the read side. Writing to a temp file and calling replace(), plus a warning before the unlink, would close it.
--force still cannot clear an entry whose liveness probe keeps failing, which was the #166 escape hatch we discussed on the issue. I checked: stop --port N --force leaves the file and exits 2. A stuck entry therefore makes bare stop permanently ambiguous and stop --all permanently exit 2, so a CI teardown never recovers. Letting --force unlink on an unconfirmed outcome, with a loud warning, would restore it.
The writeup on the PR is genuinely useful, particularly calling out your own broken test rewrite and the four old tests that encoded the bug. Thanks again!
| # positive mismatch: an unconfirmable identity is not evidence of anything, | ||
| # and refusing to signal there would be a regression for PID files written | ||
| # by older versions, where a signal is all the previous code ever sent. | ||
| if identity is Identity.MISMATCHED and not force: |
There was a problem hiding this comment.
The Identity docstring says MISMATCHED "must block every PID-directed action, not just the forceful one", but and not force lets it through. With --force against a mismatched dashboard I get:
Warning: the dashboard on port 8080 is PID 9999, but the PID file records 4242. Refusing to act on it.
Stopped workflow 'w' (PID 4242, port 8080)
SIGTERM went to 4242, the outcome came back stopped, the PID file was removed, and the exit code was 0. So we signal a process we just identified as someone else's, tell the user we refused, then tell them it stopped. The real run keeps going, and automation reading that exit code concludes it is safe to start another.
The --force help says "even when the run's identity cannot be confirmed", which reads as UNCONFIRMED only. I would keep that reading and let a positive mismatch refuse unconditionally.
| ifidentityisIdentity.MISMATCHEDandnotforce: | |
| ifidentityisIdentity.MISMATCHED: |
There is also no test for MISMATCHED with force=True. test_force_overrides_the_identity_gate patches UNCONFIRMED, and test_positive_mismatch_signals_nothing_at_all only covers the default.
| "Force-terminate even when the run's identity cannot be confirmed. " | ||
| "Dangerous: the recorded PID may have been recycled onto another process." |
There was a problem hiding this comment.
Whichever way the MISMATCHED question lands, this wording should match it. "cannot be confirmed" describes UNCONFIRMED, and right now --force also overrides a positive mismatch.
Assuming the gate above is fixed:
| "Force-terminate even when the run's identity cannot be confirmed. " | |
| "Dangerous: the recorded PID may have been recycled onto another process." | |
| "Force-terminate even when the run's identity cannot be confirmed. " | |
| "Dangerous: the recorded PID may have been recycled onto another process. " | |
| "Does not override a confirmed identity mismatch." |
| except Exception as exc: # noqa: BLE001 - any failure means "cannot confirm" | ||
| logger.debug("Identity probe on port %s failed: %s", port, exc) | ||
| return Identity.UNCONFIRMED |
There was a problem hiding this comment.
Nothing under src/conductor/ configures logging, so logger.debug never reaches a user. Connection refused is the expected case and belongs at debug, but a 5xx from the dashboard, a proxy or TLS failure, and a read timeout all vanish here too, and those are things someone can act on. Before this PR the user at least got an OSError traceback to paste into a bug report.
| exceptExceptionasexc: # noqa: BLE001 - any failure means "cannot confirm" | |
| logger.debug("Identity probe on port %s failed: %s", port, exc) | |
| returnIdentity.UNCONFIRMED | |
| except (httpx.ConnectError, httpx.ConnectTimeout): | |
| logger.debug("Identity probe on port %s: dashboard unreachable", port) | |
| returnIdentity.UNCONFIRMED | |
| exceptExceptionasexc: # noqa: BLE001 - any failure means "cannot confirm" | |
| logger.warning( | |
| "Identity probe on port %s failed (%s: %s); the graceful rung will be skipped.", | |
| port, | |
| type(exc).__name__, | |
| exc, | |
| ) | |
| returnIdentity.UNCONFIRMED |
_request_graceful_kill and _signal_process have the same problem.
| # Older dashboard: fall back to run_id when both sides have one. | ||
| expected = str(entry.get("run_id") or "") | ||
| actual = str(info.get("run_id") or "") | ||
| if not expected or not actual: | ||
| return Identity.UNCONFIRMED | ||
| return Identity.CONFIRMED if actual == expected else Identity.MISMATCHED |
There was a problem hiding this comment.
I checked this against main: _finalize_background_launch called write_pid_file(proc.pid, web_port, pid_workflow_ref) with no run_id, and there was no other production caller. Every PID file written by a released version therefore has run_id == "".
So a dashboard old enough not to report pid always comes with a PID file that has no run id, and this returns UNCONFIRMED at line 1327. The CONFIRMED and MISMATCHED returns below can only fire from a hand-edited file or a third party answering on the port.
Harmless, but the PR description offers this as the thing that keeps an older dashboard identifiable, and it cannot do that. Either drop the fallback and return UNCONFIRMED, or reword the rationale so the next reader does not rely on it.
| results = [_stop_process(entry, console, force=force) for entry in targets] | ||
| for entry, result in zip(targets, results, strict=True): | ||
| if result["outcome"] in ("stopped", "already-exited"): |
There was a problem hiding this comment.
Liveness and Identity are enums compared with is. outcome is a bare string compared with in (...), and the same success tuple is repeated at 1251, 1252, and 1256. It is also the public --json contract, so a typo at any one site silently misfiles a survivor as stopped.
StrEnum serialises through json.dumps as the plain string and compares equal to str, so an Outcome enum with an is_success property costs nothing in JSON output or in the existing tests, and puts the success rule in one place. Cheap while the file is already open.
| # an unrelated process. | ||
| if not force: | ||
| identity = _confirm_identity(entry, con) | ||
| if not (identity is Identity.CONFIRMED or force): |
There was a problem hiding this comment.
Minor: not (A or B) is not A and not B, and reading it the second way saves unpacking a De Morgan in the one gate that guards an irreversible action.
| ifnot (identityisIdentity.CONFIRMEDorforce): | |
| ifidentityisnotIdentity.CONFIRMEDandnotforce: |
| This is the last rung of the stop ladder and the only one that cannot be | ||
| ignored by the target. Callers **must** confirm process identity before | ||
| invoking it — a PID read from a stale file may since have been recycled | ||
| onto an unrelated process (see issue #344). |
There was a problem hiding this comment.
This function performs no identity check, so the bold "must" is a contract for the caller rather than anything enforced here. Worth stating that plainly so nobody reads it as a guarantee the function provides.
| Thisisthelastrungofthestopladderandtheonlyonethatcannotbe | |
| ignoredbythetarget. Callers**must**confirmprocessidentitybefore | |
| invokingit — aPIDreadfromastalefilemaysincehavebeenrecycled | |
| ontoanunrelatedprocess (seeissue#344). | |
| Thisisthelastrungofthestopladderandtheonlyonethatcannotbe | |
| ignoredbythetarget. Itperformsnoidentitycheckitself; thecalleris | |
| responsibleforconfirmingthePIDstillbelongstotheintendedrun (see | |
| ``conductor.cli.app._confirm_identity``), becauseaPIDreadfromastale | |
| filemaysincehavebeenrecycledontoanunrelatedprocess (issue#344). |
| .. deprecated:: | ||
| Matching on port alone is racy: between the caller's snapshot and this | ||
| call, the original run can exit and a *new* run can bind the same port | ||
| and write its own PID file — which this would then delete, orphaning a | ||
| live workflow (issue #344). ``conductor stop`` uses | ||
| :func:`remove_pid_file_at` instead. This remains for compatibility with | ||
| any external caller. | ||
There was a problem hiding this comment.
No production caller is left. stop uses remove_pid_file_at, and run.py uses remove_pid_file_for_current_process. The only references are the three tests in test_pid.py.
"compatibility with any external caller" is carrying a lot of weight for a module inside conductor.cli, when the wheel ships one console script and conductor/__init__.py exports only __version__. I would either delete it along with TestRemovePidFile, or name a removal version so it does not sit here indefinitely.
Addresses the review on microsoft#383. The blocker. `--force` skipped the MISMATCHED gate, so a PID the dashboard had positively identified as a *different* process was still signalled and force-terminated. `_confirm_identity` printed "Refusing to act on it", the next line said "Stopped", the outcome was `stopped` and the exit code 0. That is the bug this PR exists to fix, reachable through its own escape hatch, and the PR's own Identity docstring already said MISMATCHED "must block every PID-directed action, not just the forceful one" -- the code just did not do it. The rule is now the one the docstring states: `--force` overrides *uncertainty*, never positive evidence that a PID belongs to someone else. UNCONFIRMED still escalates under --force, so the microsoft#166 escape hatch and older PID files without a run_id are unaffected. Rung 3 also re-confirms identity under --force now; it was skipped precisely where the consequences are worst, since seconds of waiting elapse before the irreversible rung and the target's PID can be recycled in that window. A mismatch now reports outcome `mismatched` rather than `unconfirmed`. "Unconfirmed" asserts we could not tell; a mismatch is the opposite -- we know. Both consumers of `outcome` test against the success allow-list ("stopped", "already-exited"), so the new value is contract-safe and still counts as a failure. Also from the review: - `write_pid_file` published with `write_text`, which truncates and then streams. Every reader treats unparseable JSON as a dead run and unlinks it, so a `stop` landing inside that window silently deregistered a live workflow -- the orphaning failure arriving through the read side. Writes now go to a temp file and land with `os.replace`. - `read_pid_files` deleted unreadable PID files silently. It logs first now; deleting someone's background-run record on incomplete information should never be invisible. - `--force` can now clear an entry whose liveness probe keeps failing (microsoft#166). Previously that entry was permanent: bare `stop` stayed ambiguous and `stop --all` exited 2 for good, so a CI teardown never recovered. Narrow by construction -- it does not fire for `refused`, `mismatched`, or `survived`, the last because that process is demonstrably alive and removing its file would orphan it. The invocation still exits 2, because clearing a record is not the same as confirming a stop. - CHANGELOG entry added; the PR had none. Tests are the point rather than an afterthought, since nothing previously asserted any of this -- `test_force_overrides_the_identity_gate` used UNCONFIRMED, so the mismatch case was a gap, not a disagreement. Each new test was checked against the unfixed code: the two --force/MISMATCHED tests and the two pid.py tests all fail without their fix and pass with it. The temp-file test is a guard on the new write path rather than a regression test, and passes either way -- noted so it is not mistaken for coverage it does not provide. 583 passed in tests/test_cli. ruff check unchanged at 4 pre-existing findings (UP042 on three str+Enum classes, including one this PR did not touch); my files are ruff-format clean and `ty` clean; `ty check src` unchanged at 65.
Frank Li (franklixuefei)
commented
Aug 10, 2026
Thanks — the MISMATCHED gate was a real bug and you reproduced it exactly. All four items are addressed in The blocker: |
| Test | Without fix |
|---|---|
test_force_does_not_override_a_positive_mismatch | ❌ fails |
test_force_reconfirms_identity_before_the_irreversible_rung | ❌ fails |
test_write_is_atomic_so_a_reader_never_sees_a_partial_file | ❌ fails |
test_unreadable_pid_file_logs_before_unlinking | ❌ fails |
test_pid_file_without_a_pid_logs_before_unlinking | ❌ fails |
test_force_can_clear_an_entry_whose_liveness_cannot_be_probed | ❌ fails |
test_temp_file_is_not_left_behind | ✅ passes either way |
That last row is deliberate: it guards the new write path against leaving strays, it is not a regression test, and I would rather say so than let it look like coverage it is not.
Two existing assertions changed from "unconfirmed" to "mismatched", each with a comment explaining why it is intentional.
Gates
583 passed in tests/test_cli. ruff check unchanged at 4 pre-existing findings (UP042 on three str+Enum classes, one of which this PR never touched); my files are ruff format clean and ty clean — including fixing the os.replace annotation, since you flagged that exact class of ty error on #381. ty check src unchanged at 65.
CHANGELOG entry added under Unreleased → Fixed.
Jason Robert (jrob5756)
left a comment
There was a problem hiding this comment.
Re-ran everything against 9642c2a instead of taking the summary on faith. The blocker is fixed, and two of the things you did go past what I asked for.
Verified
Reran my original reproduction. Neither the signal nor the terminate fires now, the outcome comes back mismatched, and the console finally reads straight: a refusal, then another refusal, with no "Stopped" sitting three lines under a warning that says we refused.
Re-confirming identity at rung 3 under --force is better than what I suggested. I only asked you to stop --force overriding a mismatch. You spotted that the re-check was being skipped in the exact window it exists for.
The escape hatch is narrow in the right way. I ran the matrix:
| case | exit | PID file |
|---|---|---|
unprobeable + --force | 2 | cleared |
survived + --force | 2 | kept |
mismatched + --force | 2 | kept |
unprobeable, no --force | 2 | kept |
survived staying put is the one I most wanted to check, since clearing it would have been the orphaning bug again under another name.
The atomic write lands through os.replace, the .tmp suffix keeps it out of the *.pid glob, and a failed write cleans up after itself. read_pid_files logs before it unlinks anything.
I checked that the tests actually bite. Reverted both source files to 5c52e88 and reran: 8 fail, all pass again on the fixed tree. Your note about test_temp_file_is_not_left_behind passing either way holds up, it is in the 10 that passed.
741 passed and 3 skipped across test_cli and test_web here. ruff check and ruff format --check clean on both changed files.
On the exit code
Keep 2. Clearing a record is not the same as confirming a stop, and returning 0 there would put back the same "report success you cannot back up" habit this PR exists to remove.
The wedge is gone either way, which I checked: the first stop --all --force exits 2 and clears the entry, the next exits 0 and reports nothing running. CI recovers without the command lying about what it did.
Two doc gaps I would like before merge
docs/cli-reference.md still says "The stop command reads these PID files, sends SIGTERM to the process, and cleans up the file." That is the behaviour you just replaced. The options table lists only --port and --all, and the exit codes are not documented anywhere. The CHANGELOG entry is good, but people tend to read the reference page.
The --force help says nothing about it now being able to delete a PID file for a process that may still be alive. Your own warning text says "it is now untracked and must be stopped by hand", which seems worth a line in --help.
Carried over, not blocking
DEBUG-level logging swallowing real HTTP failures, the run_id fallback that cannot fire, the De Morgan at rung 3, the terminate_process docstring, and dead remove_pid_file. All fine as follow-ups.
One of them got slightly more relevant: outcome is still a bare string tested with in (...) at four sites, and "mismatched" makes that a four-value set now. Still not blocking.
Sort the two doc items and I will approve.
# Conflicts: # CHANGELOG.md
The reference page still described the behaviour this PR replaced: `stop` reads a PID file, sends SIGTERM, and cleans up the file. That is the single-signal, report-success-without-checking path microsoft#344 is about, so the page documented the bug rather than the fix. Rewrite `How It Works` around the confirmed ladder, add the missing `--force` and `--json` rows, document the exit codes, and give identity its own section -- the mismatched/unconfirmed distinction is what decides whether `--force` does anything, and it had no prose anywhere. Also extend the `--force` help. It described force-terminating an unconfirmed run but said nothing about the other thing `--force` now does: clearing the PID file of a run whose liveness cannot be probed, leaving a possibly-live process untracked. The console warning already says that; the help should too.
Frank Li (franklixuefei)
commented
Aug 11, 2026
Both doc items are done in
|
Resolves the conflict with microsoft#406, which landed after this branch's last merge and inverted the console default to ``markup=False``: styling now goes through ``styled("<template>", value)`` and ``Text.from_markup``, which parse the conductor-authored template but insert values verbatim. The stop ladder was written before that, so its output used f-strings with inline markup. Taking this branch's side of the conflict unchanged would have regressed microsoft#406 in exactly the code microsoft#344 added, and ``test_no_markup_string_literal_reaches_a_sink`` fails on it. Every ``con.print``/``console.print`` in the stop path is therefore converted: templates that interpolate a value use ``styled()``, static ones use ``Text.from_markup``. This is load-bearing rather than cosmetic. A workflow name reaches these messages via ``Path(entry["workflow"]).stem``, so a name like ``deploy[prod]`` is read as a style tag and dropped by a parsing console, or printed with raw ``[green]`` scaffolding by a non-parsing one. After microsoft#344 the whole job of these messages is naming which run survived so it can be stopped by hand, so a mangled name defeats the feature. Other resolutions: - ``bg_runner.py``: took main's version wholesale. This branch threaded ``run_id`` into ``write_pid_file``; main does the same and adds ``stderr_log``/``stdout_log``, so main's supersedes it. - ``pid.py``: auto-merged, keeping main's new log-path fields alongside this branch's atomic write and announced deletes. - ``test_stop_ladder.py``: ``write_pid_file`` no longer takes ``log_file``; the round-trip test now asserts the fields main replaced it with. - ``docs/cli-reference.md``: kept this branch's rewritten section and folded in main's sentence about the PID file recording ``run_id`` and the capture-log paths. Main's "sends SIGTERM ... and cleans up the file" sentence is dropped, since that is the behaviour microsoft#344 replaced. - ``CHANGELOG.md``: kept both entries. Adds ``TestWorkflowNamesArePrintedLiterally``, which drives a bracketed workflow name through the already-exited, survived and unconfirmed rungs plus the mismatch refusal, asserting the name survives and no markup leaks. It fails on the unconverted resolution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert (jrob5756)
commented
Aug 11, 2026
Pushed The complication was #406, which landed after your last merge and inverted the console default to It is not cosmetic. A workflow name reaches those messages through Worth saying plainly: I first assumed a naive resolution would slip through CI, and that was wrong. Your Other resolutions:
I also added Please give the merge commit a look, particularly the |
Jason Robert (jrob5756)
left a comment
There was a problem hiding this comment.
LGTM. Approved!
Uh oh!
There was an error while loading. Please reload this page.
Reconciles PR #414's self-exclusion feature (--allow-self, issue #399) with PR #383's escalating stop ladder (--force/--json, issue #344), both of which modified conductor stop's targeting logic on overlapping lines. - src/conductor/cli/app.py: stop() now partitions running PID-file entries into self/others (via self_run.partition_own_run) before applying the ladder-based _stop_process/--force/--json machinery, so self-exclusion and confirmed termination compose instead of one clobbering the other. - tests/test_cli/test_stop.py: rewrote TestStopSelfExclusion to drive the real ladder via _stops_cleanly() plus a wrapping spy on _stop_process (rather than asserting on direct os.kill calls, which no longer reflects how stop() signals a process), and updated the ambiguous-listing self-exclusion test to expect exit code 1, matching #383's fix for that case. - tests/test_cli/test_stop_ladder.py: added the same no_self_run autouse fixture used in test_stop.py -- these pre-existing ladder tests use small arbitrary PIDs (1, 2, 4242) that can collide with real process ancestry in shallow-PID-namespace environments, which self-exclusion would otherwise misclassify as the caller's own run. - CHANGELOG.md, docs/cli-reference.md: concatenated both PRs' entries; merged the duplicated 'Exit Codes' section in the stop docs into one table covering both self-exclusion and ladder outcomes. Verified: ruff check, ruff format --check, ty check src, and the full test suite (targeted stop/self_run/pid/markup suites plus pytest -m "not install_scripts") all pass. The one failure seen (test_event_log.py::test_filenames_unique_for_simultaneous_starts) is a pre-existing, environment-specific flake reproduced identically on a clean origin/main checkout, unrelated to this merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Closes#344.
The bug
conductor stopsent one best-effort signal, printedStopped, and deleted the PID file unconditionally. On Windows that signal reliably fails — as Jason Robert (@jrob5756) diagnosed on the issue,GenerateConsoleCtrlEventneeds a shared console, which a separatestopinvocation does not have. So the usual outcome was: workflow still running, user told it stopped, PID file gone. The run became an untracked orphan still burning tokens, and becausestophad deregistered it there was no supported way to find it again. In my case relaunching produced two concurrent runs racing on the same worktree.That correction also killed my original theory (I had guessed a missing
CREATE_NEW_PROCESS_GROUP; the bg child already gets it atbg_runner.py:48,79,151). Following it up surfaced something worse and cross-platform: the bg child runs--no-interactive, so no SIGTERM handler is installed anywhere.stopis therefore a hard kill with no checkpoint on every platform, not just Windows.The change
A ladder, with a bounded wait after each rung, and no rung trusted to have worked.
POST /api/kill— the only rung that lets the run write a resume checkpoint, so it is always tried first.The central rule: a PID file is removed only once its process is confirmed dead. A survivor keeps its registration and the command exits 2, so a run can never again be silently deregistered while alive.
Two details that matter more than they look:
/api/killis an acknowledgement, not a death certificate. It sets an asyncio event and returns;run.py:1441'sasyncio.gatherdrain is unbounded, so a provider that swallows cancellation leaves the child alive indefinitely after the endpoint already returned success. Every rung is followed by a real liveness check.TerminateProcesson a recycled PID kills a bystander. PID files outlive crashes and reboots, and Windows recycles PIDs aggressively.Identity is by PID, not run id
My first attempt compared
run_id, which already existed in the PID-file schema and in/api/infobut was never populated by the launcher. Wiring it up was not enough — it cannot work forresume --web-bg, and a post-implementation review caught it before this PR went up.The launcher generates a fresh run id for the PID file, but a resumed child reuses the checkpoint's run id:
event_log.py:104-116returns early onexisting_run_idbefore ever consultingCONDUCTOR_RUN_ID. So/api/inforeported an id that could never match, and every resumed run looked like somebody else's process — graceful rung skipped, a misleading "a different run took over this port" warning, and on Windows the forceful rung refused. POSIX masked it entirely, because SIGTERM terminates the child before the gate is ever reached./api/infonow reportsos.getpid()unconditionally. The dashboard runs in the same process as the workflow, so this is direct proof of ownership, and it fixes three things at once:run --web-bgresume --web-bg{}untilworkflow_started)Run-id comparison is kept as a fallback so a dashboard predating this change is still identifiable.
Identity is a tristate, and that distinction is load-bearing
SIGTERM, perfectly capable of killing the bystander the gate exists to protect.Identity is re-confirmed immediately before the irreversible rung, since up to ten seconds of waiting elapse after the first check and the PID could be recycled in that window. This narrows the race rather than closing it; the fully general fix is to retain a process handle across the whole ladder (Windows) or a pidfd (Linux), which I have deliberately kept out of this PR — happy to follow up.
Everything else
Livenessinternally, so "the probe failed" can no longer be mistaken for "confirmed running". The bool wrappers are retained, so the conductor stop crashes on Windows: OSError [WinError 11] from os.kill(pid, 0) in _is_process_alive #166 listing behaviour is byte-for-byte unchanged — importantly, UNKNOWN still keeps the PID file there, whichtest_unexpected_oserror_does_not_propagatepins.terminate_processreturning UNKNOWN reportsunconfirmed, notsurvived. A failed probe is not evidence the process lived, and JSON consumers would readsurvivedas "definitely still alive".remove_pid_file(port). Port matching could unlink a newer run's registration if that run bound the same port mid-stop— and making the unlink conditional on the old run's death makes that race more likely to fire, not less.PROCESS_TERMINATE | SYNCHRONIZEand both terminates and waits on the same handle. An open handle pins the PID, so the confirmation cannot be fooled by recycling mid-call._TERMINATION_EXIT_CODE = 1 != STILL_ACTIVE, so a later probe reads DEAD.--json, emittingpid/port/run_id/outcome/rungper run — the issue's own evidence is that automation trustedstop's prose and double-launched. An exit code says something failed; it does not say which run survived. Prose goes toconsole(stderr) and JSON tooutput_console(stdout), so they cannot corrupt each other.--forcefor the unidentifiable case.Breaking changes
Exit codes. Previously every path except an unknown
--portreturned 0.--portmatched nothing, or the target was ambiguous (multiple runs, no--port/--all)CI teardowns running
conductor stop --allunderset -ewill now fail when a run genuinely survives — which is the point, but it is a behaviour change. "Nothing running" and "already exited" are deliberately 0. The ambiguous case moved from 0 to 1 because it stops nothing, and reporting success for that is exactly the class of bug this PR is about.Pre-existing PID files (written before this change, no run id) are identified by PID as soon as their dashboard is reachable, so in practice they behave normally. If the dashboard is unreachable and the process survives both non-forceful rungs,
stopnow refuses to force-terminate and says so, instead of silently deregistering a live run.Tests
tests/test_cli/test_stop_ladder.pycovers the ladder, the identity tristate, and the removal rules;tests/test_web/test_server.pycovers/api/inforeportingpidbefore any workflow event.I verified the guards actually bite by simulating three separate reverts — each fails, and the fixed tree passes:
Four pre-existing tests in
test_stop.pyasserted the old contract — thatstopreports success merely becauseos.killdid not raise — and were rewritten to assert the new one. I want to flag that explicitly rather than bury it: those tests encoded the bug.One of my own rewrites was itself broken, and I would rather say so than have it found in review:
test_pid_file_is_removed_when_process_confirmed_gonepatchedconductor.cli.app.os.kill, which mutates the sharedosmodule object — on POSIX that also patchespid.py's liveness probe, turning DEAD into UNKNOWN. It passed on Windows and would have failed on the Ubuntu CI runners. It now patchesprocess_livenessdirectly. The--allsurvivor test had a similar latent flake, keying outcomes off call order when target order comes fromPath.glob, whose ordering is filesystem-dependent; it is now keyed off the pid.Full
tests/test_cli+tests/test_webrun is an exact failure-set match against cleanmain— zero new failures. (My sandbox has no network access to PyPI, sopytest-asynciois unavailable and ~48 async tests error identically before and after; I diffed the failure sets rather than trusting the count.)ruff checkandruff format --checkare clean.Not in this PR
stopcan checkpoint on POSIX. Notelistener.py:244-250chains to_previous_sigterm, which is normallysignal.SIG_DFL— not callable — so the existing handler silently swallows SIGTERM, contradicting its own docstring. That wants its own fix.--all. Each target is bounded (~22s worst case) but N targets are sequential with no overall cap.PYTHONUTF8=1in_build_bg_env— deliberately excluded.pid.pyreads and writing PID files with no explicit encoding, so making the child UTF-8 whilestopstays on the locale codec would split-brain them, andUnicodeDecodeErroris aValueErrorsibling that the existingexcept (json.JSONDecodeError, OSError)would not catch.Related but independent, from the same run: #342 / #381 (
UnicodeEncodeErrorin JSON output) and #382 (MarkupErrorfrom unescaped agent text). Three separate boundaries, three separate fixes.