install-script E2E tests SIGTERM-kill all live conductor processes (running the suite silently kills any active --web-bg workflow)

[jrob5756]

Summary

Running the test suite with pytest -m "not performance" (or plain pytest) executes the install-script E2E tests, and those tests run install.sh --auto-stop, whose find_running_conductor() scans the real host process table and kills (SIGTERM) every live conductor process on the machine — including any unrelated conductor run / conductor run --web-bg workflow the developer (or an orchestrating workflow) has running.

The result is a silent hard-kill of live workflows: no traceback, no faulthandler dump, the PID file is left behind, and no workflow_failed event is emitted (unhandled SIGTERM).

This was discovered because an sdd-implement --web-bg workflow kept dying: its coder agent runs the full suite as a verification step (Run the full test suite to confirm no regressions), which killed the very workflow driving it.

Reproduction (no LLM / provider needed)

1. Start any long-lived conductor process, e.g. a wait-only workflow in the background:

   # wait-repro.yaml: a single `type: wait` step holding for 240s
   conductor run wait-repro.yaml --input hold_seconds=240 --web-bg

2. In parallel, run the suite the way the docs/agents do:

   uv run pytest -q -m "not performance"

3. The background workflow is SIGTERM-killed partway through (during tests/test_integration/test_install_scripts.py), silently — its PID file remains in ~/.conductor/runs/, and nothing is logged.

Bisection: every other test directory is harmless; only tests/test_integration/test_install_scripts.py kills it. A plain python3 script.py sentinel survives the same suite — because find_running_conductor matches on the process name (conductor / python -m conductor), which a real conductor process matches but a plain python process does not.

Root cause

• install.sh — find_running_conductor() scans the whole machine, excluding only self + immediate parent:

  ps -axo pid=,command= | awk ... {
   if (leaf == "conductor" || leaf ~ /^conductor[._-]/) { print $1, cmd; next }
   if (leaf ~ /python/ && cmd ~ /[ \t]-m[ \t]+conductor/) { print $1, cmd; next }
  }

  and with --auto-stop it kill "$pid" each match.
• tests/test_integration/install_scripts_helpers.py — run_install_script(..., auto_stop: bool = True, ...) defaults to --auto-stop, so any install E2E test that runs it triggers the host-wide kill.
• tests/test_integration/test_install_scripts.py — the install E2E tests run under pytest -m "not performance" because that filter does not exclude install_scripts. Only make test (-m "not install_scripts") excludes them, which is why make test is safe but the coder's -m "not performance" is not.

Every symptom matches a plain kill <pid> (SIGTERM): silent, no faulthandler (which only fires on SIGSEGV/SIGABRT/…), PID file survives, no workflow_failed.

Impact

1. Running the suite via pytest/pytest -m "not performance" while any conductor workflow is live silently kills that workflow — a real footgun for developers and for agent-driven workflows that self-test.
2. Note: this is unrelated to real_api tests are not deselected by default; running the suite spawns real copilot subprocesses (can silently kill a live --web-bg workflow) #326 (real_api tests). fix: auto-skip real_api-marked tests by default #328 correctly fixed real_api, but the workflow kept dying afterward — this install-script path is the actual cause.

Proposed fix

A. Auto-skip install_scripts-marked tests by default (mirror the #328 real_api conftest hook). Then pytest, pytest -m "not performance", and agent-driven runs all exclude the host-destructive install E2E tests, while they still run in the dedicated -m install_scripts CI job (isolated runner with no live workflows). This makes "run the full functional suite" safe from inside/alongside a live workflow.

# tests/conftest.py — extend the existing opt-in-by-default pattern
def pytest_collection_modifyitems(config, items):
 marker_expr = config.getoption("-m") or ""
 for mark_name in ("real_api", "install_scripts"):
 if mark_name in marker_expr:
 continue # explicitly requested
 skip = pytest.mark.skip(reason=f"{mark_name}: opt in with -m {mark_name}")
 for item in items:
 if mark_name in item.keywords:
 item.add_marker(skip)

B. Defense-in-depth: default run_install_script(..., auto_stop=False) and have the single auto-stop test opt in explicitly; ideally scope the auto-stop test so its find_running_conductor match is limited to its own sandboxed process rather than the whole host, so the host-wide kill can never fire from the test harness.

(Optionally also reconsider whether install.sh --auto-stop's host-wide kill is too broad even outside tests.)

Environment

• Conductor v0.1.24
• Linux (WSL2), Python 3.12
• Reproduced deterministically without any LLM/provider (wait-only --web-bg workflow).

[jrob5756]

Starting to take a look at this.