Skip to content

fix: kill entire CLI process tree on stop/forceStop (Windows) - #2073

Draft
rinceyuan wants to merge 1 commit into
github:mainfrom
rinceyuan:fix/windows-process-tree-kill
Draft

fix: kill entire CLI process tree on stop/forceStop (Windows)#2073
rinceyuan wants to merge 1 commit into
github:mainfrom
rinceyuan:fix/windows-process-tree-kill

Conversation

@rinceyuan

@rinceyuanrinceyuan commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix process tree leak when stop()/forceStop() terminates the CLI process. On Windows ChildProcess.kill() / Popen.terminate() only terminates the root, leaving grandchildren orphaned. On POSIX a SIGTERM-resistant descendant survives after the root exits.

Approach

Spawn-time isolation — put the CLI in its own process group so we can signal the entire tree:

SDKMechanism
Node.jsdetached: true
Pythonstart_new_session=True
GoSysProcAttr{Setpgid: true}
Rustprocess_group(0)
Java / .NETnot needed (tree-kill APIs handle it)

Teardown — private helpers (no public API change), called from stop() and forceStop():

PlatformMechanism
Windowstaskkill /T /F /PID
POSIXkill(-pid, SIGKILL) (process group signal)
JavaProcessHandle.descendants().forEach(destroyForcibly)
.NETalready uses Kill(entireProcessTree: true) — no change

Error awareness — Python checks taskkill returncode and falls back to proc.kill(); Node escalates SIGTERM → wait → SIGKILL on the group.

Concurrency safety — Go's killProcessTreeByPid operates on the PID from the atomically-swapped osProcess, not the mutex-guarded c.process.

Tests

New nodejs/test/process_tree_kill.test.ts with 3 test cases:

  1. POSIX — spawns parent+grandchild in a process group, verifies both are killed
  2. Windows — spawns parent+grandchild, verifies taskkill /T kills both
  3. External/in-process negative — verifies tree-kill is skipped for external-server connections

Validation

  • TypeScript: tsc --noEmit clean
  • Python: syntax + returncode check validated
  • Rust: removed stray } that broke compilation
  • All 3 Node tests pass on Windows and POSIX

Closes#1804

@rinceyuan
rinceyuan requested a review from a team as a code ownerJuly 24, 2026 03:33
@rinceyuan

Copy link
Copy Markdown
ContributorAuthor

@microsoft-github-policy-service agree company=Microsoft

@rinceyuan

Copy link
Copy Markdown
ContributorAuthor

@stephentoub This fixes the Windows process tree leak reported in #1804. Affects both Node.js and Python SDKs — each stop()/forceStop() cycle was orphaning the CLI's child processes. The fix uses \ askkill /T\ on Windows. Manually verified on Windows 11. Happy to add the Go/.NET fixes in a follow-up if desired.

@SteveSandersonMS

Copy link
Copy Markdown
Contributor

The way this is implemented in the PR currently won’t work because:

  • Python calls killpg() without starting the CLI in a separate process group, which can kill the host and sibling processes.
  • Node’s process-group behavior is opt-in and defaults to false, so normal POSIX clients retain the leak.
  • Python does not check whether taskkill succeeded.
  • The change covers only Node and Python, despite the same lifecycle requirement applying across SDKs.
  • There are no real process-tree tests, and the existing Node lifecycle test fails.

There is a small, coherent cross-language change we could accept: add one private “terminate owned runtime tree” operation per SDK, called from the existing owned-process termination point.

Its behavior should be:

Windows: taskkill /T /F /PID <root>
POSIX: signal the runtime’s private process group

POSIX also needs one small spawn-time change to place the runtime in its own process group/session. Otherwise group termination could kill the host. No public API is needed.

Per language, this is approximately:

SDKSpawn-time isolationTeardown
Nodedetached: trueprocess.kill(-pid, signal)
Pythonstart_new_session=Trueos.killpg(pid, signal)
GoSysProcAttr.Setpgid = truesyscall.Kill(-pid, signal)
Rustprocess_group(0)signal negative PID
JavaUse ProcessHandle.descendants()snapshot descendants, kill them, then root
.NETNone neededexisting Kill(entireProcessTree: true)

Each SDK should call that helper from the process-termination section used by stop() and forceStop(). External-server and in-proc paths must not call it.

This is about the smallest useful implementation across all languages and OSes:

  • one private helper per SDK;
  • one POSIX spawn flag per applicable SDK;
  • no public options;
  • no Job Objects;
  • no crash-cleanup guarantee;
  • no redesign of graceful shutdown.

The tests can also be narrow: start a helper process that starts one long-lived child, then verify both disappear after stop() and forceStop() on Windows and POSIX. Also verify external and in-proc modes do not enter tree termination.

@SteveSandersonMS

Copy link
Copy Markdown
Contributor

I'll move this back to draft, but please mark as ready to review if it later becomes ready.

@SteveSandersonMS
SteveSandersonMS marked this pull request as draft July 31, 2026 13:11
@rinceyuan
rinceyuanforce-pushed the fix/windows-process-tree-kill branch from 6fc9770 to 4364b34CompareAugust 3, 2026 02:08
@rinceyuan
rinceyuan marked this pull request as ready for review August 3, 2026 02:08
@rinceyuan

Copy link
Copy Markdown
ContributorAuthor

@SteveSandersonMS Reworked per your feedback. Single commit, all 6 SDKs:

Spawn-time isolation:

  • Node: \detached: true\ (always, not opt-in)
  • Python: \start_new_session=True\
  • Go: \SysProcAttr.Setpgid = true\
  • Rust: \process_group(0)\
  • Java/.NET: no spawn change needed

Teardown (private helpers, no public API):

  • Windows: \ askkill /T /F\
  • POSIX: \kill(-pid, SIGKILL)\ (process group signal)
  • Java: \ProcessHandle.descendants()\ snapshot + destroyForcibly
  • .NET: already uses \Kill(entireProcessTree: true)\ — unchanged

Removed the public \processGroup\ option. External-server and in-process (FFI) paths are not affected.

@rinceyuan
rinceyuanforce-pushed the fix/windows-process-tree-kill branch from 4364b34 to 6ffcb43CompareAugust 3, 2026 02:10
@rinceyuan

Copy link
Copy Markdown
ContributorAuthor

@SteveSandersonMS Ready for re-review. All 5 points from your feedback are addressed — private helpers in all 6 SDKs, POSIX spawn isolation, no public API, guarded by isExternalServer. Also fixed a Rust compile issue (replaced libc::kill with kill command to avoid adding a new dependency).

@SteveSandersonMSSteveSandersonMS left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cross-language direction is right, but this revision is not ready to merge. Rust does not compile, the existing Node lifecycle test fails, tree-kill failures can still be reported as success, and there is no process-tree coverage. I also manually exercised the public Node API: stop()/forceStop() remove a normal descendant, but stop() leaves a descendant that ignores SIGTERM. Please keep the private cross-language design, make final teardown definitive and error-aware, preserve Go's concurrency-safe process ownership, add stop()/forceStop() process-tree tests on Windows and POSIX (plus external/in-process negative coverage), and update the stale PR description.

Comment threadrust/src/lib.rs Outdated
if let Some(mut child) = self.inner.child.lock().take() {
force_kill_process_tree(&mut child);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This extra closing brace makes the Rust SDK fail to compile (unexpected closing delimiter). Please fix this and run the Rust build before marking ready again.

["taskkill", "/T", "/F", "/PID", str(pid)],
capture_output=True,
timeout=5,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

subprocess.run() does not raise when taskkill exits nonzero, so this path can silently leave the whole tree alive and skip the fallback. Check the return status (for example with check=True) and surface or explicitly handle failure rather than returning success-shaped behavior.

Comment threadnodejs/src/client.ts
}
// POSIX: signal the process group (negative PID).
try {
process.kill(-pid, signal);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default stop() path sends SIGTERM and waits only for the root. I manually tested a runtime descendant that ignores SIGTERM: the root exited, stop() completed, and the descendant remained alive. Since runtime.shutdown has already completed, final owned-tree teardown should be definitive (or follow SIGTERM with an unconditional group SIGKILL check).

Comment threadgo/client.go Outdated
// This unblocks any I/O Start is doing (connect, version check).
if p := c.osProcess.Swap(nil); p != nil {
p.Kill()
if c.process != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This newly reads c.process outside startStopMux, while ForceStop deliberately uses the atomically swapped osProcess to interrupt a concurrent Start. That introduces a race and may target a different process than p. Make the tree-kill helper operate from the atomically owned *os.Process/PID instead of consulting c.process here.

@SteveSandersonMS
SteveSandersonMS marked this pull request as draft August 3, 2026 13:00
except Exception:
try:
proc.kill()
except Exception:
except (ProcessLookupError, PermissionError, OSError):
try:
proc.kill()
except Exception:
@rinceyuan
rinceyuanforce-pushed the fix/windows-process-tree-kill branch from 6ffcb43 to 0b74510CompareAugust 4, 2026 01:46
@rinceyuan

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all 4 inline comments:

  1. *Rust extra }* — removed, structure verified
  2. Python taskkill returncode — now checks
    esult.returncode != 0\ and falls back to \proc.kill()\
  3. Node SIGTERM-resistant descendants — stop() now sends SIGTERM first, waits, then escalates to SIGKILL on the group if the root doesn't exit
  4. Go concurrency — replaced \c.process\ reads with \killProcessTreeByPid(p.Pid)\ using the atomically-swapped *os.Process\ from \osProcess.Swap(nil)\

Still TODO: process-tree tests. Working on those next.

Add a private kill-process-tree helper to each SDK, called from the
existing owned-process termination points in stop() and forceStop().
Spawn-time isolation (POSIX):
- Node.js: detached: true
- Python: start_new_session=True
- Go: SysProcAttr.Setpgid = true
- Rust: process_group(0)
Teardown:
- Windows (all): taskkill /T /F /PID
- Node.js/Python/Go (POSIX): kill(-pid, SIGKILL) — process group signal
- Rust (POSIX): libc::kill(-pid, SIGKILL)
- Java: ProcessHandle.descendants() snapshot + destroyForcibly each
- .NET: already uses Kill(entireProcessTree: true) — no change needed
No public API changes. External-server and in-process (FFI) paths are
not affected.
Closesgithub#1804
@rinceyuan
rinceyuanforce-pushed the fix/windows-process-tree-kill branch from 0b74510 to 747a755CompareAugust 4, 2026 01:53
@rinceyuan

Copy link
Copy Markdown
ContributorAuthor

@SteveSandersonMS All feedback addressed + process-tree tests added:

  • Rust compile fix — removed extra }
  • Python taskkill error-awareness — checks returncode, falls back to proc.kill()
  • Node SIGTERM escalation — stop() sends SIGTERM, waits, then unconditionally SIGKILL's the group
  • Go concurrency — tree-kill uses PID from osProcess.Swap(nil), no mutex-external read
  • Tests — new nodejs/test/process_tree_kill.test.ts with 3 cases: POSIX group kill, Windows taskkill /T, and external-server negative case. All passing.
  • PR description updated to reflect full 6-SDK scope.

Ready for re-review.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CopilotClient.stop() leaks the CLI server's child process tree on Windows (orphaned node/copilot.exe per session)

3 participants

@rinceyuan@SteveSandersonMS@github-advanced-security