On Windows, creating a multiprocessing pool leaks handles

[vstinner]

Example:

import ctypes.wintypes
import gc
import multiprocessing.pool

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

kernel32.GetCurrentProcess.argtypes = ()
kernel32.GetCurrentProcess.restype = ctypes.wintypes.HANDLE

kernel32.GetProcessHandleCount.argtypes = (ctypes.wintypes.HANDLE, ctypes.wintypes.LPDWORD)
kernel32.GetProcessHandleCount.restype = ctypes.wintypes.BOOL

def handle_count():
 # Pseudo-handle that doesn't need to be closed
 hproc = kernel32.GetCurrentProcess()

 handle_count = ctypes.wintypes.DWORD()
 if not kernel32.GetProcessHandleCount(hproc, ctypes.byref(handle_count)):
 raise ctypes.WinError(ctypes.get_last_error())

 return handle_count.value

def func():
 pool = multiprocessing.pool.Pool(1)
 pool.terminate()
 pool.join()
 pool.close()
 pool = None
 gc.collect()

def main():
 print("Initial handle count:", handle_count())
 for _ in range(4):
 func()
 print("Handle count:", handle_count())

if __name__ == "__main__":
 main()

Output with Python 3.16 on Windows:

Initial handle count: 116
Handle count: 123
Handle count: 127
Handle count: 131
Handle count: 135

I discovered the issue while working on PR gh-154140 which detects leaks of Windows handles in the Python test suite.

Linked PRs

• gh-154208: Fix Windows handle leak when a child process dies early #154365

[vstinner]

See also the old issue gh-82444 but it seems to be unrelated.

[vstinner]

Using https://learn.microsoft.com/en-us/sysinternals/downloads/handle program, I can see that each loop iterations leaks 4 NamedPipe handles. Example:

 258: File (---) \Device\NamedPipe\pyc-11440-3-ac3e556c28fc4736
 25C: File (---) \Device\NamedPipe\pyc-11440-3-ac3e556c28fc4736
 260: File (---) \Device\NamedPipe\pyc-11440-4-1fe407d4fe79b646
 264: File (---) \Device\NamedPipe\pyc-11440-4-1fe407d4fe79b646

[vstinner]

Ok, the problem is that the pool child processes are killed before they can run spawn_main().

The normal code, when the child process is not killed early:

• The main process (pool manager) duplicates handles and pass handles to child processes.
• Child processes close these handles by calling DuplicateHandle() with DUPLICATE_CLOSE_SOURCE (call DupHandle.detatch() of multiprocessing.reduction if I follow correctly).

This issue is similar to issue gh-87134: "[Windows] multiprocessing handle leak when child process is killed during startup/unpickling".

[vstinner]

See also similar issue #114720.