forkserver fails silently under load causing Popen.poll() to return 255

[zmedico]

Bug report

Bug description:

This forkserver load test program fails reliably when run with 8 jobs on my 8 core laptop:

#!/usr/bin/env python3.14

import asyncio
import multiprocessing
import os
import sys


def target(count):
 print("success", count)


async def main(argv):

 max_jobs = int(argv[1])

 jobs = {}
 count = 0
 while True:
 count += 1

 if len(jobs) >= max_jobs:
 done, pending = await asyncio.wait(
 list(jobs), return_when=asyncio.FIRST_COMPLETED
 )
 for waiter in done:
 proc = jobs.pop(waiter)
 if proc.exitcode != os.EX_OK:
 print("failure with exitcode", proc.exitcode)
 if jobs:
 await asyncio.wait(
 list(jobs), return_when=asyncio.ALL_COMPLETED
 )
 return

 proc = multiprocessing.Process(target=target, args=(count,))
 proc.start()
 waiter = asyncio.ensure_future(asyncio.to_thread(proc.join))
 jobs[waiter] = proc


if __name__ == "__main__":
 asyncio.run(main(sys.argv))

Result:

 $ ./forkserver_load_test.py 8
success 1
success 2
success 4
success 5
success 6
success 3
success 7
success 8
failure with exitcode 255
success 9
success 10
success 11
success 12

I patched forkserver.py for debugging and found that the test case does not cause SystemExit to be raised here until the main function of the test case returns:

https://github.com/python/cpython/blob/v3.14.0/Lib/multiprocessing/forkserver.py#L272

I patched popen_forkserver.py for debugging and found that read_signed raises EOFError here:

https://github.com/python/cpython/blob/v3.14.0/Lib/multiprocessing/forkserver.py#L394

Meanwhile, it seems that the corresponding write_signed call was successful on the forkserver side.

I did some more debugging and found that the underlying issue is a thread safety issue for the self.returncode setting in Popen.poll(), and the load test runs better with this patch:

--- a/Lib/multiprocessing/popen_forkserver.py
+++ b/Lib/multiprocessing/popen_forkserver.py 
@@ -1,5 +1,6 @@
 import io
 import os
+import threading
 
 from .context import reduction, set_spawning_popen
 if not reduction.HAVE_SEND_HANDLE:
@@ -32,6 +33,7 @@
 
 def __init__(self, process_obj):
 self._fds = []
+ self._poll_lock = threading.Lock()
 super().__init__(process_obj)
 
 def duplicate_for_child(self, fd):
@@ -65,7 +67,9 @@
 if not wait([self.sentinel], timeout):
 return None
 try:
- self.returncode = forkserver.read_signed(self.sentinel)
+ with self._poll_lock:
+ if self.returncode is None:
+ self.returncode = forkserver.read_signed(self.sentinel)
 except (OSError, EOFError):
 # This should not happen usually, but perhaps the forkserver
 # process itself got killed

I created the test case because the same issue was triggered by a program called egencache here:

https://bugs.gentoo.org/965132

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[picnixz]

cc @gpshead

[zmedico]

I found that the root cause was thread safety for self.returncode = forkserver.read_signed(self.sentinel) setting in Popen.poll(), and edited my first comment to include a patch that makes the load test run better.

[duaneg]

See also #130895, which I think has the same root cause, at least with regards forkserver.