gh-128041: Try to fix transiency of test_force_shutdown_workers by csm10495 · Pull Request #130812 · python/cpython

csm10495 · 2025-03-03T22:11:29Z

On some platforms we seem to get 255 instead of -SIGNAL, so instead use mocks to know we called the correct method

Issue: Add terminate_workers to ProcessPoolExecutor #128041

On some platforms we seem to get 255 instead of -SIGNAL, so instead use mocks to know we called the correct method

bedevere-bot · 2025-03-03T22:33:12Z

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 89c355a 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F130812%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

csm10495 · 2025-03-03T23:16:36Z

buildbot/AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR looks unrelated.. i think? I'll try updating from base to see if it fixes.

picnixz · 2025-03-03T23:22:00Z

looks unrelated.. i think? I'll try updating from base to see if it fixes.

No that's my bad :) I have #130788 to fix the HMAC tests

picnixz · 2025-03-03T23:32:05Z

In the future, please avoid committing when build bots are running because GitHub doesn't keep the previous build status information.

csm10495 · 2025-03-03T23:40:58Z

Ouch. Sorry about that.

picnixz · 2025-03-03T23:43:25Z

It's fine, I can look at it on the buildbot page. At least, you know now!

csm10495 · 2025-03-03T23:50:37Z

I'm having trouble reproducing the env change error. I've ran it several times over and over with --fail-env-changed on ubuntu. I had trouble getting cpython to build directly on my mac. I guess i need to figure out the mac build since it only seems to happen there.

csm10495 · 2025-03-03T23:54:22Z

I tried pulling down the runner image but it seems to be huge:

tart clone ghcr.io/cirruslabs/macos-runner:sonoma macos-runner
pulling manifest...
pulling disk (218.5 GB compressed)...

Not sure if i have that type of disk space. I'll go back to trying to configure/build locally on mac to see if it repros.

picnixz · 2025-03-03T23:59:06Z

@hugovk Could you perhaps check on macOS please? (I think you're on macOS, right?)

hugovk · 2025-03-04T11:48:46Z

Yes, the changed env is reproducible when running as ./python.exe -m test -v test.test_concurrent_futures.test_process_pool but the whole result is a SUCCESS:

test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s Warning -- reap_children() reaped child process 61094
...
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s Warning -- reap_children() reaped child process 61163
...
----------------------------------------------------------------------
Ran 84 tests in 9.924s
OK (skipped=9)
0:00:10 load avg: 3.42 [1/1/1] test.test_concurrent_futures.test_process_pool failed (env changed)
== Tests result: SUCCESS ==
1 test altered the execution environment (env changed):
test.test_concurrent_futures.test_process_pool
Total duration: 10.0 sec
Total tests: run=84 skipped=9
Total test files: run=1/1 env_changed=1
Result: SUCCESS

Full log

❯ ./python.exe -m test -v test.test_concurrent_futures.test_process_pool
== CPython 3.14.0a5+ (heads/shutdown-workers-transiency:94011b39354, Mar 4 2025, 13:26:05) [Clang 16.0.0 (clang-1600.0.26.6)]
== macOS-15.3.1-arm64-arm-64bit-Mach-O little-endian
== Python build: debug
== cwd: /Users/hugo/github/python/cpython/main/build/test_python_worker_61008æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 813833235
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.59 Run 1 test sequentially in a single process
0:00:00 load avg: 3.59 [1/1] test.test_concurrent_futures.test_process_pool
test_force_shutdown_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers) ... 2.25s ok
test_force_shutdown_workers_dead_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_dead_workers) ... 0.10s ok
test_force_shutdown_workers_invalid_op (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_invalid_op) ... 0.00s ok
test_force_shutdown_workers_not_started_yet (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_not_started_yet) ... 0.00s ok
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s ok
test_free_reference (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_free_reference) ... 0.09s ok
test_idle_process_reuse_multiple (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_idle_process_reuse_multiple) ... skipped 'Incompatible with the fork start method.'
0.00s test_idle_process_reuse_one (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_idle_process_reuse_one) ... skipped 'Incompatible with the fork start method.'
0.00s test_kill_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_kill_workers) ... 0.00s ok
test_killed_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_killed_child) ... 0.01s ok
test_map (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map) ... 0.01s ok
test_map_chunksize (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map_chunksize) ... 0.01s ok
test_map_exception (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map_exception) ... 0.01s ok
test_map_timeout (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map_timeout) ... skipped "resource 'walltime' is not enabled"
test_max_tasks_early_shutdown (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_tasks_early_shutdown) ... skipped 'Incompatible with the fork start method.'
0.00s test_max_tasks_per_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_tasks_per_child) ... 0.00s ok
test_max_tasks_per_child_defaults_to_spawn_context (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_tasks_per_child_defaults_to_spawn_context) ... 0.00s ok
test_max_workers_negative (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_workers_negative) ... 0.00s ok
test_max_workers_too_large (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_workers_too_large) ... skipped 'Windows-only process limit'
test_no_stale_references (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_no_stale_references) ... 0.01s ok
test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_python_finalization_error) ... 0.01s ok
test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 0.03s ok
test_saturation (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_saturation) ... 0.04s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_shutdown_race_issue12456) ... 0.01s ok
test_submit (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_submit) ... 0.01s ok
test_submit_keyword (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_submit_keyword) ... 0.01s ok
test_terminate_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_terminate_workers) ... 0.00s ok
test_traceback (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_traceback) ... 0.01s ok
test_force_shutdown_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers) ... 2.32s ok
test_force_shutdown_workers_dead_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_dead_workers) ... 0.12s ok
test_force_shutdown_workers_invalid_op (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_invalid_op) ... 0.00s ok
test_force_shutdown_workers_not_started_yet (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_not_started_yet) ... 0.00s ok
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s Warning -- reap_children() reaped child process 61094
ok
test_free_reference (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_free_reference) ... 0.11s ok
test_idle_process_reuse_multiple (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_idle_process_reuse_multiple) ... 0.04s ok
test_idle_process_reuse_one (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_idle_process_reuse_one) ... 0.03s ok
test_kill_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_kill_workers) ... 0.00s ok
test_killed_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_killed_child) ... 0.00s ok
test_map (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map) ... 0.02s ok
test_map_chunksize (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map_chunksize) ... 0.02s ok
test_map_exception (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map_exception) ... 0.02s ok
test_map_timeout (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map_timeout) ... skipped "resource 'walltime' is not enabled"
test_max_tasks_early_shutdown (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_tasks_early_shutdown) ... 0.07s ok
test_max_tasks_per_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_tasks_per_child) ... 0.02s ok
test_max_tasks_per_child_defaults_to_spawn_context (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_tasks_per_child_defaults_to_spawn_context) ... 0.00s ok
test_max_workers_negative (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_workers_negative) ... 0.00s ok
test_max_workers_too_large (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_workers_too_large) ... skipped 'Windows-only process limit'
test_no_stale_references (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_no_stale_references) ... 0.03s ok
test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_python_finalization_error) ... 0.00s ok
test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 0.09s ok
test_saturation (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_saturation) ... 0.04s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_shutdown_race_issue12456) ... 0.02s ok
test_submit (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_submit) ... 0.01s ok
test_submit_keyword (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_submit_keyword) ... 0.03s ok
test_terminate_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_terminate_workers) ... 0.00s ok
test_traceback (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_traceback) ... 0.06s ok
test_force_shutdown_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers) ... 2.38s ok
test_force_shutdown_workers_dead_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_dead_workers) ... 0.11s ok
test_force_shutdown_workers_invalid_op (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_invalid_op) ... 0.00s ok
test_force_shutdown_workers_not_started_yet (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_not_started_yet) ... 0.00s ok
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s Warning -- reap_children() reaped child process 61163
ok
test_free_reference (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_free_reference) ... 0.17s ok
test_idle_process_reuse_multiple (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_idle_process_reuse_multiple) ... 0.12s ok
test_idle_process_reuse_one (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_idle_process_reuse_one) ... 0.07s ok
test_kill_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_kill_workers) ... 0.00s ok
test_killed_child (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_killed_child) ... 0.00s ok
test_map (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map) ... 0.06s ok
test_map_chunksize (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map_chunksize) ... 0.06s ok
test_map_exception (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map_exception) ... 0.06s ok
test_map_timeout (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map_timeout) ... skipped "resource 'walltime' is not enabled"
test_max_tasks_early_shutdown (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_tasks_early_shutdown) ... 0.21s ok
test_max_tasks_per_child (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_tasks_per_child) ... 0.09s ok
test_max_tasks_per_child_defaults_to_spawn_context (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_tasks_per_child_defaults_to_spawn_context) ... 0.00s ok
test_max_workers_negative (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_workers_negative) ... 0.00s ok
test_max_workers_too_large (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_workers_too_large) ... skipped 'Windows-only process limit'
test_no_stale_references (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_no_stale_references) ... 0.07s ok
test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_python_finalization_error) ... 0.01s ok
test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 0.19s ok
test_saturation (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_saturation) ... 0.07s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_shutdown_race_issue12456) ... 0.06s ok
test_submit (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_submit) ... 0.05s ok
test_submit_keyword (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_submit_keyword) ... 0.07s ok
test_terminate_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_terminate_workers) ... 0.00s ok
test_traceback (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_traceback) ... 0.10s ok
----------------------------------------------------------------------
Ran 84 tests in 9.924s
OK (skipped=9)
0:00:10 load avg: 3.42 [1/1/1] test.test_concurrent_futures.test_process_pool failed (env changed)
== Tests result: SUCCESS ==
1 test altered the execution environment (env changed):
test.test_concurrent_futures.test_process_pool
Total duration: 10.0 sec
Total tests: run=84 skipped=9
Total test files: run=1/1 env_changed=1
Result: SUCCESS

However, I do get one of the warnings on main:

test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s Warning -- reap_children() reaped child process 64455
...
----------------------------------------------------------------------
Ran 84 tests in 10.066s
OK (skipped=9)
0:00:10 load avg: 13.31 [1/1/1] test.test_concurrent_futures.test_process_pool failed (env changed)
== Tests result: SUCCESS ==
1 test altered the execution environment (env changed):
test.test_concurrent_futures.test_process_pool
Total duration: 10.1 sec
Total tests: run=84 skipped=9
Total test files: run=1/1 env_changed=1
Result: SUCCESS

Full log

❯ ./python.exe -m test -v test.test_concurrent_futures.test_process_pool
== CPython 3.14.0a5+ (heads/main:0b6e98c8417, Mar 4 2025, 13:45:48) [Clang 16.0.0 (clang-1600.0.26.6)]
== macOS-15.3.1-arm64-arm-64bit-Mach-O little-endian
== Python build: debug
== cwd: /Users/hugo/github/python/cpython/main/build/test_python_worker_64366æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 4023528438
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 15.35 Run 1 test sequentially in a single process
0:00:00 load avg: 15.35 [1/1] test.test_concurrent_futures.test_process_pool
test_force_shutdown_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers) ... 2.28s ok
test_force_shutdown_workers_dead_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_dead_workers) ... 0.14s ok
test_force_shutdown_workers_invalid_op (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_invalid_op) ... 0.00s ok
test_force_shutdown_workers_not_started_yet (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_not_started_yet) ... 0.00s ok
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s ok
test_free_reference (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_free_reference) ... 0.10s ok
test_idle_process_reuse_multiple (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_idle_process_reuse_multiple) ... skipped 'Incompatible with the fork start method.'
0.00s test_idle_process_reuse_one (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_idle_process_reuse_one) ... skipped 'Incompatible with the fork start method.'
0.00s test_kill_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_kill_workers) ... 0.00s ok
test_killed_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_killed_child) ... 0.01s ok
test_map (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map) ... 0.01s ok
test_map_chunksize (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map_chunksize) ... 0.01s ok
test_map_exception (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map_exception) ... 0.01s ok
test_map_timeout (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_map_timeout) ... skipped "resource 'walltime' is not enabled"
test_max_tasks_early_shutdown (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_tasks_early_shutdown) ... skipped 'Incompatible with the fork start method.'
0.00s test_max_tasks_per_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_tasks_per_child) ... 0.00s ok
test_max_tasks_per_child_defaults_to_spawn_context (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_tasks_per_child_defaults_to_spawn_context) ... 0.00s ok
test_max_workers_negative (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_workers_negative) ... 0.00s ok
test_max_workers_too_large (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_max_workers_too_large) ... skipped 'Windows-only process limit'
test_no_stale_references (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_no_stale_references) ... 0.01s ok
test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_python_finalization_error) ... 0.01s ok
test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 0.04s ok
test_saturation (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_saturation) ... 0.04s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_shutdown_race_issue12456) ... 0.01s ok
test_submit (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_submit) ... 0.01s ok
test_submit_keyword (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_submit_keyword) ... 0.01s ok
test_terminate_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_terminate_workers) ... 0.00s ok
test_traceback (test.test_concurrent_futures.test_process_pool.ProcessPoolForkProcessPoolExecutorTest.test_traceback) ... 0.01s ok
test_force_shutdown_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers) ... 2.34s ok
test_force_shutdown_workers_dead_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_dead_workers) ... 0.13s ok
test_force_shutdown_workers_invalid_op (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_invalid_op) ... 0.00s ok
test_force_shutdown_workers_not_started_yet (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_not_started_yet) ... 0.00s ok
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s Warning -- reap_children() reaped child process 64455
ok
test_free_reference (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_free_reference) ... 0.12s ok
test_idle_process_reuse_multiple (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_idle_process_reuse_multiple) ... 0.04s ok
test_idle_process_reuse_one (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_idle_process_reuse_one) ... 0.03s ok
test_kill_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_kill_workers) ... 0.00s ok
test_killed_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_killed_child) ... 0.00s ok
test_map (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map) ... 0.02s ok
test_map_chunksize (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map_chunksize) ... 0.02s ok
test_map_exception (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map_exception) ... 0.02s ok
test_map_timeout (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_map_timeout) ... skipped "resource 'walltime' is not enabled"
test_max_tasks_early_shutdown (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_tasks_early_shutdown) ... 0.07s ok
test_max_tasks_per_child (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_tasks_per_child) ... 0.02s ok
test_max_tasks_per_child_defaults_to_spawn_context (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_tasks_per_child_defaults_to_spawn_context) ... 0.00s ok
test_max_workers_negative (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_workers_negative) ... 0.00s ok
test_max_workers_too_large (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_max_workers_too_large) ... skipped 'Windows-only process limit'
test_no_stale_references (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_no_stale_references) ... 0.03s ok
test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_python_finalization_error) ... 0.00s ok
test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 0.10s ok
test_saturation (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_saturation) ... 0.04s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_shutdown_race_issue12456) ... 0.02s ok
test_submit (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_submit) ... 0.01s ok
test_submit_keyword (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_submit_keyword) ... 0.03s ok
test_terminate_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_terminate_workers) ... 0.00s ok
test_traceback (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_traceback) ... 0.06s ok
test_force_shutdown_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers) ... 2.37s ok
test_force_shutdown_workers_dead_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_dead_workers) ... 0.11s ok
test_force_shutdown_workers_invalid_op (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_invalid_op) ... 0.00s ok
test_force_shutdown_workers_not_started_yet (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_not_started_yet) ... 0.00s ok
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... 0.09s ok
test_free_reference (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_free_reference) ... 0.16s ok
test_idle_process_reuse_multiple (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_idle_process_reuse_multiple) ... 0.12s ok
test_idle_process_reuse_one (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_idle_process_reuse_one) ... 0.07s ok
test_kill_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_kill_workers) ... 0.00s ok
test_killed_child (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_killed_child) ... 0.00s ok
test_map (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map) ... 0.06s ok
test_map_chunksize (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map_chunksize) ... 0.06s ok
test_map_exception (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map_exception) ... 0.06s ok
test_map_timeout (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_map_timeout) ... skipped "resource 'walltime' is not enabled"
test_max_tasks_early_shutdown (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_tasks_early_shutdown) ... 0.21s ok
test_max_tasks_per_child (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_tasks_per_child) ... 0.10s ok
test_max_tasks_per_child_defaults_to_spawn_context (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_tasks_per_child_defaults_to_spawn_context) ... 0.00s ok
test_max_workers_negative (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_workers_negative) ... 0.00s ok
test_max_workers_too_large (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_max_workers_too_large) ... skipped 'Windows-only process limit'
test_no_stale_references (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_no_stale_references) ... 0.07s ok
test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_python_finalization_error) ... 0.01s ok
test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 0.19s ok
test_saturation (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_saturation) ... 0.07s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_shutdown_race_issue12456) ... 0.06s ok
test_submit (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_submit) ... 0.05s ok
test_submit_keyword (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_submit_keyword) ... 0.07s ok
test_terminate_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_terminate_workers) ... 0.00s ok
test_traceback (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_traceback) ... 0.11s ok
----------------------------------------------------------------------
Ran 84 tests in 10.066s
OK (skipped=9)
0:00:10 load avg: 13.31 [1/1/1] test.test_concurrent_futures.test_process_pool failed (env changed)
== Tests result: SUCCESS ==
1 test altered the execution environment (env changed):
test.test_concurrent_futures.test_process_pool
Total duration: 10.1 sec
Total tests: run=84 skipped=9
Total test files: run=1/1 env_changed=1
Result: SUCCESS

encukou · 2025-03-04T11:59:59Z

I can also reproduce.

Details

 ./python.exe -m test test_concurrent_futures.test_process_pool -m '*SpawnProcess*test_force_shutdown_workers_stops_pool*' -v --fail-env-changed
== CPython 3.14.0a5+ (heads/shutdown-workers-transiency:94011b39354, Mar 4 2025, 11:47:35) [Clang 16.0.0 (clang-1600.0.26.4)]
== macOS-15.1-arm64-arm-64bit-Mach-O little-endian
== Python build: debug
== cwd: /Users/encukou/dev/cpython/build/test_python_worker_83984æ
== CPU count: 8
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 3015704270
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 1.48 Run 1 test sequentially in a single process
0:00:00 load avg: 1.48 [1/1] test_concurrent_futures.test_process_pool
test_force_shutdown_workers_stops_pool (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_force_shutdown_workers_stops_pool) ... terminate_workers
kill_workers
0.08s Warning -- reap_children() reaped child process 83989
ok
----------------------------------------------------------------------
Ran 1 test in 0.112s
OK
0:00:00 load avg: 1.48 [1/1/1] test_concurrent_futures.test_process_pool failed (env changed)
== Tests result: ENV CHANGED ==
1 test altered the execution environment (env changed):
test_concurrent_futures.test_process_pool
Total duration: 180 ms
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) env_changed=1
Result: ENV CHANGED

Commenting out self.assertRaises(RuntimeError, executor.submit, time.sleep, 0) makes the test pass.

csm10495 · 2025-03-04T16:43:17Z

@colesbury backed out the original change. (Thanks!).

I'm playing with this more and finally got my mac to build. Just in case someone Googles around and needs a similar fix, this worked for me:
I was getting:

...
Undefined symbols for architecture arm64:
"_libintl_bindtextdomain", referenced from:
__locale_bindtextdomain in _localemodule.o
__locale_bindtextdomain in _localemodule.o
"_libintl_dcgettext", referenced from:
__locale_dcgettext in _localemodule.o
"_libintl_dgettext", referenced from:
__locale_dgettext in _localemodule.o
"_libintl_gettext", referenced from:
__locale_gettext in _localemodule.o
"_libintl_setlocale", referenced from:
__locale_setlocale in _localemodule.o
__locale_setlocale in _localemodule.o
__locale_localeconv in _localemodule.o
__locale_localeconv in _localemodule.o
__locale_localeconv in _localemodule.o
__locale_localeconv in _localemodule.o
__locale_nl_langinfo_impl in _localemodule.o
__locale_nl_langinfo_impl in _localemodule.o
__locale_nl_langinfo_impl in _localemodule.o
__locale_nl_langinfo_impl in _localemodule.o
...
"_libintl_textdomain", referenced from:
__locale_textdomain in _localemodule.o
ld: symbol(s) not found for architecture arm6

re-configuring with:

./configure LDFLAGS="-L/opt/homebrew/lib"

then running make -j8 worked and i got a mac build. I can finally reproduce. It's really transient and I can only see it if I do that final call to submit. It seems to not happen if I get rid of that, which doesn't make a ton of sense. I'll play around with it and get a fresh PR out with both this fix and that fix and the original stuff back when I get to it.

Thanks again folks.

Try to fix transiency of test_force_shutdown_workers
89c355a
On some platforms we seem to get 255 instead of -SIGNAL, so instead use mocks to know we called the correct method

bedevere-appBot added tests Tests in the Lib/test dir awaiting review labels Mar 3, 2025

bedevere-appBot mentioned this pull request Mar 3, 2025
Add terminate_workers to ProcessPoolExecutor#128041
Closed

colesbury added the skip news label Mar 3, 2025

picnixz added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 3, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 3, 2025

Merge branch 'main' into shutdown-workers-transiency
94011b3

csm10495 closed this Mar 4, 2025

Navigation Menu

Uh oh!

Uh oh!

gh-128041: Try to fix transiency of test_force_shutdown_workers - #130812

gh-128041: Try to fix transiency of test_force_shutdown_workers#130812
csm10495 wants to merge 2 commits into
python:mainfrom
csm10495:shutdown-workers-transiency

csm10495 commented Mar 3, 2025•
edited by bedevere-app Bot
Loading

Uh oh!

bedevere-bot commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

picnixz commented Mar 3, 2025•
edited
Loading

Uh oh!

picnixz commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

picnixz commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

picnixz commented Mar 3, 2025

Uh oh!

hugovk commented Mar 4, 2025

Uh oh!

encukou commented Mar 4, 2025

Uh oh!

csm10495 commented Mar 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Uh oh!

Conversation

csm10495 commented Mar 3, 2025• edited by bedevere-app BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

picnixz commented Mar 3, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

picnixz commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

csm10495 commented Mar 3, 2025

Uh oh!

picnixz commented Mar 3, 2025

Uh oh!

hugovk commented Mar 4, 2025

Uh oh!

encukou commented Mar 4, 2025

Uh oh!

csm10495 commented Mar 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

csm10495 commented Mar 3, 2025•
edited by bedevere-app Bot
Loading

picnixz commented Mar 3, 2025•
edited
Loading