gh-107219: Fix concurrent.futures terminate_broken()

[vstinner]

Fix a race condition in _ExecutorManagerThread.terminate_broken(): ignore the InvalidStateError on future.set_exception(). It can happen if the future is cancelled before the caller.

Moreover, test_crash_big_data() now waits explicitly until the executor completes.

• Issue: test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219

[vstinner]

This change doesn't fix the most annoying bug of issue #107219. test_crash_big_data() still hangs on Windows. But it fix this error:

test_crash_big_data (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest.test_crash_big_data) ... Warning -- Uncaugh
t thread exception: InvalidStateError
Exception in thread Thread-9:
Traceback (most recent call last):
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\threading.py", line 1059, in _bootstrap_inner
self.run()
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 344, in run
self.terminate_broken(cause)
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 492, in terminate_broken
work_item.future.set_exception(bpe)
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\_base.py", line 559, in set_exception
raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2cf561cf260 state=cancelled>
0.55s Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
Warning -- Dangling thread: <_MainThread(MainThread, started 11880)>
Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)>
ok

link: #107219 (comment)

[vstinner]

cc @pitrou

[vstinner]

This change doesn't fix the most annoying bug of issue #107219. test_crash_big_data() still hangs on Windows.

If it wasn't obvious enough, the test decided to hang on Windows x64 CI :-)

0:41:15 load avg: 0.02 [462/462/3] test.test_concurrent_futures.test_deadlock process crashed (Exit code 1)
Timeout (0:20:00)!
Thread 0x00000c90 (most recent call first):
File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 282 in _send_bytes
File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 199 in send_bytes
File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 246 in _feed
File "D:\a\cpython\cpython\Lib\threading.py", line 996 in run
File "D:\a\cpython\cpython\Lib\threading.py", line 1059 in _bootstrap_inner
File "D:\a\cpython\cpython\Lib\threading.py", line 1016 in _bootstrap
Thread 0x00001b78 (most recent call first):
File "D:\a\cpython\cpython\Lib\threading.py", line 1153 in _wait_for_tstate_lock
File "D:\a\cpython\cpython\Lib\threading.py", line 1133 in join
File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 199 in _finalize_join
File "D:\a\cpython\cpython\Lib\multiprocessing\util.py", line 224 in __call__
File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 151 in join_thread
File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 560 in join_executor_internals
File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 514 in terminate_broken
File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 344 in run
File "D:\a\cpython\cpython\Lib\threading.py", line 1059 in _bootstrap_inner
File "D:\a\cpython\cpython\Lib\threading.py", line 1016 in _bootstrap
Thread 0x000012cc (most recent call first):
File "D:\a\cpython\cpython\Lib\threading.py", line 1153 in _wait_for_tstate_lock
File "D:\a\cpython\cpython\Lib\threading.py", line 1133 in join
File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 843 in shutdown
File "D:\a\cpython\cpython\Lib\concurrent\futures\_base.py", line 647 in __exit__
File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures\test_deadlock.py", line 236 in test_crash_big_data