GH-66285: fix forking in asyncio

[kumaraditya303]

Closes#66285
Closes#94454

• Issue: asyncio: support multiprocessing (support fork) #66285

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit d6ef1a6 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[gvanrossum]

See my comments on the issue, @kumaraditya303 .

[kumaraditya303]

See my comments on the issue,

@gvanrossum: I have added more tests and also added a subprocess test.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x has failed when building commit 0c1fbc1.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/469/builds/3768) and take a look at the build logs.
4. Check if the failure is related to this commit (0c1fbc1) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/469/builds/3768

Failed tests:

• test_asyncio

Failed subtests:

• test_fork_signal_handling - test.test_asyncio.test_unix_events.TestFork.test_fork_signal_handling
• test_fork_asyncio_run - test.test_asyncio.test_unix_events.TestFork.test_fork_asyncio_run
• test_fork_asyncio_subprocess - test.test_asyncio.test_unix_events.TestFork.test_fork_asyncio_subprocess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

416 tests OK.

10 slowest tests:

• test_gdb: 6 min 6 sec
• test_concurrent_futures: 2 min 46 sec
• test_capi: 2 min 7 sec
• test_asyncio: 1 min 57 sec
• test_multiprocessing_spawn: 1 min 44 sec
• test_multiprocessing_forkserver: 1 min 10 sec
• test_multiprocessing_fork: 1 min 3 sec
• test_tokenize: 59.6 sec
• test_signal: 48.9 sec
• test_socket: 46.8 sec

1 test failed:
test_asyncio

16 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_peg_generator test_startfile
test_tix test_tkinter test_ttk test_winconsoleio test_winreg
test_winsound test_wmi test_zipfile64

1 re-run test:
test_asyncio

Total duration: 10 min 8 sec

Click to see traceback logs

Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethodifself._callMaybeAsync(method) isnotNone:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsyncreturnself._asyncioTestContext.run(func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1953, in test_fork_asyncio_subprocess
result = manager.Value('i', 1)
^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
token, exp =self._create(typeid, *args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
conn =self._Client(self._address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
answer_challenge(c, authkey)
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
digest = hmac.new(authkey, message, 'md5').digest()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in newreturn HMAC(key, msg, digestmod)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__self._init_hmac(key, msg, digestmod)
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmacself._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: no reason supplied
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethodifself._callMaybeAsync(method) isnotNone:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsyncreturnself._asyncioTestContext.run(func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1902, in test_fork_signal_handling
child_started = manager.Event()
^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
token, exp =self._create(typeid, *args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
conn =self._Client(self._address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
answer_challenge(c, authkey)
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
digest = hmac.new(authkey, message, 'md5').digest()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in newreturn HMAC(key, msg, digestmod)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__self._init_hmac(key, msg, digestmod)
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmacself._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: no reason supplied
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethodifself._callMaybeAsync(method) isnotNone:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsyncreturnself._asyncioTestContext.run(func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1937, in test_fork_asyncio_run
result = manager.Value('i', 0)
^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
token, exp =self._create(typeid, *args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
conn =self._Client(self._address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
answer_challenge(c, authkey)
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
digest = hmac.new(authkey, message, 'md5').digest()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in newreturn HMAC(key, msg, digestmod)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__self._init_hmac(key, msg, digestmod)
File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmacself._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: no reason supplied

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 0c1fbc1.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/954) and take a look at the build logs.
4. Check if the failure is related to this commit (0c1fbc1) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/954

Failed tests:

• test_traceback

Failed subtests:

• test_import_from_error_bad_suggestions_do_not_trigger_for_small_names - test.test_traceback.CPythonSuggestionFormattingTests.test_import_from_error_bad_suggestions_do_not_trigger_for_small_names

Summary of the results of the build (if available):

== Tests result: FAILURE ==

325 tests OK.

10 slowest tests:

• test_tokenize: 51.3 sec
• test_unparse: 42.7 sec
• test_capi: 33.1 sec
• test_lib2to3: 29.2 sec
• test_unicodedata: 25.0 sec
• test_decimal: 16.0 sec
• test_statistics: 14.2 sec
• test_pickle: 13.1 sec
• test_zipfile: 9.4 sec
• test_buffer: 7.5 sec

1 test failed:
test_traceback

107 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio test_bz2
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_gzip test_httplib test_httpservers test_idle
test_imaplib test_interpreters test_ioctl test_kqueue
test_launcher test_lzma test_mailbox test_mmap test_msilib
test_multiprocessing_fork test_multiprocessing_forkserver
test_multiprocessing_main_handling test_multiprocessing_spawn
test_nis test_openpty test_ossaudiodev test_pdb test_peg_generator
test_perf_profiler test_pipes test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_spwd test_sqlite3 test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib test_urllib2 test_urllib2_localnet test_urllib2net
test_urllib_response test_urllibnet test_venv test_wait3
test_wait4 test_webbrowser test_winconsoleio test_winreg
test_winsound test_wmi test_wsgiref test_xmlrpc test_xmlrpc_net
test_xxlimited test_zipfile64 test_zipimport_support test_zlib
test_zoneinfo
0:04:47 load avg: 2.78
0:04:47 load avg: 2.78 Re-running failed tests is not supported with --python host runner option.

Total duration: 4 min 47 sec

Click to see traceback logs

TracebackFormatTests.test_format_stack) ... ok
TracebackFormatTests.test_exception_group_deep_recursion_traceback) ... ok
TracebackCases.test_format_exception_only_exc) ... ok
TracebackException.test_compact_no_cause) ... ok
TracebackCases.test_base_exception) ... ok
TracebackErrorCaretTests.test_specialization_variations) ... ok
TracebackException.test_locals) ... ok
TracebackFormatTests.test_print_exception_bad_type_python) ... ok
TracebackErrorCaretTests.test_caret_multiline_expression) ... ok
TracebackCases.test_print_traceback_at_exit) ... skipped 'requires subprocess support'
TracebackErrorCaretTests.test_traceback_specialization_with_syntax_error) ... ok
TracebackException.test_compact_with_cause) ... ok
TracebackFormatTests.test_recursive_traceback_python) ... ok
TracebackException.test_long_context_chain) ... ok
TracebackException_ExceptionGroups.test_exception_group_format_exception_only) ... ok
TracebackFormatTests.test_print_exception_bad_type_capi) ... ok
TracebackCases.test_extract_stack) ... ok
TracebackCases.test_nocaret) ... ok
TracebackErrorCaretTests.test_multiline_method_call_a) ... ok
TracebackCases.test_no_caret_with_no_debug_ranges_flag) ... skipped 'requires subprocess support'
TracebackErrorCaretTests.test_traceback_very_long_line) ... ok
TracebackErrorCaretTests.test_caret_exception_group) ... ok
TracebackErrorCaretTests.test_basic_caret) ... ok
Traceback (most recent call last):
File "/Lib/test/test_traceback.py", line 3173, in test_import_from_error_bad_suggestions_do_not_trigger_for_small_namesself.assertNotIn("mom", actual)
AssertionError: 'mom' unexpectedly found in "ImportError: cannot import name 'b' from 'nmQWXqKJcBDACXLN' (/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.wasi/build/build_oot/host/build/test_python_496508æ/test_python_worker_830271æ/tmp6fv8mom7/nmQWXqKJcBDACXLN.py)"
TracebackException.test_comparison_equivalent_exceptions_are_equal) ... ok
TracebackErrorCaretTests.test_multiline_method_call_c) ... ok
TracebackErrorCaretTests.test_caret_for_subscript_unicode) ... ok
TracebackCases.test_no_caret_with_no_debug_ranges_flag_python_traceback) ... skipped 'requires subprocess support'
TracebackErrorCaretTests.test_multiline_method_call_b) ... ok
TracebackCases.test_recursion_error_during_traceback) ... skipped 'requires subprocess support'
TracebackErrorCaretTests.test_decorator_application_lineno_correct) ... ok
TracebackErrorCaretTests.test_caret_in_type_annotation) ... ok
TracebackException.test_no_refs_to_exception_and_traceback_objects) ... ok
TracebackFormatTests.test_recursive_traceback_cpython_internal) ... ok
TracebackFormatTests.test_stack_format) ... ok
TracebackErrorCaretTests.test_wide_characters_unicode_with_problematic_byte_offset) ... ok
TracebackCases.test_caret) ... ok
TracebackCases.test_format_exception_exc) ... ok
TracebackErrorCaretTests.test_secondary_caret_not_elided) ... ok
TracebackException.test_from_exception) ... ok
TracebackCases.test_print_exception) ... ok
TracebackException.test_print) ... ok
TracebackException.test_no_locals) ... ok
TracebackCases.test_clear) ... ok
TracebackCases.test_format_exception_only_bad__str__) ... ok
TracebackCases.test_exception_is_None) ... ok
TracebackException.test_lookup_lines) ... ok
TracebackException_ExceptionGroups.test_max_group_width) ... ok
TracebackCases.test_bad_indentation) ... ok
TracebackException.test_traceback_header) ... ok
TracebackCases.test_signatures) ... ok
TracebackException_ExceptionGroups.test_exception_group_format) ... ok
TracebackFormatTests.test_print_stack) ... ok
TracebackErrorCaretTests.test_caret_for_binary_operators_with_unicode) ... ok
TracebackErrorCaretTests.test_line_with_unicode) ... ok
TracebackFormatTests.test_exception_group_deep_recursion_capi) ... ok
TracebackException.test_limit) ... ok
TracebackErrorCaretTests.test_caret_for_binary_operators) ... ok
TracebackException.test_comparison_basic) ... ok
TracebackErrorCaretTests.test_byte_offset_with_wide_characters_middle) ... ok
TracebackErrorCaretTests.test_caret_for_subscript) ... ok
TracebackException_ExceptionGroups.test_comparison) ... ok
TracebackErrorCaretTests.test_byte_offset_multiline) ... ok
TracebackException.test_smoke) ... ok
TracebackErrorCaretTests.test_caret_multiline_expression_bin_op) ... ok
TracebackCases.test_encoded_file) ... skipped 'requires subprocess support'
TracebackErrorCaretTests.test_caret_for_binary_operators_two_char) ... ok
TracebackFormatTests.test_traceback_format_with_cleared_frames) ... ok
TracebackFormatTests.test_unhashable) ... ok
TracebackException_ExceptionGroups.test_max_group_depth) ... ok
TracebackException.test_cause) ... ok
TracebackFormatTests.test_traceback_format) ... ok
TracebackException.test_comparison_params_variations) ... ok
TracebackException.test_context) ... ok
TracebackException_ExceptionGroups.test_exception_group_construction) ... ok
TracebackErrorCaretTests.test_caret_multiline_expression_syntax_error) ... ok
TracebackException.test_unhashable) ... ok
TracebackCases.test_print_exception_exc) ... ok

[brettcannon]

Looks like the failure was random as subsequent builds succeeded.

https://buildbot.python.org/all/#/builders/1046

[vstinner]

Yeah, I saw the test_traceback failure, but it only occurred once, so I ignored it.