Skip to content

gh-119819: Fix regression to allow logging configuration with multipr… - #120030

Merged
vsajip merged 2 commits into
python:mainfrom
vsajip:fix-119819
Jun 4, 2024
Merged

gh-119819: Fix regression to allow logging configuration with multipr…#120030
vsajip merged 2 commits into
python:mainfrom
vsajip:fix-119819

Conversation

@vsajip

@vsajipvsajip commented Jun 4, 2024

Copy link
Copy Markdown
Member

@vsajip
vsajip merged commit 99d945c into python:mainJun 4, 2024
@vsajip
vsajip deleted the fix-119819 branch June 4, 2024 12:20
@vsajipvsajip added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Jun 4, 2024
@miss-islington-app

Copy link
Copy Markdown

Thanks @vsajip for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

@miss-islington-app

Copy link
Copy Markdown

Thanks @vsajip for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 4, 2024
…ultipr… (pythonGH-120030)
(cherry picked from commit 99d945c)
Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>
@bedevere-app

Copy link
Copy Markdown

GH-120034 is a backport of this pull request to the 3.12 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.12 only security fixes label Jun 4, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 4, 2024
…ultipr… (pythonGH-120030)
(cherry picked from commit 99d945c)
Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>
@bedevere-app

Copy link
Copy Markdown

GH-120035 is a backport of this pull request to the 3.13 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.13 bugs and security fixes label Jun 4, 2024
vsajip pushed a commit that referenced this pull request Jun 4, 2024
vsajip pushed a commit that referenced this pull request Jun 4, 2024
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x has failed when building commit 99d945c.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1380/builds/507) and take a look at the build logs.
  4. Check if the failure is related to this commit (99d945c) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1380/builds/507

Failed tests:

  • test_logging

Failed subtests:

  • test_multiprocessing_queues - test.test_logging.ConfigDictTest.test_multiprocessing_queues
  • test_config_queue_handler - test.test_logging.ConfigDictTest.test_config_queue_handler

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 577, in configure
handler =self.configure_handler(handlers[name])
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 785, in configure_handler
proxy_queue = MM().Queue()
~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/context.py", line 57, in Manager
m.start()
~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/managers.py", line 569, in startself._address = reader.recv()
~~~~~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/connection.py", line 250, in recv
buf =self._recv_bytes()
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/connection.py", line 430, in _recv_bytes
buf =self._recv(4)
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/connection.py", line 399, in _recvraiseEOFError
EOFError
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3916, in test_config_queue_handlerself.do_queuehandler_configuration(qspec, lspec)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3870, in do_queuehandler_configurationself.apply_config(cd)
~~~~~~~~~~~~~~~~~^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3248, in apply_config
logging.config.dictConfig(conf)
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 911, in dictConfig
dictConfigClass(config).configure()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 584, in configureraiseValueError('Unable to configure handler ''%r'% name) from e
ValueError: Unable to configure handler 'ah'
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3934, in test_multiprocessing_queues
q1 = MQ() # this can't be pickled
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/context.py", line 103, in Queuereturn Queue(maxsize, ctx=self.get_context())
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/queues.py", line 38, in __init__from .synchronize importSEM_VALUE_MAXas maxsize
File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/synchronize.py", line 17, in <module>import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'

msg = str(ctx.exception)
self.assertEqual(msg, "Unable to configure handler 'ah'")

@unittest.skipIf(support.is_wasi, "WASI does not have multiprocessing.")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of checking specifically for WASI, tests that use subprocesses should use @support.requires_subprocess().

@vsajipvsajipJun 4, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this test doesn't use subprocess, only multiprocessing - I'm not sure if that requires_subprocess is appropriate. I can't (yet) tell whether the problem is inherent to the platform (iOS ARM64 Simulator) or, if I need to tighten the skip conditions for the test, exactly what the condition should be. It doesn't appear, from the tracebacks, that multiprocessing isn't supported - on WASI, for example, you get an error because _multiprocessing can't be imported. I'll do some more digging.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error on iOS is exactly the same: see the bottom of the traceback above. The fact that multiprocessing isn't available on iOS is already mentioned in the documentation.

An even more specific check would be support.skip_if_broken_multiprocessing_synchronize(), which is already used a few times in test_logging.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error on iOS is exactly the same

Yes, I missed that for some reason. Perhaps the answer is to add a support.skip_if_no_multiprocessing, as support.skip_if_broken_multiprocessing_synchronize seems too specific for this particular test.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than adding a specific skip function, you could just do import_helper.import_module('_multiprocessing').

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I've already done that 😄

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.13 has failed when building commit 720a44d.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1386/builds/159) and take a look at the build logs.
  4. Check if the failure is related to this commit (720a44d) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1386/builds/159

Failed tests:

  • test_logging

Failed subtests:

  • test_multiprocessing_queues - test.test_logging.ConfigDictTest.test_multiprocessing_queues
  • test_config_queue_handler - test.test_logging.ConfigDictTest.test_config_queue_handler

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 577, in configure
handler =self.configure_handler(handlers[name])
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 785, in configure_handler
proxy_queue = MM().Queue()
~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/context.py", line 57, in Manager
m.start()
~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/managers.py", line 569, in startself._address = reader.recv()
~~~~~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/connection.py", line 250, in recv
buf =self._recv_bytes()
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
buf =self._recv(4)
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/connection.py", line 399, in _recvraiseEOFError
EOFError
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3934, in test_multiprocessing_queues
q1 = MQ() # this can't be pickled
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/context.py", line 103, in Queuereturn Queue(maxsize, ctx=self.get_context())
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/queues.py", line 38, in __init__from .synchronize importSEM_VALUE_MAXas maxsize
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/synchronize.py", line 17, in <module>import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3916, in test_config_queue_handlerself.do_queuehandler_configuration(qspec, lspec)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3870, in do_queuehandler_configurationself.apply_config(cd)
~~~~~~~~~~~~~~~~~^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3248, in apply_config
logging.config.dictConfig(conf)
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 911, in dictConfig
dictConfigClass(config).configure()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 584, in configureraiseValueError('Unable to configure handler ''%r'% name) from e
ValueError: Unable to configure handler 'ah'

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@vsajip@bedevere-bot@mhsmith