gh-119819: Fix regression to allow logging configuration with multipr… #120030

vsajip · 2024-06-04T11:28:15Z

…ocessing queue types.

Issue: Python 3.12+ breaks backwards compatibility for logging QueueHandler with some Queue classes #119819

…ultiprocessing queue types.

miss-islington-app · 2024-06-04T12:21:14Z

Thanks @vsajip for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

miss-islington-app · 2024-06-04T12:21:14Z

Thanks @vsajip for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

…ultipr… (pythonGH-120030) (cherry picked from commit 99d945c) Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>

bedevere-app · 2024-06-04T12:21:23Z

GH-120034 is a backport of this pull request to the 3.12 branch.

…ultipr… (pythonGH-120030) (cherry picked from commit 99d945c) Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>

bedevere-app · 2024-06-04T12:21:27Z

GH-120035 is a backport of this pull request to the 3.13 branch.

…multipr… (GH-120030) (GH-120034) (cherry picked from commit 99d945c)

…multipr… (GH-120030) (GH-120035) (cherry picked from commit 99d945c)

bedevere-bot · 2024-06-04T12:59:11Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x has failed when building commit 99d945c.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1380/builds/507) and take a look at the build logs.
Check if the failure is related to this commit (99d945c) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1380/builds/507

Failed tests:

test_logging

Failed subtests:

test_multiprocessing_queues - test.test_logging.ConfigDictTest.test_multiprocessing_queues
test_config_queue_handler - test.test_logging.ConfigDictTest.test_config_queue_handler

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 577, in configure
    handler = self.configure_handler(handlers[name])
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 785, in configure_handler
    proxy_queue = MM().Queue()
                  ~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/context.py", line 57, in Manager
    m.start()
    ~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/managers.py", line 569, in start
    self._address = reader.recv()
                    ~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3916, in test_config_queue_handler
    self.do_queuehandler_configuration(qspec, lspec)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3870, in do_queuehandler_configuration
    self.apply_config(cd)
    ~~~~~~~~~~~~~~~~~^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3248, in apply_config
    logging.config.dictConfig(conf)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 911, in dictConfig
    dictConfigClass(config).configure()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/logging/config.py", line 584, in configure
    raise ValueError('Unable to configure handler '
                     '%r' % name) from e
ValueError: Unable to configure handler 'ah'


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/test/test_logging.py", line 3934, in test_multiprocessing_queues
    q1 = MQ()  # this can't be pickled
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/queues.py", line 38, in __init__
    from .synchronize import SEM_VALUE_MAX as maxsize
  File "/Users/buildbot/Library/Developer/XCTestDevices/7A3EB9F9-D350-462D-BB82-3EAAC58EB7AA/data/Containers/Bundle/Application/10922577-9A1D-434B-9BD7-FD5EEB6AD890/iOSTestbed.app/python/lib/python3.14/multiprocessing/synchronize.py", line 17, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'

mhsmith · 2024-06-04T13:27:08Z

Lib/test/test_logging.py

@@ -3926,6 +3926,32 @@ def test_config_queue_handler(self):
            msg = str(ctx.exception)
            self.assertEqual(msg, "Unable to configure handler 'ah'")

+    @unittest.skipIf(support.is_wasi, "WASI does not have multiprocessing.")


Instead of checking specifically for WASI, tests that use subprocesses should use @support.requires_subprocess().

But this test doesn't use subprocess, only multiprocessing - I'm not sure if that requires_subprocess is appropriate. I can't (yet) tell whether the problem is inherent to the platform (iOS ARM64 Simulator) or, if I need to tighten the skip conditions for the test, exactly what the condition should be. It doesn't appear, from the tracebacks, that multiprocessing isn't supported - on WASI, for example, you get an error because _multiprocessing can't be imported. I'll do some more digging.

The error on iOS is exactly the same: see the bottom of the traceback above. The fact that multiprocessing isn't available on iOS is already mentioned in the documentation.

An even more specific check would be support.skip_if_broken_multiprocessing_synchronize(), which is already used a few times in test_logging.

The error on iOS is exactly the same

Yes, I missed that for some reason. Perhaps the answer is to add a support.skip_if_no_multiprocessing, as support.skip_if_broken_multiprocessing_synchronize seems too specific for this particular test.

Rather than adding a specific skip function, you could just do import_helper.import_module('_multiprocessing').

Yes, I've already done that 😄

bedevere-bot · 2024-06-04T13:39:21Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.13 has failed when building commit 720a44d.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1386/builds/159) and take a look at the build logs.
Check if the failure is related to this commit (720a44d) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1386/builds/159

Failed tests:

test_logging

Failed subtests:

test_multiprocessing_queues - test.test_logging.ConfigDictTest.test_multiprocessing_queues
test_config_queue_handler - test.test_logging.ConfigDictTest.test_config_queue_handler

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 577, in configure
    handler = self.configure_handler(handlers[name])
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 785, in configure_handler
    proxy_queue = MM().Queue()
                  ~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/context.py", line 57, in Manager
    m.start()
    ~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/managers.py", line 569, in start
    self._address = reader.recv()
                    ~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3934, in test_multiprocessing_queues
    q1 = MQ()  # this can't be pickled
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/queues.py", line 38, in __init__
    from .synchronize import SEM_VALUE_MAX as maxsize
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/multiprocessing/synchronize.py", line 17, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3916, in test_config_queue_handler
    self.do_queuehandler_configuration(qspec, lspec)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3870, in do_queuehandler_configuration
    self.apply_config(cd)
    ~~~~~~~~~~~~~~~~~^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/test/test_logging.py", line 3248, in apply_config
    logging.config.dictConfig(conf)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 911, in dictConfig
    dictConfigClass(config).configure()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/640DD7EB-8493-4634-8DEC-7C97E593BF6F/data/Containers/Bundle/Application/7FA6A294-B3E2-4D50-BE13-B50B25689F58/iOSTestbed.app/python/lib/python3.13/logging/config.py", line 584, in configure
    raise ValueError('Unable to configure handler '
                     '%r' % name) from e
ValueError: Unable to configure handler 'ah'

…ultipr… (pythonGH-120030)

pythongh-119819: Fix regression to allow logging configuration with m…

6125ac2

…ultiprocessing queue types.

vsajip self-assigned this Jun 4, 2024

bedevere-app bot added the awaiting core review label Jun 4, 2024

bedevere-app bot mentioned this pull request Jun 4, 2024

Python 3.12+ breaks backwards compatibility for logging QueueHandler with some Queue classes #119819

Closed

Skip test for WASI.

cf2e6ed

vsajip merged commit 99d945c into python:main Jun 4, 2024
33 checks passed

vsajip deleted the fix-119819 branch June 4, 2024 12:20

bedevere-app bot removed the awaiting core review label Jun 4, 2024

vsajip added needs backport to 3.12 bug and security fixes needs backport to 3.13 bugs and security fixes labels Jun 4, 2024

bedevere-app bot removed the needs backport to 3.12 bug and security fixes label Jun 4, 2024

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jun 4, 2024

vsajip pushed a commit that referenced this pull request Jun 4, 2024

[3.12] gh-119819: Fix regression to allow logging configuration with …

0e150c3

…multipr… (GH-120030) (GH-120034) (cherry picked from commit 99d945c)

vsajip pushed a commit that referenced this pull request Jun 4, 2024

[3.13] gh-119819: Fix regression to allow logging configuration with …

720a44d

…multipr… (GH-120030) (GH-120035) (cherry picked from commit 99d945c)

mhsmith reviewed Jun 4, 2024

View reviewed changes

barneygale pushed a commit to barneygale/cpython that referenced this pull request Jun 5, 2024

pythongh-119819: Fix regression to allow logging configuration with m…

aff9a10

…ultipr… (pythonGH-120030)

provinzkraut mentioned this pull request Jun 22, 2024

3.12.4 breakslogging.config.DictConfig with logging.handlers.QueueHandler on read-only file systems #120868

Closed

noahbkim pushed a commit to hudson-trading/cpython that referenced this pull request Jul 11, 2024

pythongh-119819: Fix regression to allow logging configuration with m…

d851700

…ultipr… (pythonGH-120030)

estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024

pythongh-119819: Fix regression to allow logging configuration with m…

f01bcec

…ultipr… (pythonGH-120030)

provinzkraut mentioned this pull request Jul 26, 2024

Bug: Litestar Logging in Python3.12: Unable to configure handler 'queue_listener' (Pycharm Debugger) litestar-org/litestar#3647

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-119819: Fix regression to allow logging configuration with multipr… #120030

gh-119819: Fix regression to allow logging configuration with multipr… #120030

vsajip commented Jun 4, 2024 •

edited by bedevere-app bot

Loading

miss-islington-app bot commented Jun 4, 2024

miss-islington-app bot commented Jun 4, 2024

bedevere-app bot commented Jun 4, 2024

bedevere-app bot commented Jun 4, 2024

bedevere-bot commented Jun 4, 2024

mhsmith Jun 4, 2024

vsajip Jun 4, 2024 •

edited

Loading

mhsmith Jun 4, 2024

vsajip Jun 4, 2024

mhsmith Jun 4, 2024

vsajip Jun 4, 2024

bedevere-bot commented Jun 4, 2024

gh-119819: Fix regression to allow logging configuration with multipr… #120030

gh-119819: Fix regression to allow logging configuration with multipr… #120030

Conversation

vsajip commented Jun 4, 2024 • edited by bedevere-app bot Loading

miss-islington-app bot commented Jun 4, 2024

miss-islington-app bot commented Jun 4, 2024

bedevere-app bot commented Jun 4, 2024

bedevere-app bot commented Jun 4, 2024

bedevere-bot commented Jun 4, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

mhsmith Jun 4, 2024

Choose a reason for hiding this comment

vsajip Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

mhsmith Jun 4, 2024

Choose a reason for hiding this comment

vsajip Jun 4, 2024

Choose a reason for hiding this comment

mhsmith Jun 4, 2024

Choose a reason for hiding this comment

vsajip Jun 4, 2024

Choose a reason for hiding this comment

bedevere-bot commented Jun 4, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

vsajip commented Jun 4, 2024 •

edited by bedevere-app bot

Loading

vsajip Jun 4, 2024 •

edited

Loading