gh-99595: Fix bug in multiprocessing.managers.BaseProxy that causes AttributeError when passing a proxy between processes #99598
Conversation
…ssing a proxy between processes When a proxy derived from multiprocessing.managers.BaseProxy is passed between processes, the self._manager attribute becomes None. self._manager is however accessed to retrieve its _registry attribute, which contain the method_to_typeid map recorded into the manager viaBaseManager.register(), causing an AttributeError.
…uses AttributeError when passing a proxy between processes Decouple the role of the manager from the role of the registry in the proxy and initialize the proxy with the registry instance separately from the manager instance.
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
Fixed typo in blurb
| @@ -755,7 +754,7 @@ class BaseProxy(object): | |||
| _address_to_local = {} | |||
| _mutex = util.ForkAwareThreadLock() | |||
|
|
|||
| def __init__(self, token, serializer, manager=None, | |||
| def __init__(self, token, serializer, manager=None, registry=None, | |||
There was a problem hiding this comment.
Adding a positional parameter before other positional parameters is not compatible and can break existing code. You should only add it at the end.
| @@ -824,11 +824,11 @@ def _callmethod(self, methodname, args=(), kwds={}): | |||
| return result | |||
| elif kind == '#PROXY': | |||
| exposed, token = result | |||
| proxytype = self._manager._registry[token.typeid][-1] | |||
| proxytype = self._registry[token.typeid][-1] | |||
There was a problem hiding this comment.
For compatibility, it should fall back to self._manager._registry if registry was not explicitly specified. Either here, or in the constructor, I do not know what is better.
| @@ -3004,10 +3022,34 @@ def test_mymanager_context_prestarted(self): | |||
| self.common(manager) | |||
| self.assertEqual(manager._process.exitcode, 0) | |||
|
|
|||
| def test_proxy_can_access_registry_in_separate_process(self): | |||
| # Given | |||
There was a problem hiding this comment.
What do these Give/When/Then mean?
|
I have a simpler solution which I talks about in this blog post: https://zpz.github.io/blog/python-mp-manager-2/#bug-related-to-pickling However, this issue is gone in some further simplifications in mpservice.multiprocessing.server_process. I'm thinking of creating a new issue with my fixes, and am researching existing issues in prep. That's why I got here. |
When a proxy derived from
multiprocessing.managers.BaseProxyis passed between processes, theself._managerattribute becomesNone.self._manageris however accessed to retrieve its_registryattribute, which contain themethod_to_typeidmap recorded into the manager viaBaseManager.register(), causing anAttributeError. The solution is to decouple the role of the manager from the role of the registry and propagate the registry separately.