Multiple bugs in multiprocessing.managers
: pickling, nesting, etc #119937
Open
Description
Bug report
Bug description:
I noticed multiple bugs in multiprocessing.mangers
and explained them, with fixes, in a blog post: https://zpz.github.io/blog/python-mp-manager-2/
A summary of the bugs are as follows:
- Pickling of
BaseProxy
objects: easily causes object release before unpickling happens in another process - Unable to call proxy-returning methods on a proxy that has been passed to another process. (There is an open PR gh-99595: Fix bug in multiprocessing.managers.BaseProxy that causes AttributeError when passing a proxy between processes #99598 but my solution appears simpler and cleaner.)
- "Nested proxies" are never released (also reported in this isseu Memory leak in multiprocessing.managers.Server #103136)
- A simple enhancement (not exactly a bug): server-side exceptions lose all traceback info when propagated to user (also reported here in this issue multiprocessing.Server swallows original exception traceback #78279)
In addition, this issue
- The specification of
exposed
is a mess. (See https://zpz.github.io/blog/python-mp-manager-2/#exposed-methods)
My package mpservice.multiprocessing.server_process fixes these issues with some additional enhancements and simplifications. The enhancements are covered in another blog post: https://zpz.github.io/blog/python-mp-manager-3/
I hope to engage in some discussions (esp with a core developer) before working on some PR, either local fixes or something a little more holistic.
CPython versions tested on:
3.10, 3.13
Operating systems tested on:
Linux
Metadata
Assignees
Type
Projects
Status
No status