multiprocessing.Queue.put can fail silently due to pickle errors

[SanderLand]

BPO	40195
Files	minimal_repr.py: Minimal example for reproducing the behaviour

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-05.10:55:01.927>
labels = ['3.7', 'type-bug', 'library']
title = 'multiprocessing.Queue.put can fail silently due to pickle errors'
updated_at = <Date 2020-04-05.10:55:01.927>
user = 'https://bugs.python.org/SanderLand'

bugs.python.org fields:

activity = <Date 2020-04-05.10:55:01.927>
actor = 'Sander Land'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-04-05.10:55:01.927>
creator = 'Sander Land'
dependencies = []
files = ['49036']
hgrepos = []
issue_num = 40195
keywords = []
message_count = 1.0
messages = ['365816']
nosy_count = 1.0
nosy_names = ['Sander Land']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40195'
versions = ['Python 3.7']

[SanderLand]

The multiprocessing Queue uses a thread to pickle and send the object after a call to put. When pickling fails (e.g. due to recursion depth) the exception is not returned to the caller to .put but instead dumped on the screen, leaving any multiprocessing pools in a very unhappy state.

Suggested fix: pickle the object in the same thread as the caller and send the pickled object to the thread, ensuring the caller to .put can catch the exception.

Sad workaround: (un)pickle anything sent via this queue yourself.

[iantra]

Recently ran into this and it was quite annoying to debug, so just putting in my 2 cents and a vote to solve this 👍

From looking at the tests it seems like not raising anything on failed serialization is the intended behavior, to make sure the queue still works afterwards. All exceptions in the feeder thread are simply printed as you can see here and here

I think keeping the background thread running makes sense, but an exception should probably be raised in the main thread so that people can actually deal with a pickling error in a nice way (errors should never pass silently and all that.) This is actually how it's done in a Pipe, so it should be relatively simple to get right, but it would be a breaking change that might actually affect a few workflows. One way to make it backward compatible would be to add a flag (e.g. raise_pickling_errors) that you can turn on/off when making the Queue, although personally I think it should just be the default since people can ignore exceptions on their own, and it would then be more consistent with the behavior of pipes.

Also adding a minimal example here for clarity:

import multiprocessing
q = multiprocessing.Queue()
try:
    q.put(lambda: None)
    print("Success!")
except Exception:
    print("Failure")

And the (very confusing) output if you run this:

Success!
>>> Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7fbe76711090>: attribute lookup <lambda> on __main__ failed

I'd be happy to open a PR if others consider it a worthwhile change.

[serhiy-storchaka]

See also #55095. It was closed as superseded by #61227, but the latter did not change anything with this issue.

[ekneg54]

I fall into that and yes it would be nice to raise somehow in the main thread or to add a note in the documentation about this behavior: https://docs.python.org/3.11/library/multiprocessing.html#multiprocessing.Queue.put