Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close #31913

Merged
merged 4 commits into from
May 3, 2022
Merged

bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close #31913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d227300
Fix a BrokenPipeError when a multiprocessing.Queue is garbage collected
geryogam Mar 15, 2022
9950aaa
Close self._reader from the queue thread instead of the main thread
geryogam Apr 26, 2022
cf7d8ec
Update queues.py
geryogam Apr 26, 2022
d8e8957
Create 2022-04-26-19-01-13.bpo-47029.qkT42X.rst
geryogam Apr 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions Lib/multiprocessing/queues.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,13 +139,10 @@ def put_nowait(self, obj):

def close(self):
self._closed = True
try:
self._reader.close()
finally:
close = self._close
if close:
self._close = None
close()
close = self._close
if close:
self._close = None
close()

def join_thread(self):
debug('Queue.join_thread()')
Expand All @@ -169,8 +166,9 @@ def _start_thread(self):
self._thread = threading.Thread(
target=Queue._feed,
args=(self._buffer, self._notempty, self._send_bytes,
self._wlock, self._writer.close, self._ignore_epipe,
self._on_queue_feeder_error, self._sem),
self._wlock, self._reader.close, self._writer.close,
self._ignore_epipe, self._on_queue_feeder_error,
self._sem),
name='QueueFeederThread'
)
self._thread.daemon = True
Expand Down Expand Up @@ -211,8 +209,8 @@ def _finalize_close(buffer, notempty):
notempty.notify()

@staticmethod
def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
onerror, queue_sem):
def _feed(buffer, notempty, send_bytes, writelock, reader_close,
writer_close, ignore_epipe, onerror, queue_sem):
debug('starting thread to feed data to pipe')
nacquire = notempty.acquire
nrelease = notempty.release
Expand All @@ -238,7 +236,8 @@ def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
obj = bpopleft()
if obj is sentinel:
debug('feeder thread got sentinel -- exiting')
close()
reader_close()
writer_close()
return

# serialize the data before acquiring the lock
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Library/2022-04-26-19-01-13.bpo-47029.qkT42X.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Always close the read end of the pipe used by :class:`multiprocessing.Queue`
*after* the last write of buffered data to the write end of the pipe to avoid
:exc:`BrokenPipeError` at garbage collection and at
:meth:`multiprocessing.Queue.close` calls. Patch by Géry Ogam.