fix: Unregister semaphores from stdlib tracker immediately after creation #474
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
On Python 3.13+, loky users see harmless but annoying semaphore leak warnings at process exit:
Root Cause
In Python 3.13+,
_multiprocessing.SemLockautomatically registers semaphores with Python's stdlibmultiprocessing.resource_tracker. Loky also registers these semaphores with its own resource tracker. During cleanup:The semaphore isn't actually leaked (it's properly cleaned up), but the double-registration causes false warnings.
Solution
This PR implements a two-pronged fix:
1. Library-Level Fix (Commit 93d882b)
File:
loky/backend/synchronize.pyImmediately after each
_SemLock()creation, unregister from stdlib's resource tracker:This is done in two locations:
name=None, before break at line 93)nameis provided)2. Application-Level Cleanup (Documented)
For defense-in-depth, applications can add an atexit handler to forcefully shutdown loky executors:
Testing
Before Fix
$ python test_loky.py # ... training completes ... /python3.13/multiprocessing/resource_tracker.py:324: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects...After Fix
Tested with:
Compatibility
Impact
Related Issues
This addresses the same underlying issue as other multiprocessing libraries experiencing Python 3.13 compatibility problems with resource tracking.