Correctly import eventlet to prevent threads blocking each other #131
+20
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aieri
requested review from
a team,
Pjack,
aieri,
samuelallan72,
jneo8,
gabrielcocenza,
rgildein and
chanchiwai-ray
samuelallan72
requested changes
Jun 12, 2024
lathiat
force-pushed
the
fix_eventlet_blocking
branch
from
0e57e4b
to
88e8810
Compare
samuelallan72
approved these changes
Jun 12, 2024
Newer versions of pycodestyle started erroring on comparing type(obj), switch syntax to avoid the error.
Add additional debug-level logging for the progress of the statistics gathering, so we can see each time the refresh starts and finishes as well as how long each specific check took.
Currently, slow running OpenStack API Requests (either stuck connecting or still waiting for the actual response) from the periodic DataGatherer task will block HTTPServer connections from being processed. Blocked HTTPServer connections will also block both other connections and the DataGatherer task. Observed Symptoms: - Slow or failed prometheus requests - Statistics not being updated as often as you would expect - HTTP 500 responses and BrokenPipeError tracebacks being logged due to later trying to respond to prometheus clients which timed out and disconnected the socket - Hitting the forked process limit This happens because in the current code, we are intending to use the eventlet library for asynchronous non-blocking I/O, but we are not using it correctly. All code within the main application and all imported dependencies must import the special eventlet "green" versions of many python libraries (e.g. socket, time, threading, SimpleHTTPServer, etc) which yield to other green threads when they would have blocked waiting for I/O or to sleep. Currently this does not always happen. Fix this by importing eventlet and using eventlet.patcher.monkey_patch() before importing any other modules. This will automatically intercept all future imports (including those inside dependencies) and automatically load the green versions of relevant libraries. Documentation on correctly import eventlet can be found here: https://eventlet.readthedocs.io/en/latest/patching.html A detailed and comprehensive analysis of the issue and multiple previous attempts to fix it can be found in Issue canonical#130. If you intend to make further related changes to the use of eventlet, threads or forked processes please read the detailed history lesson available there. Fixes: canonical#130, canonical#126, canonical#124, canonical#116, canonical#115, canonical#112
lathiat
force-pushed
the
fix_eventlet_blocking
branch
from
88e8810
to
76500ab
Compare
|
Pushed new otherwise unchanged commits which are signed |
samuelallan72
approved these changes
Jun 12, 2024
jneo8
approved these changes
Jun 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, slow running OpenStack API Requests (either stuck connecting
or still waiting for the actual response) from the periodic DataGatherer
task will block HTTPServer connections from being processed. Blocked
HTTPServer connections will also block both other connections and the
DataGatherer task.
Observed Symptoms:
later trying to respond to prometheus clients which timed out and
disconnected the socket
This happens because in the current code, we are intending to use the
eventlet library for asynchronous non-blocking I/O, but we are not using
it correctly. All code within the main application and all imported
dependencies must import the special eventlet "green" versions of many
python libraries (e.g. socket, time, threading, SimpleHTTPServer, etc)
which yield to other green threads when they would have blocked waiting
for I/O or to sleep. Currently this does not always happen.
Fix this by importing eventlet and using eventlet.patcher.monkey_patch()
before importing any other modules. This will automatically intercept
all future imports (including those inside dependencies) and
automatically load the green versions of relevant libraries.
Documentation on correctly import eventlet can be found here:
https://eventlet.readthedocs.io/en/latest/patching.html
A detailed and comprehensive analysis of the issue and multiple previous
attempts to fix it can be found in Issue #130. If you intend to make
further related changes to the use of eventlet, threads or forked
processes please read the detailed history lesson available there.