-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RuntimeError due to a warning being logged on SIGHUP #2564
Comments
I am seeing the same thing with Traceback:
|
the same |
The final solution: see this issue: #2215 |
@onefor1 Are you sure you encountered the same issue? The issue you linked seems to be quite different and I'm not using kubernetes. It also seems to be quite strange, that one additional log.warning call is causing an OOM |
At least that's how I solved the problem. You can try |
I'm getting the same error. Increasing the memory for the pod did not help in my case. |
I am seeing this with near identical logs for |
I am seeing the same error after upgrading gunicorn to |
We are seeing this issue every now and then and would very much like it to get fixed. Usually starts with a lot of --- Logging error ---
Traceback (most recent call last):
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1029, in emit
self.flush()
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1009, in flush
self.stream.flush()
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
Call stack:
File "/opt/project/.venv/bin/gunicorn", line 8, in <module>
sys.exit(run())
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 211, in run
self.manage_workers()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 623, in spawn_workers
time.sleep(0.1 * random.random())
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 533, in reap_workers
os.WTERMSIG(status)
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/glogging.py", line 261, in warning
self.error_log.warning(msg, *args, **kwargs)
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1390, in warning
self._log(WARNING, msg, args, **kwargs)
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1514, in _log
self.handle(record)
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1524, in handle
self.callHandlers(record)
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1586, in callHandlers
hdlr.handle(record)
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 894, in handle
self.emit(record)
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1029, in emit
self.flush()
File "/opt/pyenv/versions/3.7.9/lib/python3.7/logging/__init__.py", line 1009, in flush
self.stream.flush()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 533, in reap_workers
os.WTERMSIG(status)
File "/opt/project/.venv/lib/python3.7/site-packages/gunicorn/glogging.py", line 261, in warning
self.error_log.warning(msg, *args, **kwargs)
Message: 'Worker with pid %s was terminated due to signal %s'
Arguments: (64247, 9) and at some point gunicorn will drop out with We are on Python 3.7.9, gunicorn 20.1.0, using ~50 workers. |
b695b49. has been reverted which shoudl indeed fix the error . More generally speaking we should find a way to handle the write of the errors to not allows concurrent writes imo. Please test master if you can and let me know if you still reproduce the issue. I am also interrested by a test case as I don't reproduced it myself yet. |
@benoitc Thank you for the swift reaction! Let me see if I can reproduce it somehow |
I fiddled around with it a bit, building on existing tests + threading, but was unsuccessful to reproduce the problem for now. I think it may be worthwile setting up a more realistic test environment, where the gunicorn process and its workers are run similar to production, without mocking out too many things. I have installed 76f8da2 to our server and will observe if we see the problem again, will set a reminder to get back here in a while 👋 |
I cannot find any errors in our logs like above with the "fixed" version, so I think this has solved the issue 👍 |
We are still seeing the same issue running our Django web app using gunicorn 20.1.0 via Azure.
The traceback goes on for a while after what I've pasted here with each error being a RuntimeError: reentrant call inside <_io.BufferedWriter name=''> As mentioned this does not occur every time but when the app is being redeployed or reloaded, it tends to happen fairly often. |
@sppaskal no, there has been no release yet (https://github.com/benoitc/gunicorn/tags). you can install directly from master though or even pinned to the exact commit if you need it quickly (https://stackoverflow.com/a/13754517). |
@cb109 Alright. Thank you for taking the time! |
the release will land tomorrow . |
My proposal is to log only if the situation is unexpected. After a WORKER TIMEOUT (worker.aborted=True), a SIGKILL is perfectly excpected (and already logged) and needs no further logging. |
En mi caso pude resolver ese problema aumentando mi memoria RAM de mi VPS de DIGITALOCEAN |
Original: b695b49 (benoitc#2475) Reverted: 76f8da2 The warnings are printed later in the main loop, not directly in the signal handler. This should fix the RuntimeError from benoitc#2564.
@benoitc any update on when the release will land? |
@benoitc I would love to get an update for when we can expect a new release with the fix for this issue. Thanks! |
For anyone recently dealing with this issue as i did (Python 3.9, Azure app service) The issue most of us are having is that the Gunicorn workers timeout during startup, usually on slower cloud containers as i experienced on Azure App Services. This is also why it can happen occasionally. Check your defaults and ensure that your Gunicorn timeout is set long enough to start the workers. In my case for fastapi/uvicorn increasing timeout to 600 using a custom app service startup script fixed it:
Doesn't solve the logging reentrant bug, but for anyone here waiting for a fix this is probably secondary to the real problem anyway. |
Same error - also on azure app service -
|
Is the reason machine memory allocation exceeded? |
Hi,
gunicorn might crash due to a warning being logged in case of a SIGHUP. I think this was introduced by this commit: b695b49
Here is the Traceback of the error:
This won't happen always.
The text was updated successfully, but these errors were encountered: