-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Configure uvicorn timeout_worker_healthcheck
#57731
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
d828d10 to
9aa71c0
Compare
|
@potiuk You confirmed this issue in september (#52270 (comment)), any chance of getting a review for this PR? |
|
Lgtm but maybe also @pierrejeambrun and @ashb can take a look - I mostly tangentially worked on API server and way how we integrate with unicorn - it does look promising though |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems roughly okay, but I'm not sure using the same value is right is all.
(#55707 should have addressed this for most people already)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I guess we can refine later or add a specific value if needed.
…r-timeout CLI option (#57731) - Set the minimum version for uvicorn to 0.37.0, so that this setting can be provided. - Add "timeout_worker_healthcheck" to the uvicorn_kwargs, and set it to the worker_timeout value that is already used to configure the "timeout_keep_alive" and "timeout_graceful_shutdown" settings. This increases the timeout value from 5 seconds to 120 seconds by default. This increase is the same as for the "timeout_keep_alive" setting. (cherry picked from commit e98dd9f) Co-authored-by: vlieven <verswyvel.lieven@gmail.com>
…r-timeout CLI option (#57731) (#57854) - Set the minimum version for uvicorn to 0.37.0, so that this setting can be provided. - Add "timeout_worker_healthcheck" to the uvicorn_kwargs, and set it to the worker_timeout value that is already used to configure the "timeout_keep_alive" and "timeout_graceful_shutdown" settings. This increases the timeout value from 5 seconds to 120 seconds by default. This increase is the same as for the "timeout_keep_alive" setting. (cherry picked from commit e98dd9f) Co-authored-by: vlieven <verswyvel.lieven@gmail.com>
…r-timeout CLI option (#57731) (#57854) - Set the minimum version for uvicorn to 0.37.0, so that this setting can be provided. - Add "timeout_worker_healthcheck" to the uvicorn_kwargs, and set it to the worker_timeout value that is already used to configure the "timeout_keep_alive" and "timeout_graceful_shutdown" settings. This increases the timeout value from 5 seconds to 120 seconds by default. This increase is the same as for the "timeout_keep_alive" setting. (cherry picked from commit e98dd9f) Co-authored-by: vlieven <verswyvel.lieven@gmail.com>
…I option (apache#57731) - Set the minimum version for uvicorn to 0.37.0, so that this setting can be provided. - Add "timeout_worker_healthcheck" to the uvicorn_kwargs, and set it to the worker_timeout value that is already used to configure the "timeout_keep_alive" and "timeout_graceful_shutdown" settings. This increases the timeout value from 5 seconds to 120 seconds by default. This increase is the same as for the "timeout_keep_alive" setting.
We noticed the child process death issue, also reported in #52270 on our kubernetes-based deployment. The issue was partly mitigated by #55707, but the root cause in uvicorn was not fixed at that time (related: Kludex/uvicorn#2397, stackabletech/airflow-operator#641).
Essentially, uvicorn had a healthcheck timeout that was set too low, and could not be configured.
Fortunately, as of release 0.37.0, uvicorn allows configuring this timeout value (PR: Kludex/uvicorn#2711).
This PR does two things:
worker_timeoutvalue that is already used to configure the "timeout_keep_alive" and "timeout_graceful_shutdown" settings. This increases the timeout value from 5 seconds to 120 seconds by default. This increase is the same as for the "timeout_keep_alive" setting.