-
-
Notifications
You must be signed in to change notification settings - Fork 758
Shutdown kombu consumer on worker shutdown #5338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shutdown kombu consumer on worker shutdown #5338
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice.
For other reviewers. Here are the docs for should_stop
:
https://docs.celeryproject.org/projects/kombu/en/stable/reference/kombu.mixins.html#kombu.mixins.ConsumerMixin.should_stop
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @khushboobhatia01, appreciate the contribution based on running st2 in HA prod!
Could you please also add a CHANGELOG.rst
record for this fix?
@armab Yes, updated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the Changelog! 👍
LGTM, but can you please also add a test case for it? |
@Kami Added test case. Can you please merge? Thank you! :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
An excellent test. It feels more like an integration test though. Could you move the new test to a new file under st2actions/tests/integration
?
Thanks!
Done! |
As of now, only the disptacher is shutdown in the shutdown method. This change takes care of shutting down Kombu consumer thread.
In HA environment this will ensure other healthy action runner pods to pick up requested actions instead of an unhealthy pod picking the action from queue and abandoning it.
In a non-HA environment, actions requested after worker has stopped will be in scheduled state and will be picked after new action runner comes up.