Skip to content

Shutdown kombu consumer on worker shutdown #5338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 6, 2021

Conversation

khushboobhatia01
Copy link
Contributor

As of now, only the disptacher is shutdown in the shutdown method. This change takes care of shutting down Kombu consumer thread.

In HA environment this will ensure other healthy action runner pods to pick up requested actions instead of an unhealthy pod picking the action from queue and abandoning it.

In a non-HA environment, actions requested after worker has stopped will be in scheduled state and will be picked after new action runner comes up.

@pull-request-size pull-request-size bot added the size/XS PR that changes 0-9 lines. Quick fix/merge. label Aug 23, 2021
@khushboobhatia01 khushboobhatia01 changed the title Shutdown kombu consumer on shutdown Shutdown kombu consumer on worker shutdown Aug 23, 2021
Copy link
Member

@cognifloyd cognifloyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @khushboobhatia01, appreciate the contribution based on running st2 in HA prod!

Could you please also add a CHANGELOG.rst record for this fix?

@khushboobhatia01
Copy link
Contributor Author

khushboobhatia01 commented Aug 25, 2021

Thanks @khushboobhatia01, appreciate the contribution based on running st2 in HA prod!

Could you please also add a CHANGELOG.rst record for this fix?

@armab Yes, updated.

Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the Changelog! 👍

@Kami
Copy link
Member

Kami commented Aug 27, 2021

LGTM, but can you please also add a test case for it?

@pull-request-size pull-request-size bot added size/M PR that changes 30-99 lines. Good size to review. and removed size/XS PR that changes 0-9 lines. Quick fix/merge. labels Sep 5, 2021
@khushboobhatia01
Copy link
Contributor Author

LGTM, but can you please also add a test case for it?

@Kami Added test case. Can you please merge? Thank you! :)

Copy link
Member

@cognifloyd cognifloyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An excellent test. It feels more like an integration test though. Could you move the new test to a new file under st2actions/tests/integration?

Thanks!

@CLAassistant
Copy link

CLAassistant commented Sep 6, 2021

CLA assistant check
All committers have signed the CLA.

@khushboobhatia01
Copy link
Contributor Author

An excellent test. It feels more like an integration test though. Could you move the new test to a new file under st2actions/tests/integration?

Thanks!

Done!

@cognifloyd cognifloyd enabled auto-merge September 6, 2021 06:28
@cognifloyd cognifloyd merged commit d6969a4 into StackStorm:master Sep 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug fix rabbitmq service: action runner size/M PR that changes 30-99 lines. Good size to review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants