Make sure temporary stream output queues get deleted, make all queues used by the stream service auto deletable #3746
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This way they are deleted after no one is consuming from them anymore.
This way they get automatically deleted when stream is shut down. We use random names for those queues so if we don't do that, they will stay around when stream is restart (new queues get created on restart) and nothing will consume from them hence we are left with zombie queues.
Kami
added a commit
that referenced
this pull request
Sep 15, 2017
Kami
added a commit
that referenced
this pull request
Sep 19, 2017
…e_delete Update changelog for #3746
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The issue with execution output queues used by stream service never getting deleted was caught by deploying v2.5dev on CI server and by @armab, @enykeev.
The issue was that the queue wasn't exclusive.
This pull request fixes that and on top of that, it also makes all queues used by the stream service auto deletable.
We use random queue names for queues used by the stream service, but we don't have
auto_delete=True. This means that when stream service restart, existing queue will persist, but we will create a new queue on start which will be used by the consumers. This means old queue will be left around, but nothing will ever consume from it again.Also, as far as I know, we basically want (almost) always use
exclusive=Trueandauto_delete=Truefor queues with random names. If we don't do that, we will have zombie queues laying around.Somewhat related to #3640 - I also recommended using auto_delete=True there.
This also brings up an issue with testing for things like that - we need to test for CPU usage, memory usage, number of queues, etc. over time. This means we should invest some time into a framework so we can better test for issues like that.