Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added log when a decrease in partition count has been detected #683

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

amdeel
Copy link
Contributor

@amdeel amdeel commented Feb 25, 2022

When customers decrease their partition count without switching to a new task hub or removing their unused storage resources old blobs associated with partitions will cause partition load balancing to be inefficient.

This PR adds a partition error log message when this case is detected.

@amdeel amdeel requested a review from cgillum February 25, 2022 18:50
@cgillum
Copy link
Collaborator

cgillum commented Feb 25, 2022

If the partition count is decreased, do we still poll all the queues (including the ones which we technically shouldn't be)? I'm not familiar with the expected behavior in this scenario. I thought we'd start polling only a subset of the queues (which could result in bigger problems) but that's different from the behavior you're describing and my own knowledge may be outdated (or wrong).

@amdeel
Copy link
Contributor Author

amdeel commented Feb 25, 2022

@cgillum AzureStorageOrchestrationService only adds messages to ControlQueues based on the partition count. So these control queues will get all of the messages while we will still balance partitions based on workers according to the old partition count.

@cgillum
Copy link
Collaborator

cgillum commented Feb 25, 2022

Okay, so clients will correctly load balance to the smaller number of queues but workers will still poll the larger number of queues? I was trying to understand if orchestrations could get stuck after reducing the partition count, but from what you’re saying, it sounds like that’s not the case. (If it was, I was thinking we’d want to add that information to the error message)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants