-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Broker considers fail-over consumer priority-level #2954
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Can you also clarify on the ConsumerBuilder
javadoc how the priority is being used?
yes, added. |
run java8 tests |
@rdhabalia One more question: how would this impact the assignment across multiple partitions? |
priority-level gets configured at consumer level so, each partitioned-consumer will present its priority-level to broker and broker chooses active consumer for each partition. Each partitioned will be a separate topic so, there will not be any behavior difference in case of multiple partitions. |
The question is that broker is using ordinal position for consumers to ensure even distribution. Simple example: 4 partitions, 2 consumers. With equal priorities, they
If I set c-1 with higher priority, I would expect to have c1 on all the partitions. however the priority will only affect the sorting, so each partition will see the same list of consumers |
that's true, I missed it. if we have C1 (priority=1), C2 (priority=0), C3 (priority=0) then C2,C3 consumers with highest priority should serve partitions evenly. let me fix it. |
@merlimat fixed it. |
rerun integration tests |
@rdhabalia can you rebase this issue to latest master? |
aaaf5db
to
4b46092
Compare
add java doc fix partitioned-topic distribution
4b46092
to
7a50e45
Compare
retest this please |
rerun java8 tests |
1 similar comment
rerun java8 tests |
These doc updates is for the PR: apache#2954 Docs for master is updated and merged. Update the following releases based on the master updates: 2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
…er mode (apache#7136) This doc is updated for the Pulsar 2.4.0 released PR: apache#2954. The doc is updated for the master right now. once it is reviewed and approved, it will be added to the previous releases.
…pache#7172) These doc updates is for the PR: apache#2954 Docs for master is updated and merged. Update the following releases based on the master updates: 2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
…er mode (apache#7136) This doc is updated for the Pulsar 2.4.0 released PR: apache#2954. The doc is updated for the master right now. once it is reviewed and approved, it will be added to the previous releases.
…pache#7172) These doc updates is for the PR: apache#2954 Docs for master is updated and merged. Update the following releases based on the master updates: 2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
Motivation
Right now, Broker selects failover consumer based on lexicographical sorting on consumer name. However, one of the user has usecase to switch fail-over consumer (based on hot/cold colo)in a deterministic way without changing/managing consumer-name. Consumer already has priority-level but it's not implemented in Failover consumer which can be useful in certain usecase.
Modifications
Broker also considers consumer's priority-level while selecting active consumer.