Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling NO Coordinator Scenario and Data Loss in the current Design #11298

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

kumarpritam863
Copy link

@kumarpritam863 kumarpritam863 commented Oct 10, 2024

This PR combines the issues I have already handled by these two PRs:

  1. No Coordinator Scenario in ICR Mode -> Workers gets stuck as there is no-coordinator for emitting Start_Commit request in Incremental Cooperative Rebalancing[ICR] Mode #11288
  2. Data Loss in ICR Mode -> Data loss in the Incremental Co-operative Mode of Rebalancing #11289

With this PR, I have made few changes to handle both the issues in more robust, resilient and performant ways.
I have another design change proposal with this PR which handles much more scenarios without doing any OffsetSync or other efforts in much more resilient ways.

The explanantion and the pictorial view of the issues ca be found in this Design change proposal doc which is proposing the changes made by this PR

@kumarpritam863
Copy link
Author

kumarpritam863 commented Oct 10, 2024

@bryanck Sir, can we please review this. This PR handles the most of the ICR issues in the current design itself with slight modifications. We can discuss more on this PR which proposes more robust and resilient and connect friendly design.

@kumarpritam863
Copy link
Author

@bryanck Sir can we please review this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant