Force commit consuming segments (#9197)#9315
Merged
xiangfu0 merged 1 commit intoapache:release-0.11-rcfrom Sep 1, 2022
Merged
Conversation
walterddr
added
the
release-notes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this is a cherry-pick of #9197
Description
For realtime segments, if there's any changes to stream configs in the table config, the changes don't get picked up immediately in the existing consuming segments. After the existing consuming segments complete - which may take hours depending on flush thresholds defined in table config - the new consuming segments will pick up the stream config changes.
In this PR, the force commit functionality that was introduced in pause/resume feature (#8986) is used in a controller endpoint to reset the consumption of a realtime table. After "consumption reset" is issued, the current consuming segments will be forced to commit immediately and then the new consuming segments will pick up any changes in the table config.
Testing Done
Verified the expected behavior using
LLCRealtimeClusterIntegrationTest. During ingestion, I added a consumption rate limit parameter to the table config and then usedresetConsumptionendpoint and verified that the existing consuming segments completed and then the new consuming segments consumed according to the specified consumption rate limit.