[Enhancement] Optimize Tablet Report #54848
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gengjun-git
force-pushed
the
opt_tablet_report
branch
from
a0a9f14 to
00a3ee5
Compare
gengjun-git
force-pushed
the
opt_tablet_report
branch
from
6007573 to
b25d1a8
Compare
gengjun-git
force-pushed
the
opt_tablet_report
branch
2 times, most recently
from
3aa14d9 to
fdc42ae
Compare
HangyuanLiu
reviewed
Jan 16, 2025
| @@ -34,6 +34,7 @@ struct TMasterInfo { | |||
| 11: optional list<string> disabled_disks | |||
| 12: optional list<string> decommissioned_disks | |||
| 13: optional bool encrypted; | |||
| 14: optional bool stop_regular_tablet_report; | |||
There was a problem hiding this comment.
Add comment to explain what is flag mean, And which version can deprecate this flag
gengjun-git
force-pushed
the
opt_tablet_report
branch
from
fdc42ae to
cd578d0
Compare
gengjun-git
force-pushed
the
opt_tablet_report
branch
from
cd578d0 to
22888be
Compare
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 71 / 84 (84.52%) file detail
|
[BE Incremental Coverage Report]✅ pass : 28 / 30 (93.33%) file detail
|
HangyuanLiu
approved these changes
Jan 23, 2025
wangruin
approved these changes
Jan 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
In StarRocks, FE will periodically diff the tablets in BE and the tablets recorded in metadata, and then process the inconsistent tablets. The current implementation is that BE reports the full number of tablets to the FE Leader regularly (default 1 minute), and the Leader maintains a reporting queue, and then retrieves one BE tablet from the queue each time for single-threaded processing. For large-scale clusters, the speed of FE processing usually cannot keep up with the speed of BE reporting, resulting in the existence of all BE tablets in the reporting queue, which causes memory waste. This optimization uses the Leader's active pull mode to control the tablets in the reporting queue within a BE range.
What I'm doing:
After optimization, a new TabletController daemon is added to regularly pull the full number of tablets from the Backend. The pull condition is
BE still retains the ability to actively report tablets to FE Leader, but only for emergency situations, such as disk corruption and the need to immediately remove replicas from FE metadata.
Test(a cluster with 5 million tablets)
after optimization
before optimization
We can see the GC time has become smoother.
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: