feat: support random partition rule #1193
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ShiKaiWi
force-pushed
the
feat-support-partition-no-rule
branch
2 times, most recently
from
080b7f4
to
c702375
Compare
Rachelint
reviewed
Sep 7, 2023
ShiKaiWi
force-pushed
the
feat-support-partition-no-rule
branch
from
55cc18a
to
3c05f0e
Compare
ShiKaiWi
force-pushed
the
feat-support-partition-no-rule
branch
from
3c05f0e
to
36e48d4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale
Currently, hash/key rule are the only supported rules for partition table. However, it will lead to small write batches if the user tries to use primary key or all tags as the partition key for a uniform distribution. And it is known that the small write batches hurt write performance badly.
Actually, for a partition table in
appendmode with primary key or all tags as partition key, most of the queries to this table will fan out to all the partitions because the partition key won't be provided in most cases. So what is the gain from the cost (the massive small write batches) to do such complex partition?It will work much better if a simple partition rule is supported, by which a random partition is chosen for a given write request.
Detailed Changes
RandomPartitionRulevalidate_create_tabletoTableEnginetraitTest Plan
Add unit test to apply
RandomPartitionRulefor a partition table in different mode.