docs/design: add the table partition proposal #7969
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
OK. I will review it later~ |
CaitinChen
reviewed
Oct 24, 2018
|
Comment addressed, PTAL, thanks @CaitinChen |
CaitinChen
reviewed
Nov 8, 2018
|
PTAL @CaitinChen |
|
/run-all-tests |
|
CI fail, blocked by #8287 |
|
/run-all-tests |
1 similar comment
|
/run-all-tests |
|
PTAL @shenli |
|
/run-all-tests |
|
PTAL @shenli |
shenli
reviewed
Nov 20, 2018
|
|
||
| ## Background | ||
|
|
||
| MySQL has the [table partition](https://dev.mysql.com/doc/refman/8.0/en/partitioning.html) feature. If this feature is supported in TiDB, many issues could be addressed. For example, drop ranged partitions could be used to remove old data; partition by hash could address the hot data issue and thus the write performance is improved; query on partitioned tables could be faster than on manual sharding tables because of the partition pruning. |
There was a problem hiding this comment.
query on partitioned tables could be faster than unpartitioned tables because of the partition pruning.
| There are two levels of mapping in TiDB: SQL data to a logical key range, logical key range to the physical storage. | ||
| When TiDB maps table data to key-value storage, it first encodes `table id + row id` as key, row data as value. Then the logical key range is split into regions and stored into TiKV. | ||
|
|
||
| Table partition works on the first level of mapping, partition ID will be made equivalent of the table ID. A partitioned table row uses `partition id + row id` as encoded key. |
There was a problem hiding this comment.
Please mention that PartitionID is unique in cluster scope.
| select * from p3 where id < 30) | ||
| ``` | ||
|
|
||
| During the logical optimization phase, the `DataSource` plan is translated into `UnionAll`, and then each partition generates its own `TableReader` in the physical optimization phase. |
There was a problem hiding this comment.
DataSource plan -> DataSource operator
UnionAll -> UnionAll operator
| The drawbacks of this implementation are: | ||
|
|
||
| * If the table has many partitions, there will be many readers, and then the `explain` result is not friendly to the user | ||
| * The `UnionAll` executor cannot keep the results in order, so if some executor needs ordered results such as `IndexReader`, an extra `Sort` executor is needed |
There was a problem hiding this comment.
We are talking about plan phrase, so there is no executor.
|
|
||
| ### How to write to the partitioned table | ||
|
|
||
| All the write operation calls functions like `table.AddRecord` eventually, so implementing the write operation on a partitioned table simply implements this interface method on the `PartitionedTable` struct. |
There was a problem hiding this comment.
singular? or plural？
All the write operation calls
tiancaiamao
force-pushed
the
proposal
branch
from
89b2117 to
91e26ef
Compare
|
Comment addressed @shenli |
|
LGTM |
shenli
approved these changes
Nov 20, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Add a table partition design proposal document #7907
@CaitinChen @shenli
This change is