tools, media, readme: add DM shard merge #887
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM |
IANTHEREAL
reviewed
Feb 11, 2019
tools/dm/shard-merge.md
Outdated
| - For each sharding group, it is recommended to use one independent task to perform the replication. | ||
| - In a sharding group, the corresponding DDL statements should be executed in all upstream sharded tables. | ||
| - For example, if DDL statements are not executed on one or more upstream sharded tables corresponding to `DM-worker-2`, then other DM-workers that have executed the DDL statements will pause their synchronization task and wait for `DM-worker-2` to receive the upstream DDL statements. | ||
| - In a sharding group, the next DDL statement cannot be executed unless the current DDL operation is completely finished. |
There was a problem hiding this comment.
merge it with In a logical **sharding group** (composed of all sharded tables that need to be merged and replicated into one same downstream table), the same DDL statements must be executed in the same order in all upstream sharded tables (the schema name and the table name can be different) at L17?
IANTHEREAL
reviewed
Feb 11, 2019
tools/dm/shard-merge.md
Outdated
| - During the replication of sharding DDL statements, an error is reported if you use `dmctl` to change `router-rules`. | ||
| - If you need to `CREATE` a new table to a sharding group where DDL statements are being executed, you have to make sure that the table schema is the same as the newly modified table schema. | ||
| - For example, the original `table_1` and `table_2` has two columns (a, b) initially, and has three columns (a, b, c) after the sharding DDL operation, so after the replication the newly created table should also have three columns (a, b, c). | ||
| - Because the DM-worker that has received the DDL statements will pause the task to wait for other DM-workers to receive their DDL statements, the data replication will be increased. |
There was a problem hiding this comment.
what's meaning of the data replication will be increased.?
There was a problem hiding this comment.
maybe the delay of data replication?
IANTHEREAL
reviewed
Feb 11, 2019
| - A single `RENAME TABLE` statement can only involve a single `RENAME` operation. (Online DDL is supported in another solution) | ||
| - The table schema of each sharded table must be the same at the starting point of the incremental replication task, so as to make sure the DML statements of different sharded tables can be replicated into the downstream with a definite table schema, and the subsequent sharding DDL statements can be correctly matched and replicated. | ||
| - If you need to change the [table routing](../../tools/dm/data-synchronization-features.md#table-routing) rule, you have to wait for the replication of all sharding DDL statements to complete. | ||
| - During the replication of sharding DDL statements, an error is reported if you use `dmctl` to change `router-rules`. |
There was a problem hiding this comment.
do we have a test case for it? @amyangfei @csuzhangxc
There was a problem hiding this comment.
I think no test case exists, and we only add a check in DM.
There was a problem hiding this comment.
we should make sure it works
IANTHEREAL
reviewed
Feb 11, 2019
tools/dm/shard-merge.md
Outdated
| - In a sharding group, the next DDL statement cannot be executed unless the current DDL operation is completely finished. | ||
| - The sharding group replication task does not support `DROP DATABASE/DROP TABLE`. | ||
| - The Syncer unit in DM-worker automatically ignores the `DROP DATABASE/DROP TABLE` statement of upstream sharded tables. | ||
| - The sharding group replication task supports `RENAME TABLE`, but with the following limitations: |
There was a problem hiding this comment.
I think we should add (Online DDL is supported in another solution) here
|
Rest LGTM |
csuzhangxc
reviewed
Feb 11, 2019
| - [Troubleshoot](/tools/dm/troubleshooting-sharding-ddl-locks.md) | ||
| - [Introduction](tools/dm/shard-merge.md) | ||
| - [Restrictions](tools/dm/shard-merge.md#restrictions) | ||
| - [Handle Sharding DDL Locks manually](tools/dm/manually-handling-sharding-ddl-locks.md) |
csuzhangxc
reviewed
Feb 11, 2019
|
@GregoryIan @csuzhangxc Thanks! PTAL again |
csuzhangxc
reviewed
Feb 11, 2019
Co-Authored-By: lilin90 <lilin@pingcap.com>
|
LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the DM shard merge file, updates the TOC, adds the images that are ignored in #886.
Note: Since the file in #876 (created by @yikeke) has links that point to the file added in this PR, #876 can only be merged after this PR is merged.
@GregoryIan @csuzhangxc @CaitinChen @yikeke PTAL
Related Issue: #874
Related PRs: