update sync_diff_inspector: sync_diff_inspector v2.0 #6774

Liuxiaozhen12 · 2021-11-10T08:50:28Z

First-time contributors' checklist

I've signed Contributor License Agreement that's required for repo owners to accept my contribution.

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions.

What is the related PR or file link(s)?

This PR is translated from: update sync_diff_inspector: sync_diff_inspector v2.0 docs-cn#7353 & update sync-diff-inspector: update settings docs docs-cn#7510
Other reference link(s):

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

ti-chi-bot · 2021-11-10T08:50:30Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

hfxsd

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Liuxiaozhen12 · 2021-11-11T03:01:19Z

/verify

hfxsd · 2021-11-11T05:50:28Z

sync-diff-inspector/dm-diff.md

+[task]
+    output-dir = "./output"
+
+    # The tables of downstream databases to be compared. Each table needs to contain schema name and table name, separated by '.'


Suggested change

# The tables of downstream databases to be compared. Each table needs to contain schema name and table name, separated by '.'

# The tables of downstream databases to be compared. Each table needs to contain the schema name and the table name, separated by '.'

hfxsd · 2021-11-11T06:19:48Z

sync-diff-inspector/route-diff.md

+    password = ""
+########################### Routes ###########################
+[routes.rule1]
+schema-pattern = "test_1"      # # Matches the schema name of the data source. Supports the wildcards "*" and "?"


Suggested change

schema-pattern = "test_1" # # Matches the schema name of the data source. Supports the wildcards "*" and "?"

schema-pattern = "test_1" # Matches the schema name of the data source. Supports the wildcards "*" and "?"

hfxsd · 2021-11-11T06:29:57Z

sync-diff-inspector/route-diff.md

+
+## Note
+
+If `t_2` exists in the upstream database, the downstream databse also compares this table.


这句话不太明白。意思是说会检查上游 test_1中所有的表的数据，如果恰好有个表也叫 t_2，和target-table重名，那么也会检查。但是这么说明似乎就有点儿多余了，* 就是匹配全部表，所以即使重名当然也会检查。可以考虑改成下面说法：
If there is a table with the same name as the target table ( t_2 in the above case) existing in the upstream database, the downstream databse also compares this table.

我理解的也是这个意思。@Leavrth PTAL

这里少了，应该是如果上游数据库存在test_2.t_2，也会检查

hfxsd · 2021-11-11T06:38:25Z

sync-diff-inspector/shard-diff.md


-You can use `table-config` to configure `table-0`, set `is-sharding=true` and configure the upstream table information in `table-config.source-tables`. This configuration method requires setting all sharded tables, which is suitable for scenarios where the number of upstream sharded tables is small and the naming rules of sharded tables do not have a pattern as shown below.
+You can use `Datasource config` to configure `table-0`, set corresponding `rules` and configure the tables that have the mapping relationship between the upstream and downstream databases. This configuration method requires setting all sharded tables, which is suitable for scenarios where the number of upstream sharded tables is small and the naming rules of sharded tables do not have a pattern as shown below.


需要加几句话解释一下为啥图中的关系不能使用route rules，没看明白。另外，建议再画一个可以使用route rules的图例。有对比，更便于帮助用户理解。

hfxsd · 2021-11-11T06:40:39Z

sync-diff-inspector/shard-diff.md

-    user = "root"
-    password = "123456"
-    instance-id = "target-1"
+    # The tables of downstream databases to be compared. Each table needs to contain schema name and table name, separated by '.'


Suggested change

# The tables of downstream databases to be compared. Each table needs to contain schema name and table name, separated by '.'

# The tables of downstream databases to be compared. Each table needs to contain the schema name and the table name, separated by '.'

sync-diff-inspector/sync-diff-inspector-overview.md

hfxsd · 2021-11-11T07:22:55Z

sync-diff-inspector/sync-diff-inspector-overview.md


 - sync-diff-inspector consumes a certain amount of server resources when checking data. Avoid using sync-diff-inspector to check data during peak business hours.
 - TiDB uses the `utf8_bin` collation. If you need to compare the data in MySQL with that in TiDB, pay attention to the collation configuration of MySQL tables. If the primary key or unique key is the `varchar` type and the collation configuration in MySQL differs from that in TiDB, then the final check result might be incorrect because of the collation issue. You need to add collation to the sync-diff-inspector configuration file.
 - sync-diff-inspector divides data into chunks first according to TiDB statistics and you need to guarantee the accuracy of the statistics. You can manually run the `analyze table {table_name}` command when the TiDB server's *workload is light*.
- Pay special attention to `table-rules`. If you configure `schema-pattern="test1"` and `target-schema="test2"`, the `test1` schema in the source database and the `test2` schema in the target database are compared. If the source database has a `test2` schema, this `test2` schema is also compared with the `test2` schema in the target database.
- The generated `fix.sql` is only used as a reference for repairing data, and you need to confirm it before executing these SQL statements to repair data.
+- Pay special attention to `table-rules`. If you configure `schema-pattern="test1"`, `table-pattern = "t_1"`, `target-schema="test2"` and `target-table = "t_2"`, the `test1`.`t_1` schema in the source database and the `test2`.`t_2` schema in the target database are compared. Sharding is enabled by default in sync-diff-inspector, so if the source database has a `test2`.`t_2` table,  the `test1`.`t_1` table and `test2`.`t_2` table in the source database serving as sharding are compared with the `test2`.`t_2` table in the target database.


出现这种问题会对用户有什么影响呢？如果有风险，可以加个说明明确指出。

@Leavrth PTAL~

一般来说，如果没有rules，那么上游数据库会匹配下游数据库相同名称的表，相当于每个表都有个默认的自己匹配自己的rule。但是加上rules并不会去掉这个默认的匹配规则，也就是如上面说的，如果用户想要做 test1.t_1 -> test2.t_2 的匹配，但是默认还存在 test2.t_2 -> test2.t_2 的匹配，那么上游存不存在 test2.t_2 的结果是不一样的。

hfxsd

LGTM

Co-authored-by: xixirangrang <hfxsd@hotmail.com>

…iff22

sync-diff-inspector/sync-diff-inspector-overview.md

Leavrth · 2021-11-12T06:13:47Z

sync-diff-inspector/sync-diff-inspector-overview.md

 - `Databases config`: Configures the instances of the upstream and downstream databases.
+- `Tables config`: Special configurations for specific tables, including specified ranges, columns to be ignored and so on (optional).
+- `Routes`: Rules for upstream multiple schema names to regularly match downstream single schema names (optional).


regularly match。直接使用 match 会不会好些

Leavrth · 2021-11-12T06:22:47Z

sync-diff-inspector/sync-diff-inspector-overview.md


 Below is the description of a complete configuration file:

 ``` toml
 # Diff Configuration.

 ######################### Global config #########################
+# The number of goroutines created to check data. The number of connections between upstream and downstream databases are slightly greater than this value.


原文是上下游数据库的连接数会略大于该值，意思是 sync-diff-inspector 与上游数据库的连接数/下游数据库的连接数都会略大于这个值

Leavrth · 2021-11-12T06:26:50Z

sync-diff-inspector/sync-diff-inspector-overview.md

-check-thread-count = 4
+######################### Datasource config #########################
+[data-sources]
+[data-sources.mysql1] # mysql1 is the only ID for the database instance. It is used in the following `task.source-instances/task.target-instance` files.


task.source-instances/task.target-instance。

应该不是 files，这两个是下面 task config 配置里面的 source-instances 和 target-instance 两项。

Leavrth · 2021-11-12T07:08:28Z

sync-diff-inspector/route-diff.md

+
+## Note
+
+If `t_2` exists in the upstream database, the downstream databse also compares this table.


这里少了，应该是如果上游数据库存在test_2.t_2，也会检查

sync-diff-inspector/route-diff.md

sync-diff-inspector/sync-diff-inspector-overview.md

sync-diff-inspector/shard-diff.md

sync-diff-inspector/route-diff.md

ti-chi-bot · 2021-11-16T06:46:25Z

@Leavrth: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Liuxiaozhen12 · 2021-11-16T06:48:47Z

/remove-status LGT1
/status LGT2

Liuxiaozhen12 · 2021-11-16T06:49:14Z

/merge

ti-chi-bot · 2021-11-16T06:49:17Z

This pull request has been accepted and is ready to merge.

Commit hash: 06ee509

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-22T09:14:19Z

In response to a cherrypick label: new pull request created: #6900.

update sync_diff_inspector: sync_diff_inspector v2.0

0e3a0fb

Liuxiaozhen12 added translation/from-docs-cn This PR is translated from a PR in pingcap/docs-cn. area/migrate Indicates that the Issue or PR belongs to the area of TiDB migration tools. v5.3 This PR/issue applies to TiDB v5.3. labels Nov 10, 2021

Liuxiaozhen12 requested review from 3pointer, zier-one, hfxsd and Leavrth November 10, 2021 08:50

ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 10, 2021

update

3ad2073

hfxsd reviewed Nov 11, 2021

View reviewed changes

hfxsd approved these changes Nov 11, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 11, 2021

Liuxiaozhen12 and others added 3 commits November 12, 2021 14:00

Apply suggestions from code review

0713596

Co-authored-by: xixirangrang <hfxsd@hotmail.com>

address comments

8dbd338

Merge branch 'diff22' of https://github.com/Liuxiaozhen12/docs into d…

6609cec

…iff22

Liuxiaozhen12 commented Nov 12, 2021

View reviewed changes

sync-diff-inspector/sync-diff-inspector-overview.md Outdated Show resolved Hide resolved

Update sync-diff-inspector/sync-diff-inspector-overview.md

057e2bf

Leavrth reviewed Nov 12, 2021

View reviewed changes

Liuxiaozhen12 commented Nov 15, 2021

View reviewed changes

Apply suggestions from code review

ce4d873

Liuxiaozhen12 commented Nov 15, 2021

View reviewed changes

sync-diff-inspector/shard-diff.md Outdated Show resolved Hide resolved

Liuxiaozhen12 commented Nov 15, 2021

View reviewed changes

sync-diff-inspector/route-diff.md Outdated Show resolved Hide resolved

Liuxiaozhen12 and others added 3 commits November 15, 2021 13:50

Apply suggestions from code review

909e159

fix a typo in the picture

bb01939

new translation based on pingcap/docs-cn#7510

06ee509

Leavrth approved these changes Nov 16, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 16, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 16, 2021

ti-chi-bot merged commit fb5cc29 into pingcap:master Nov 16, 2021

Liuxiaozhen12 added the needs-cherry-pick-release-5.3 label Nov 22, 2021

Liuxiaozhen12 deleted the diff22 branch November 22, 2021 09:14

ti-chi-bot pushed a commit to ti-chi-bot/docs that referenced this pull request Nov 22, 2021

This is an automated cherry-pick of pingcap#6774

dc3c771

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 22, 2021

update sync_diff_inspector: sync_diff_inspector v2.0 (#6774) #6900

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update sync_diff_inspector: sync_diff_inspector v2.0 #6774

update sync_diff_inspector: sync_diff_inspector v2.0 #6774

Liuxiaozhen12 commented Nov 10, 2021 •

edited

Loading

ti-chi-bot commented Nov 10, 2021 •

edited

Loading

Liuxiaozhen12 commented Nov 11, 2021

hfxsd Nov 11, 2021

hfxsd Nov 11, 2021

hfxsd Nov 11, 2021

Liuxiaozhen12 Nov 12, 2021

Leavrth Nov 12, 2021

hfxsd Nov 11, 2021

hfxsd Nov 11, 2021

hfxsd Nov 11, 2021

Liuxiaozhen12 Nov 12, 2021

Leavrth Nov 12, 2021

hfxsd left a comment

Leavrth Nov 12, 2021

Leavrth Nov 12, 2021

Leavrth Nov 12, 2021

Leavrth Nov 12, 2021

ti-chi-bot commented Nov 16, 2021

Liuxiaozhen12 commented Nov 16, 2021

Liuxiaozhen12 commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 22, 2021

	# The tables of downstream databases to be compared. Each table needs to contain schema name and table name, separated by '.'
	# The tables of downstream databases to be compared. Each table needs to contain the schema name and the table name, separated by '.'

	schema-pattern = "test_1" # # Matches the schema name of the data source. Supports the wildcards "*" and "?"
	schema-pattern = "test_1" # Matches the schema name of the data source. Supports the wildcards "*" and "?"


		## Note

		If `t_2` exists in the upstream database, the downstream databse also compares this table.


		You can use `table-config` to configure `table-0`, set `is-sharding=true` and configure the upstream table information in `table-config.source-tables`. This configuration method requires setting all sharded tables, which is suitable for scenarios where the number of upstream sharded tables is small and the naming rules of sharded tables do not have a pattern as shown below.
		You can use `Datasource config` to configure `table-0`, set corresponding `rules` and configure the tables that have the mapping relationship between the upstream and downstream databases. This configuration method requires setting all sharded tables, which is suitable for scenarios where the number of upstream sharded tables is small and the naming rules of sharded tables do not have a pattern as shown below.

update sync_diff_inspector: sync_diff_inspector v2.0 #6774

update sync_diff_inspector: sync_diff_inspector v2.0 #6774

Conversation

Liuxiaozhen12 commented Nov 10, 2021 • edited Loading

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

ti-chi-bot commented Nov 10, 2021 • edited Loading

Liuxiaozhen12 commented Nov 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hfxsd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot commented Nov 16, 2021

Liuxiaozhen12 commented Nov 16, 2021

Liuxiaozhen12 commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 22, 2021

Liuxiaozhen12 commented Nov 10, 2021 •

edited

Loading

ti-chi-bot commented Nov 10, 2021 •

edited

Loading