dm: checkpoint mechanism and common QA #8749

buchuitoudegou · 2022-03-24T05:34:33Z

First-time contributors' checklist

I've signed Contributor License Agreement that's required for repo owners to accept my contribution.

What is changed, added or deleted? (Required)

introduction of checkpoint mechanism and some common QA

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

ti-chi-bot · 2022-03-24T05:34:34Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

CLAassistant · 2022-03-24T05:34:37Z

All committers have signed the CLA.

dm/dm-checkpoint.md

sunzhaoyang · 2022-03-28T06:08:55Z

/cc @lance6716

dm/dm-checkpoint.md

Co-authored-by: sunzy <sunzy2@gmail.com>

lance6716

rest lgtm

dm/dm-checkpoint.md

lance6716 · 2022-03-28T06:21:23Z

dm/dm-checkpoint.md

+
+## Q&A
+
+### 为什么根据配置文件启动，实际的 position 地址却没有根据配置文件定义的 position ？


如果是这个 position 疑问的话，可能不会找到 checkpoint 页面，可以放在外面的 FAQ

这里留着吧。其他地方我也提一下。如果是用户在刚开始学习的时候就能直接知晓这个问题了。

Co-authored-by: lance6716 <lance6716@gmail.com>

sunzhaoyang

LGTM

dm/dm-checkpoint.md

ran-huang · 2022-04-19T04:01:09Z

dm/dm-checkpoint.md

+- table checkpoints：每个表同步的 binlog 位置以及此时的表结构（以 TiDB TableInfo JSON 表示）。
+- safe mode exit point：如果不为空，小于这个 checkpoint 的 binlog 有可能在同步队列中，但大于它的则一定不在。在任务从错误恢复重启时，它可以清楚地标识出可能已经同步过的 binlog，因此 (table checkpoint, safe mode exit point] 之间的同步作业必须开启 safe mode，**以免出现重复处理而导致的下游报错**。当重启之后，binlog position 超过 safe mode exit point 之后，safe mode 关闭。
+
+由它们的语义我们可以知道，这三个信息的顺序是：global checkpoint ≤ table checkpoints ≤ safe mode exit point。


能否具体说明是什么的顺序？比如 DM 读取 checkpoint 的顺序？优先级顺序？

cc @dsdashun

指的是 binlog position 的前后顺序。

我觉得然然应该还是没懂。你的意思是，这三个信息中，global checkpoint 保存的 binlog 位置必然小于等于 table checkpoints 对吗？

对的，可能说“xxx位置小于yyy“有点奇怪。。详细点说就是：

global checkpoint: binlog-0001 table checkpoints: [binlog-0003, binlog-0004], # 每个表有一个 checkpoint safe mode exit point: binlog-0007

dm/dm-checkpoint.md

ran-huang · 2022-04-19T05:42:37Z

dm/dm-checkpoint.md

+1. 当 XID event 或者 rows event 被处理完，会调用 checkShouldFlush 函数，如果超过一定 interval 都还没有 flush 过，会尝试将 checkpoint 写入到下游。
+2. 处理完 DDL event 会尝试将内存 checkpoint 写入到下游。
+3. 在尝试写入 checkpoint 时，会先判断 global checkpoint 和 table checkpoints 的值是否有改变，没有改变会不写。
+4. safe mode exit point：只在 syncer 退出时，等待同步队列写入下游后 flush。


这一句能否再明确一下，flush 的是什么？

就是刷新内存中的 checkpoint 信息到持久化存储

flush 还是一个比较常用的词汇，应该不用特别解释了。

dm/dm-checkpoint.md

ran-huang · 2022-04-20T03:17:38Z

dm/dm-checkpoint.md

+
+checkpoint 一般存放在下游数据库中。如果配置文件没有定义 meta 信息的库名，则默认在 dm_meta 库中。不同的任务的同步进度存放在以任务名为表名的表中。可以通过清理这些表内容的方式来清理 checkpoint（不建议），或者在下次启动任务时新增 `--remove-meta` flag 来清理已存在的 checkpoint。
+
+### async checkpoint 的机制？


正文里没有提到同步和异步 checkpoint，是否需要补充？

67 行这里就是先解释了同步 checkpoint

同步和异步 checkpoint 的机制是一样的，只不过是 persist 到数据库里的方法不同。

dm/dm-checkpoint.md

Co-authored-by: Ran <huangran@pingcap.com>

dm/dm-checkpoint.md

ran-huang · 2022-04-25T03:21:46Z

dm/dm-checkpoint.md

+
+## 内存 checkpoint
+
+### 任务启动/重启


### 任务启动/重启 是什么意思，感觉 heading 和内容没有直接的对应关系。

意思是，在任务启动或者重启时，DM 对内存 checkpoint 做的初始化或者恢复工作。请问需要改成什么比较好呢

ran-huang · 2022-04-25T03:24:54Z

dm/dm-checkpoint.md

+由于每次访问和写入 flushed checkpoint 的开销很大，因此，将内存 checkpoint 作为 flushed checkpoint 在某种意义上的缓存，可以减少访问和写入 flushed checkpoint 的开销。
+
+当 DM 启动任务时，DM 会从数据库中读取该任务的 flushed checkpoint，并将 flushed checkpoint 恢复成内存 checkpoint：
+
+* 如果 flushed checkpoint 存在，说明这个任务是重启任务，binlog 的同步位置会被定位到 global checkpoint，这里将作为下次同步任务的开始位置。
+* 如果 flushed checkpoint 不存在，说明该任务是第一次启动或者重新开始同步，DM 会在下游新建一个 checkpoint。


这里有个困惑的地方，标题是 内存 checkpoint，但内容上大半是 flushed checkpoint。整篇文档里没有很好地解释清楚这两个 checkpoint 的区别与联系。

开头的地方有讲两种 checkpoint 代表的含义，简单来说

内存：已被接收并处理的 binlog 位置

flushed：已被成功同步到下游的 binlog 位置

实际上，这里就是在解释两种 checkpoint 之间的联系——内存 checkpoint 是从 flushed checkpoint 恢复而来的。

ran-huang · 2022-04-25T03:28:58Z

dm/dm-checkpoint.md

+
+### 内存 checkpoint 的更新时机
+
+- global checkpoint：当遇到 XID event 或 DDL event 时，将当前位置写入 global checkpoint。


XID event 在文档里从未出现过，似乎是个 MySQL 的术语。为了避免读者疑惑，建议添加到 DM glossary 并链接过去。

Signed-off-by: Ran <huangran@pingcap.com>

ti-chi-bot · 2022-09-24T09:21:30Z

@buchuitoudegou: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

qiancai · 2023-07-07T06:15:47Z

Removed the needs-cherry-pick-release-6.6 label because the v6.6 docs have been archived at https://docs-archive.pingcap.com/zh/tidb/v6.6 and will no longer receive new updates.

qiancai · 2023-08-25T03:51:13Z

Removed the needs-cherry-pick-release-7.0 label because the v7.0 docs have been archived at https://docs-archive.pingcap.com/zh/tidb/v7.0 and will no longer receive new updates.

qiancai · 2023-10-20T07:57:56Z

Removed the needs-cherry-pick-release-7.2 label because the v7.2 docs have been archived at https://docs-archive.pingcap.com/zh/tidb/v7.2 and will no longer receive new updates.

checkpoint mechanism and QA

0900e23

ti-chi-bot added missing-translation-status This PR does not have translation status info. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 24, 2022

buchuitoudegou changed the title ~~checkpoint mechanism and QA~~ dm: checkpoint mechanism and common QA Mar 24, 2022

buchuitoudegou added the area/dm label Mar 24, 2022

buchuitoudegou requested a review from sunzhaoyang March 24, 2022 05:36

TomShawn added type/enhancement The issue or PR belongs to an enhancement. translation/doing This PR’s assignee is translating this PR. and removed missing-translation-status This PR does not have translation status info. labels Mar 24, 2022

sunzhaoyang reviewed Mar 25, 2022

View reviewed changes

dm/dm-checkpoint.md Outdated Show resolved Hide resolved

sunzhaoyang reviewed Mar 25, 2022

View reviewed changes

dm/dm-checkpoint.md Outdated Show resolved Hide resolved

sunzhaoyang reviewed Mar 25, 2022

View reviewed changes

dm/dm-checkpoint.md Show resolved Hide resolved

fix: comment

51de817

ti-chi-bot requested a review from lance6716 March 28, 2022 06:08

sunzhaoyang reviewed Mar 28, 2022

View reviewed changes

dm/dm-checkpoint.md Outdated Show resolved Hide resolved

sunzhaoyang reviewed Mar 28, 2022

View reviewed changes

dm/dm-checkpoint.md Outdated Show resolved Hide resolved

buchuitoudegou and others added 2 commits March 28, 2022 14:14

Update dm/dm-checkpoint.md

2a8325e

Co-authored-by: sunzy <sunzy2@gmail.com>

Update dm/dm-checkpoint.md

500f480

Co-authored-by: sunzy <sunzy2@gmail.com>

lance6716 reviewed Mar 28, 2022

View reviewed changes

buchuitoudegou and others added 7 commits March 28, 2022 14:32

Update dm/dm-checkpoint.md

e370e4d

Co-authored-by: lance6716 <lance6716@gmail.com>

Update dm/dm-checkpoint.md

f3cdff5

Co-authored-by: lance6716 <lance6716@gmail.com>

Update dm/dm-checkpoint.md

67fe3ef

Co-authored-by: lance6716 <lance6716@gmail.com>

Update dm-checkpoint.md

bf78428

Update dm-checkpoint.md

2857b44

fix: lint

8e1035a

add toc

6e6d5c3

sunzhaoyang approved these changes Mar 29, 2022

View reviewed changes

shichun-0415 self-requested a review March 29, 2022 07:44

ran-huang self-assigned this Apr 12, 2022

ran-huang requested review from ran-huang and removed request for shichun-0415 April 12, 2022 06:32

ran-huang reviewed Apr 19, 2022

View reviewed changes

ran-huang reviewed Apr 20, 2022

View reviewed changes

ran-huang and others added 4 commits April 20, 2022 11:19

Merge branch 'master' into master

1421c9c

Apply suggestions from code review

89798ee

Co-authored-by: Ran <huangran@pingcap.com>

Update dm/dm-checkpoint.md

31856b5

Co-authored-by: Ran <huangran@pingcap.com>

Update dm-checkpoint.md

256203d

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 25, 2022

Merge branch 'master' into master

f74d2cc

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 25, 2022

ran-huang reviewed Apr 25, 2022

View reviewed changes

buchuitoudegou and others added 2 commits May 5, 2022 18:26

fix: explain checkpoint

e60b7b5

update wording

9d602a7

Signed-off-by: Ran <huangran@pingcap.com>

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 24, 2022

ran-huang added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Dec 29, 2022

ran-huang added the needs-cherry-pick-release-6.6 label Feb 17, 2023

ran-huang added the needs-cherry-pick-release-7.0 label Mar 29, 2023

ran-huang added needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.2 labels Jun 28, 2023

qiancai removed the needs-cherry-pick-release-6.6 label Jul 7, 2023

qiancai removed the needs-cherry-pick-release-7.0 label Aug 25, 2023

qiancai removed the needs-cherry-pick-release-7.2 label Oct 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dm: checkpoint mechanism and common QA #8749

dm: checkpoint mechanism and common QA #8749

buchuitoudegou commented Mar 24, 2022

ti-chi-bot commented Mar 24, 2022

CLAassistant commented Mar 24, 2022 •

edited

Loading

sunzhaoyang commented Mar 28, 2022

lance6716 left a comment

lance6716 Mar 28, 2022

sunzhaoyang Mar 28, 2022

sunzhaoyang left a comment

ran-huang Apr 19, 2022

sunzhaoyang Apr 20, 2022

buchuitoudegou Apr 20, 2022

sunzhaoyang Apr 20, 2022

buchuitoudegou Apr 20, 2022

ran-huang Apr 19, 2022

sunzhaoyang Apr 20, 2022

sunzhaoyang May 11, 2022

ran-huang Apr 20, 2022

sunzhaoyang Apr 20, 2022

buchuitoudegou Apr 20, 2022

ran-huang Apr 25, 2022

buchuitoudegou May 5, 2022

ran-huang Apr 25, 2022

buchuitoudegou May 5, 2022

ran-huang Apr 25, 2022

ti-chi-bot commented Sep 24, 2022

qiancai commented Jul 7, 2023

qiancai commented Aug 25, 2023

qiancai commented Oct 20, 2023


		## Q&A

		### 为什么根据配置文件启动，实际的 position 地址却没有根据配置文件定义的 position ？


		checkpoint 一般存放在下游数据库中。如果配置文件没有定义 meta 信息的库名，则默认在 dm_meta 库中。不同的任务的同步进度存放在以任务名为表名的表中。可以通过清理这些表内容的方式来清理 checkpoint（不建议），或者在下次启动任务时新增 `--remove-meta` flag 来清理已存在的 checkpoint。

		### async checkpoint 的机制？


		### 内存 checkpoint 的更新时机

		- global checkpoint：当遇到 XID event 或 DDL event 时，将当前位置写入 global checkpoint。

dm: checkpoint mechanism and common QA #8749

Are you sure you want to change the base?

dm: checkpoint mechanism and common QA #8749

Conversation

buchuitoudegou commented Mar 24, 2022

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

ti-chi-bot commented Mar 24, 2022

CLAassistant commented Mar 24, 2022 • edited Loading

sunzhaoyang commented Mar 28, 2022

lance6716 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunzhaoyang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot commented Sep 24, 2022

qiancai commented Jul 7, 2023

qiancai commented Aug 25, 2023

qiancai commented Oct 20, 2023

CLAassistant commented Mar 24, 2022 •

edited

Loading