kvclient(ticdc): fix kvclient takes too long time to recover #3612

maxshuang · 2021-11-24T15:32:11Z

close #3191
close flaky test in kvclient: #2694 #3302 #2349 #2688 #2747

What problem does this PR solve?

When a tikv node which has 10k region leaders fail, we found it need more than 30min to recover.
We think it is abnormal for the reason that pd and tikv only need about 30s to tag a node 'disconnect' and elect a new leader.

What is changed and how it works?

Decrease the retry num when a new stream fail to establish to make other region try ASAP.
Remove PartialClone when the region fail.

Reason

Expectation
We use a tikv node which has about 3k region leaders as our comparison object. After test in normal case and abnormal case, we got follow results:

(1) ~ 1min for 3k region to finish init scan and get event (normal case)
(2) > 20min for all 3k region to recover the stream (tikv node fail)

But, pd and tikv only need about 30s to tag a node 'disconnect' and elect a new leader, so a reasonable timespan to recover is about:

       1min30s(normal case + region failover) ~ 5min (retry control in cdc)

Consider the probability of network jitter >> tikv temporary down >> tikv permanent down, we still need some retry logic, so

Decrease the retry num when a new stream fail to establish  to make other region try ASAP.

When a region fail and call RegionCache.OnRegionFail, it will (1) mark the store as 'needcheck' and fetch new store state of the store asynchronous, (2) move its leader to other peer to try next time and fetch new leader back. This mechanism is suitable in our case, but we find all failed region still connect to the old store which is quite strange. So we

Remove wrong PartialClone logic when the region fail in region_worker.

Result

3min30s to recover from 3k fail (after this pr)  compared to >20min (before)

Check List

Tests

Manual test (add detailed scripts or steps below)

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update key monitor metrics in both TiCDC document and official document

Release note

Please add a release note.
fix kvclient takes too long time to recover

ti-chi-bot · 2021-11-24T15:32:12Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

amyangfei
overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

cdc/kv/client.go

amyangfei · 2021-11-26T03:53:31Z

/run-all-tests /tidb=release-5.3

amyangfei · 2021-11-26T04:05:09Z

/run-all-tests /tidb=pr/30008

close #3191

codecov-commenter · 2021-11-28T17:13:07Z

Codecov Report

Merging #3612 (69da71e) into master (24030f8) will increase coverage by 1.5585%.
The diff coverage is 68.1107%.

@@               Coverage Diff                @@
##             master      #3612        +/-   ##
================================================
+ Coverage   56.5356%   58.0941%   +1.5585%     
================================================
  Files           211        243        +32     
  Lines         22798      24567      +1769     
================================================
+ Hits          12889      14272      +1383     
- Misses         8598       8864       +266     
- Partials       1311       1431       +120

maxshuang · 2021-11-29T02:59:31Z

/run-kafka-tests
/run-all-tests

maxshuang · 2021-11-29T03:10:24Z

/run-kafka-tests

ti-chi-bot · 2021-11-29T10:24:30Z

In response to a cherrypick label: new pull request created: #3659.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-29T10:25:08Z

In response to a cherrypick label: new pull request created: #3660.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-29T10:25:45Z

In response to a cherrypick label: new pull request created: #3661.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-29T10:26:22Z

In response to a cherrypick label: new pull request created: #3662.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-29T10:26:59Z

In response to a cherrypick label: new pull request created: #3663.

…#3612) (pingcap#3661)

…#3612) (pingcap#3662)

…#3612) (pingcap#3663)

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ * address comments * add comments for exported functions * fix the compiling problem * workerpool: limit the rate to output deadlock warning (#3775) (#3799) * metrics: add changefeed checkepoint catch-up ETA (#3300) (#3311) * pkg,cdc: do not use log package (#3902) (#3939) * *: rename repo from pingcap/ticdc to pingcap/tiflow (#3957) * kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#3662) * tz (ticdc): fix timezone error (#3887) (#3910) * http_*: add log for http api and refine the err handle logic (#2997) (#3306) * ticdc/alert: add no owner alert rule (#3809) (#3831) * etcd_worker: batch etcd patch (#3277) (#3393) * ticdc/owner: Fix ddl special comment syntax error (#3845) (#3977) * Clean old owner and old processor in release 5.2 branch (#4019) * tests(ticdc): set up the sync diff output directory correctly (#3725) (#3745) * owner,scheduler(cdc): fix nil pointer panic in owner scheduler (#2980) (#4007) (#4015) * config(ticdc): Fix old value configuration check for maxwell protocol (#3747) (#3782) * ticdc/processor: Fix backoff base delay misconfiguration (#3992) (#4027) * sink(ticdc): cherry pick sink bug fix to release 5.2 (#4083) (#4119) * metrics(ticdc): add resolved ts and add changefeed to dataflow (#4038) (#4103) * kv(ticdc): reduce eventfeed rate limited log (#4072) (#4110) close #4006 * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * http_api (ticdc): fix http api 'get processor' panic. (#4117) (#4122) close #3840 * cdc/sink: adjust kafka initialization logic (#3192) (#3568) * This is an automated cherry-pick of #3192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix conflicts. * This is an automated cherry-pick of #3682 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix import. * fix failpoint path. * try to fix initialize. * remove table_sink. * remove initialization. * remove owner. * fix mq. Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> Co-authored-by: 3AceShowHand <jinl1037@hotmail.com> * sink (ticdc): fix a deadlock due to checkpointTs fall back in sinkNode (#4084) (#4098) close #4055 * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix conflicts. * fix conflicts. * fix conflicts. Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com> Co-authored-by: amyangfei <yangfei@pingcap.com> Co-authored-by: dongmen <20351731+asddongmen@users.noreply.github.com> Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ * address comments * add comments for exported functions * fix the compiling problem * workerpool: limit the rate to output deadlock warning (#3775) (#3795) * tests(ticdc): set up the sync diff output directory correctly (#3725) (#3741) * relay(dm): use binlog name comparison (#3710) (#3712) * dm/load: fix concurrent call Loader.Status (#3459) (#3468) * cdc/sorter: make unified sorter cgroup aware (#3436) (#3439) * tz (ticdc): fix timezone error (#3887) (#3906) * pkg,cdc: do not use log package (#3902) (#3940) * *: rename repo from pingcap/ticdc to pingcap/tiflow (#3959) * http_*: add log for http api and refine the err handle logic (#2997) (#3307) * etcd_worker: batch etcd patch (#3277) (#3389) * http_api (ticdc): check --cert-allowed-cn before add server common name (#3628) (#3882) * kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#3663) * owner: fix owner tick block http request (#3490) (#3530) * dm/syncer: use downstream PK/UK to generate DML (#3168) (#3256) * dep(dm): update go-mysql (#3914) (#3934) * dm/syncer: multiple rows use downstream schema (#3308) (#3953) * errorutil,sink,syncer: add errorutil to handle ignorable error (#3264) (#3995) * dm/worker: don't exit when failed to read checkpoint in relay (#3345) (#4005) * syncer(dm): use an early location to reset binlog and open safemode (#3860) * ticdc/owner: Fix ddl special comment syntax error (#3845) (#3978) * dm/scheduler: fix inconsistent of relay status (#3474) (#4009) * owner,scheduler(cdc): fix nil pointer panic in owner scheduler (#2980) (#4007) (#4016) * config(ticdc): Fix old value configuration check for maxwell protocol (#3747) (#3783) * sink(ticdc): cherry pick sink bug fix to release 5.3 (#4083) * master(dm): clean and treat invalid load task (#4004) (#4145) * loader: fix wrong progress in query-status for loader (#4093) (#4143) close #3252 * ticdc/processor: Fix backoff base delay misconfiguration (#3992) (#4028) * dm: load table structure from dump files (#3295) (#4163) * compactor: fix duplicate entry in safemode (#3432) (#3434) (#4088) * kv(ticdc): reduce eventfeed rate limited log (#4072) (#4111) close #4006 * metrics(ticdc): add resolved ts and add changefeed to dataflow (#4038) (#4104) * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * retry(dm): align with tidb latest error message (#4172) (#4254) close #4159, close #4246 * owner(ticdc): Add bootstrap and try to fix the meta information in it (#3838) (#3865) * redolog: add a precleanup process when s3 enable (#3525) (#3878) * ddl(dm): make skipped ddl pass `SplitDDL()` (#4176) (#4227) close #4173 * cdc/sink: remove Initialize method from the sink interface (#3682) (#3765) Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> * http_api (ticdc): fix http api 'get processor' panic. (#4117) (#4123) close #3840 * sink (ticdc): fix a deadlock due to checkpointTs fall back in sinkNode (#4084) (#4099) close #4055 * cdc/sink: adjust kafka initialization logic (#3192) (#4162) * try fix conflicts. * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix conflicts. * fix conflicts. Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com> Co-authored-by: amyangfei <yangfei@pingcap.com> Co-authored-by: lance6716 <lance6716@gmail.com> Co-authored-by: sdojjy <sdojjy@qq.com> Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>

maxshuang requested review from amyangfei, overvenus and asddongmen November 24, 2021 15:32

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 24, 2021

maxshuang requested a review from liuzix November 25, 2021 04:44

amyangfei reviewed Nov 25, 2021

View reviewed changes

cdc/kv/client.go Show resolved Hide resolved

amyangfei modified the milestones: v5.3.1, v5.4.0 Nov 26, 2021

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2021

kvclient(ticdc): fix kvclient takes too long time to recover

40eae57

close #3191

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2021

format(kvclient): fix format

fec7205

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2021

Merge branch 'master' into fix-slow-region-getstream

e4313d1

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2021

Merge branch 'master' into fix-slow-region-getstream

cdff6b8

ti-chi-bot mentioned this pull request Nov 29, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) #3659

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2021

This is an automated cherry-pick of pingcap#3612

ae7527a

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2021

This is an automated cherry-pick of pingcap#3612

587d670

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 29, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) #3660

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2021

This is an automated cherry-pick of pingcap#3612

6d08623

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 29, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) #3661

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2021

This is an automated cherry-pick of pingcap#3612

96dcf69

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 29, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) #3662

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2021

This is an automated cherry-pick of pingcap#3612

d09737d

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 29, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) #3663

Merged

okJiang pushed a commit to okJiang/tiflow that referenced this pull request Dec 8, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (pingcap…

ea573c7

…#3612)

ti-chi-bot added a commit that referenced this pull request Dec 20, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#…

62101ec

…3662)

ti-chi-bot added a commit that referenced this pull request Dec 20, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#…

c5de914

…3661)

ti-chi-bot added a commit that referenced this pull request Dec 20, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#…

2b0f9b7

…3663)

3AceShowHand pushed a commit to ti-chi-bot/tiflow that referenced this pull request Dec 25, 2021

kvclient(ticdc): fix kvclient takes too long time to recover (pingcap…

92b5021

…#3612) (pingcap#3661)

3AceShowHand pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 13, 2022

kvclient(ticdc): fix kvclient takes too long time to recover (pingcap…

d93f48e

…#3612) (pingcap#3662)

3AceShowHand pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 13, 2022

kvclient(ticdc): fix kvclient takes too long time to recover (pingcap…

998285e

…#3612) (pingcap#3663)

3AceShowHand mentioned this pull request Jul 21, 2022

cdc owner switched and latency spikes when reload the tikv instance #6415

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvclient(ticdc): fix kvclient takes too long time to recover #3612

kvclient(ticdc): fix kvclient takes too long time to recover #3612

maxshuang commented Nov 24, 2021 •

edited

Loading

ti-chi-bot commented Nov 24, 2021 •

edited

Loading

amyangfei commented Nov 26, 2021

amyangfei commented Nov 26, 2021

codecov-commenter commented Nov 28, 2021 •

edited

Loading

maxshuang commented Nov 29, 2021

maxshuang commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

kvclient(ticdc): fix kvclient takes too long time to recover #3612

kvclient(ticdc): fix kvclient takes too long time to recover #3612

Conversation

maxshuang commented Nov 24, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Reason

Result

Check List

Release note

ti-chi-bot commented Nov 24, 2021 • edited Loading

amyangfei commented Nov 26, 2021

amyangfei commented Nov 26, 2021

codecov-commenter commented Nov 28, 2021 • edited Loading

Codecov Report

maxshuang commented Nov 29, 2021

maxshuang commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

ti-chi-bot commented Nov 29, 2021

maxshuang commented Nov 24, 2021 •

edited

Loading

ti-chi-bot commented Nov 24, 2021 •

edited

Loading

codecov-commenter commented Nov 28, 2021 •

edited

Loading