Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

Optimize mysql consistency #121

Merged
merged 9 commits into from
Aug 3, 2020

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Jul 13, 2020

What problem does this PR solve?

fix #118

  1. Now dumpling will use FTWRL lock tables from start to the end, which is not suitable.
  2. Dumpling --threads argument can't control concurrently dumping tables.
  3. ResolveAutoConsistency in adjustConfig will lead consistency to none because serverType is not got yet.

What is changed and how it works?

  1. Change rateLimit to connectionsPool to control database connections like mydumper.
  2. Add Start for TableIR. concurrentDumpTable will return an array of not started TableIRs.
  3. Move ResolveAutoConsistency after detect ServerType.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

Side effects

  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release note

  • Release FTWRL after connections are created to reduce lock time for mysql.

@lichunzhu lichunzhu requested review from 3pointer and kennytm July 13, 2020 04:58
@codecov
Copy link

codecov bot commented Jul 13, 2020

Codecov Report

Merging #121 into master will decrease coverage by 1.43%.
The diff coverage is 35.18%.

@@            Coverage Diff             @@
##           master     #121      +/-   ##
==========================================
- Coverage   52.19%   50.75%   -1.44%     
==========================================
  Files          17       17              
  Lines        1868     1909      +41     
==========================================
- Hits          975      969       -6     
- Misses        825      868      +43     
- Partials       68       72       +4     

Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lichunzhu lichunzhu merged commit 4c8f99c into pingcap:master Aug 3, 2020
@lichunzhu lichunzhu deleted the optimizeMySQLConsistency branch August 3, 2020 09:35
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce the time dumpling holds the FTWRL lock
2 participants