Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

schema: reduce memory usage when init the schema #534

Merged
merged 7 commits into from
May 8, 2020
Merged

schema: reduce memory usage when init the schema #534

merged 7 commits into from
May 8, 2020

Conversation

zier-one
Copy link
Contributor

@zier-one zier-one commented May 6, 2020 •
edited by amyangfei
Loading

What problem does this PR solve?

reduce memory usage when init the schema, fix #533

What is changed and how it works?

Release note

  • Use TableInfo snapshot to init schema storage, which reduces memory consumption and speeds up the schema storage initialization, especially when DDL history is quite large.

Check List

Tests

  • Unit test
  • Integration test

Release note

  • reduce the memory usage when initializing the schema

@zier-one
Copy link
Contributor Author

zier-one commented May 7, 2020

/run-integration-tests

@zier-one zier-one changed the title cli: reduce memory usage in cli schema: reduce memory usage when init the schema May 7, 2020
@zier-one zier-one requested a review from amyangfei May 7, 2020 04:22
@codecov-io
Copy link

codecov-io commented May 7, 2020 •
edited
Loading

Codecov Report

Merging #534 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master       #534   +/-   ##
===========================================
  Coverage   26.3039%   26.3039%           
===========================================
  Files            68         68           
  Lines          6653       6653           
===========================================
  Hits           1750       1750           
  Misses         4786       4786           
  Partials        117        117

@zier-one zier-one added status/ptal Could you please take a look? subject/performance Denotes an issue or pull request is related to replication performance. labels May 7, 2020
amyangfei
amyangfei reviewed May 7, 2020
View reviewed changes
cdc/owner.go Show resolved Hide resolved
@zier-one
Copy link
Contributor Author

zier-one commented May 7, 2020

fix #533

overvenus
overvenus approved these changes May 8, 2020
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amyangfei
Copy link
Contributor

Please resolve conflicts

@sre-bot
Copy link

sre-bot commented May 8, 2020

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

Merge remote-tracking branch 'pingcap/master' into reduce_mem_cli
957a27c 
# Conflicts:
#	cdc/entry/schema_storage.go
#	cdc/owner.go
#	cdc/processor.go
@zier-one
Copy link
Contributor Author

zier-one commented May 8, 2020

/run-integration-tests

amyangfei
amyangfei approved these changes May 8, 2020
View reviewed changes
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amyangfei amyangfei added LGT2 and removed status/ptal Could you please take a look? labels May 8, 2020
@amyangfei amyangfei added this to the v4.0.0-rc.2 milestone May 8, 2020
@zier-one zier-one merged commit b645867 into pingcap:master May 8, 2020
@zier-one zier-one deleted the reduce_mem_cli branch May 8, 2020 06:06
5kbpers pushed a commit to 5kbpers/ticdc that referenced this pull request Aug 24, 2020
schema: reduce memory usage when init the schema (pingcap#534)
df2565c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei approved these changes

@overvenus overvenus overvenus approved these changes

Assignees
No one assigned
Labels
subject/performance Denotes an issue or pull request is related to replication performance.
Projects
None yet
Milestone
v4.0.0-rc.2
Development

Successfully merging this pull request may close these issues.

CDC cli hang and OOM
5 participants
@zier-one @codecov-io @amyangfei @sre-bot @overvenus