*(ticdc): optimize memory usage of RowChangedEvent #10483

lidezhu · 2024-01-16T08:59:37Z

What problem does this PR solve?

Issue Number: close #10386

What is changed and how it works?

Add a more memory efficient struct ColumnData to represent column data, and the schema info of column can only be fetched from TableInfo;
Replace []*Column in RowChangedEvent with []*ColumnData;
Add method RowChangedEvent.GetColumns and RowChangedEvent.GetPreColumns to get the Column representation of columns;
There should be no nil in RowChangedEvent.Columns and RowChangedEvent.PreColumns, so this pr remove some code in sink module to handle nil value.
Maintain a correct TableInfo in all kinds of tests;

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

disable tidb gc;
load 10M rows of a wide table;
create changefeed with memory-quota set 8GB;
compare the peak memory usage;
Before optimization, peak memory usage: 26.5GB,

After optimization, peak memory usage: 10.2GB,

The peak memory usage decrease about 60%.

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None.

codecov · 2024-01-28T14:04:29Z

Codecov Report

Merging #10483 (fb658b9) into master (2eadc08) will increase coverage by 0.0427%.
Report is 1 commits behind head on master.
The diff coverage is 62.4595%.

Additional details and impacted files

Components	Coverage Δ
cdc	`62.0021% <62.4595%> (-0.1076%)`	⬇️
dm	`51.5055% <ø> (+0.3023%)`	⬆️
engine	`63.4494% <ø> (ø)`

Flag	Coverage Δ
unit	`57.6760% <62.4595%> (+0.0427%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #10483        +/-   ##
================================================
+ Coverage   57.6333%   57.6760%   +0.0427%     
================================================
  Files           849        849                
  Lines        126085     127025       +940     
================================================
+ Hits          72667      73263       +596     
- Misses        47988      48298       +310     
- Partials       5430       5464        +34

lidezhu · 2024-01-29T01:00:05Z

/test cdc-integration-mysql-test

lidezhu · 2024-01-29T11:33:35Z

/test verify

lidezhu · 2024-01-29T14:47:20Z

/test cdc-integration-storage-test

lidezhu · 2024-01-29T15:11:19Z

/test verify

lidezhu · 2024-01-30T12:40:10Z

/test cdc-integration-storage-test

lidezhu · 2024-01-30T16:24:09Z

/test cdc-integration-mysql-test

cdc/entry/mounter_test.go

asddongmen · 2024-02-06T09:02:06Z

Excellent work! Respect!

ti-chi-bot · 2024-02-20T05:14:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [asddongmen,sdojjy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-02-20T05:14:31Z

[LGTM Timeline notifier]

Timeline:

2024-02-06 08:58:21.910152165 +0000 UTC m=+260827.476922039: ☑️ agreed by asddongmen.
2024-02-20 05:14:30.12287006 +0000 UTC m=+334158.870493170: ☑️ agreed by sdojjy.

lidezhu · 2024-02-20T05:34:47Z

/retest

lidezhu · 2024-02-20T06:26:31Z

/retest required

ti-chi-bot · 2024-02-20T06:26:33Z

@lidezhu: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test cdc-integration-kafka-test
/test cdc-integration-mysql-test
/test cdc-integration-pulsar-test
/test cdc-integration-storage-test
/test dm-compatibility-test
/test dm-integration-test
/test engine-integration-test
/test verify
/test wip-pull-build
/test wip-pull-check
/test wip-pull-unit-test-cdc
/test wip-pull-unit-test-dm
/test wip-pull-unit-test-engine

Use /test all to run the following jobs that were automatically triggered:

pingcap/tiflow/ghpr_verify
pingcap/tiflow/pull_cdc_integration_kafka_test
pingcap/tiflow/pull_cdc_integration_pulsar_test
pingcap/tiflow/pull_cdc_integration_storage_test
pingcap/tiflow/pull_cdc_integration_test
pingcap/tiflow/pull_dm_compatibility_test
pingcap/tiflow/pull_dm_integration_test
pingcap/tiflow/pull_engine_integration_test

In response to this:

/retest required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lidezhu · 2024-02-20T07:08:03Z

/test cdc-integration-mysql-test

lidezhu · 2024-02-20T09:24:38Z

/test cdc-integration-storage-test

lidezhu · 2024-02-20T10:24:47Z

/test verify

lidezhu · 2024-02-20T11:01:56Z

/retest

…0483)" This reverts commit a8c7563.

…ingcap#10483)"" This reverts commit 4909878.

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 16, 2024

lidezhu force-pushed the reduce-cdc-mem-usage branch 2 times, most recently from 1cf2afc to d0c061e Compare January 24, 2024 09:37

lidezhu changed the title ~~[DNM] Optimize memory usage of RowChangedEvent~~ Optimize memory usage of RowChangedEvent Jan 25, 2024

asddongmen requested a review from 3AceShowHand January 25, 2024 11:59

asddongmen self-assigned this Jan 25, 2024

lidezhu force-pushed the reduce-cdc-mem-usage branch 3 times, most recently from b187bf7 to ebd4e2f Compare January 27, 2024 08:14

asddongmen removed their assignment Jan 28, 2024

asddongmen self-requested a review January 28, 2024 05:02

lidezhu force-pushed the reduce-cdc-mem-usage branch from 8646e93 to 401c574 Compare January 30, 2024 10:04

alpha version

8c4221d

lidezhu force-pushed the reduce-cdc-mem-usage branch from 3a8379a to 8901f4f Compare January 30, 2024 11:42

lidezhu force-pushed the reduce-cdc-mem-usage branch from 48605e3 to 7178975 Compare January 31, 2024 13:32

fix tests

b8dedf3

lidezhu force-pushed the reduce-cdc-mem-usage branch from 7178975 to b8dedf3 Compare January 31, 2024 13:41

lidezhu added 2 commits January 31, 2024 23:14

remove unnecessary log

f9cee8d

remove more unnecessary log

fd51374

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Feb 1, 2024

lidezhu changed the title ~~Optimize memory usage of RowChangedEvent~~ *(ticdc): optimize memory usage of RowChangedEvent Feb 1, 2024

add some comment and tests

2949bee

lidezhu force-pushed the reduce-cdc-mem-usage branch from 19df8fe to 2949bee Compare February 1, 2024 10:24

asddongmen reviewed Feb 6, 2024

View reviewed changes

cdc/entry/mounter_test.go Show resolved Hide resolved

asddongmen approved these changes Feb 6, 2024

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Feb 6, 2024

asddongmen requested a review from sdojjy February 6, 2024 09:02

asddongmen mentioned this pull request Feb 6, 2024

mysql(ticdc): Add a tableInfo cache to reduce the memory and cpu overhead of BuildTiDBTableInfo #10512

Closed

address comment

f44a661

sdojjy approved these changes Feb 20, 2024

View reviewed changes

ti-chi-bot bot added the lgtm label Feb 20, 2024

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Feb 20, 2024

Merge branch 'master' into reduce-cdc-mem-usage

3551329

fix test

fb658b9

lidezhu force-pushed the reduce-cdc-mem-usage branch from 4fab31e to fb658b9 Compare February 20, 2024 08:36

ti-chi-bot bot merged commit a8c7563 into pingcap:master Feb 20, 2024
28 checks passed

lidezhu deleted the reduce-cdc-mem-usage branch February 20, 2024 11:56

GMHDBJD added a commit to 3AceShowHand/tiflow that referenced this pull request Feb 21, 2024

Revert "*(ticdc): optimize memory usage of RowChangedEvent (pingcap#1…

4909878

…0483)" This reverts commit a8c7563.

GMHDBJD added a commit to 3AceShowHand/tiflow that referenced this pull request Feb 21, 2024

Revert "Revert "*(ticdc): optimize memory usage of RowChangedEvent (p…

42a5664

…ingcap#10483)"" This reverts commit 4909878.

lidezhu mentioned this pull request Feb 21, 2024

*: fix dm integration test failure #10631

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*(ticdc): optimize memory usage of RowChangedEvent #10483

*(ticdc): optimize memory usage of RowChangedEvent #10483

lidezhu commented Jan 16, 2024 •

edited

Loading

codecov bot commented Jan 28, 2024 •

edited

Loading

lidezhu commented Jan 29, 2024

lidezhu commented Jan 29, 2024

lidezhu commented Jan 29, 2024

lidezhu commented Jan 29, 2024

lidezhu commented Jan 30, 2024

lidezhu commented Jan 30, 2024

asddongmen commented Feb 6, 2024

ti-chi-bot bot commented Feb 20, 2024

ti-chi-bot bot commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

ti-chi-bot bot commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

*(ticdc): optimize memory usage of RowChangedEvent #10483

*(ticdc): optimize memory usage of RowChangedEvent #10483

Conversation

lidezhu commented Jan 16, 2024 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

codecov bot commented Jan 28, 2024 • edited Loading

Codecov Report

lidezhu commented Jan 29, 2024

lidezhu commented Jan 29, 2024

lidezhu commented Jan 29, 2024

lidezhu commented Jan 29, 2024

lidezhu commented Jan 30, 2024

lidezhu commented Jan 30, 2024

asddongmen commented Feb 6, 2024

ti-chi-bot bot commented Feb 20, 2024

ti-chi-bot bot commented Feb 20, 2024

[LGTM Timeline notifier]

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

ti-chi-bot bot commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Feb 20, 2024

lidezhu commented Jan 16, 2024 •

edited

Loading

codecov bot commented Jan 28, 2024 •

edited

Loading