statistics: Update stats meta count and modify count for dropped table partitions #48929

Rustin170506 · 2023-11-27T09:37:48Z

What problem does this PR solve?

Issue Number: close #48182

Problem Summary:

What changed and how does it work?

Based on this design doc, we will only update the count and modfiy_count for it. Then it will use the count and modify_count to determine if we need to trigger an auto-analysis task.

So in this PR, I updated the count and modify count of the global table.

Check List

Tests

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

dropped table partitions Signed-off-by: hi-rustin <rustin.liu@gmail.com>

codecov · 2023-11-27T09:50:51Z

Codecov Report

Merging #48929 (acf1509) into master (d3f399f) will increase coverage by 1.6541%.
Report is 18 commits behind head on master.
The diff coverage is 4.0816%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #48929        +/-   ##
================================================
+ Coverage   70.9584%   72.6126%   +1.6541%     
================================================
  Files          1368       1390        +22     
  Lines        404999     415363     +10364     
================================================
+ Hits         287381     301606     +14225     
+ Misses        97587      94740      -2847     
+ Partials      20031      19017      -1014

Flag	Coverage Δ
integration	`43.7069% <4.1666%> (?)`
unit	`70.9733% <8.6956%> (+0.0149%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9663% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`48.4274% <ø> (-4.6208%)`	⬇️

hawkingrei · 2023-11-28T02:12:41Z

/retest

count Signed-off-by: hi-rustin <rustin.liu@gmail.com>

UpdateStatsMetaDelta Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Rustin170506 · 2023-11-28T08:04:24Z

/retest

pkg/statistics/handle/ddl/ddl.go

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

pkg/statistics/handle/ddl/ddl.go

time-and-fate · 2023-11-28T15:38:36Z

pkg/statistics/handle/ddl/ddl.go

+				logutil.StatsLogger.Warn(
+					"drop partition with pseudo stats, "+
+						"usually it won't happen because we always load stats when initializing the handle",
+					zap.String("table", globalTableInfo.Name.O),
+					zap.String("partition", def.Name.O),


I think db name should also be printed, but looks like it would take much extra work to implement that.

We need to pass the info schema here. And it involves a lot of test changes. I will do it in a separate PR.

time-and-fate

Not big problems. I'm OK to merge it now.

ti-chi-bot · 2023-11-28T15:54:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qw4990, time-and-fate

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [qw4990,time-and-fate]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-11-28T15:54:33Z

[LGTM Timeline notifier]

Timeline:

2023-11-28 08:45:03.920014677 +0000 UTC m=+912332.585240872: ☑️ agreed by qw4990.
2023-11-28 15:54:31.618621606 +0000 UTC m=+938100.283847802: ☑️ agreed by time-and-fate.

time-and-fate · 2023-11-28T15:55:05Z

/hold

time-and-fate · 2023-11-28T15:55:36Z

Just /hold it in case it's merged accidentally by the bot.
Feel free to /unhold it.

hawkingrei · 2023-11-29T01:47:31Z

/test all

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Rustin170506 · 2023-11-29T02:25:36Z

/unhold

Rustin170506 added 2 commits November 27, 2023 16:40

Refactor SaveStatsToStorage function signature

40e39bc

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

statistics: Update stats meta count and modify count for

7796880

dropped table partitions Signed-off-by: hi-rustin <rustin.liu@gmail.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 27, 2023

Rustin170506 added 2 commits November 28, 2023 14:48

Fix drop partition issue and update stats meta

a8baaa0

count Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Add a test and fix parameter order in

9ff2ade

UpdateStatsMetaDelta Signed-off-by: hi-rustin <rustin.liu@gmail.com>

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 28, 2023

Rustin170506 added 2 commits November 28, 2023 15:22

More rows in partition0

11f075a

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Always drop the partition stats

8a41f12

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Rustin170506 requested review from qw4990 and time-and-fate November 28, 2023 08:10

hawkingrei reviewed Nov 28, 2023

View reviewed changes

pkg/statistics/handle/ddl/ddl.go Outdated Show resolved Hide resolved

qw4990 approved these changes Nov 28, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Nov 28, 2023

Fix import statements in ddl.go and BUILD.bazel

bb47af0

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Rustin170506 requested a review from hawkingrei November 28, 2023 09:40

time-and-fate reviewed Nov 28, 2023

View reviewed changes

time-and-fate approved these changes Nov 28, 2023

View reviewed changes

ti-chi-bot bot added the lgtm label Nov 28, 2023

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 28, 2023

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 28, 2023

Better count calculation in HandleDDLEvent function

acf1509

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2023

ti-chi-bot bot merged commit c426cd9 into pingcap:master Nov 29, 2023
16 checks passed

Rustin170506 deleted the rustin-patch-drop-partition-count branch November 29, 2023 03:06

Rustin170506 mentioned this pull request Nov 29, 2023

statistics: log the db name #49011

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: Update stats meta count and modify count for dropped table partitions #48929

statistics: Update stats meta count and modify count for dropped table partitions #48929

Rustin170506 commented Nov 27, 2023 •

edited

Loading

codecov bot commented Nov 27, 2023 •

edited

Loading

hawkingrei commented Nov 28, 2023

Rustin170506 commented Nov 28, 2023

time-and-fate Nov 28, 2023

Rustin170506 Nov 29, 2023

Rustin170506 Nov 29, 2023

time-and-fate left a comment

ti-chi-bot bot commented Nov 28, 2023

ti-chi-bot bot commented Nov 28, 2023

time-and-fate commented Nov 28, 2023

time-and-fate commented Nov 28, 2023 •

edited

Loading

hawkingrei commented Nov 29, 2023

Rustin170506 commented Nov 29, 2023

statistics: Update stats meta count and modify count for dropped table partitions #48929

statistics: Update stats meta count and modify count for dropped table partitions #48929

Conversation

Rustin170506 commented Nov 27, 2023 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

codecov bot commented Nov 27, 2023 • edited Loading

Codecov Report

hawkingrei commented Nov 28, 2023

Rustin170506 commented Nov 28, 2023

time-and-fate Nov 28, 2023

Choose a reason for hiding this comment

Rustin170506 Nov 29, 2023

Choose a reason for hiding this comment

Rustin170506 Nov 29, 2023

Choose a reason for hiding this comment

time-and-fate left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 28, 2023

ti-chi-bot bot commented Nov 28, 2023

[LGTM Timeline notifier]

time-and-fate commented Nov 28, 2023

time-and-fate commented Nov 28, 2023 • edited Loading

hawkingrei commented Nov 29, 2023

Rustin170506 commented Nov 29, 2023

Rustin170506 commented Nov 27, 2023 •

edited

Loading

codecov bot commented Nov 27, 2023 •

edited

Loading

time-and-fate commented Nov 28, 2023 •

edited

Loading