[Enhancement] Reduce the memory useage of TableStatistic #50316

Seaven · 2024-08-27T07:39:53Z

Why I'm doing:

What I'm doing:

to reduce memory usage
for avoid bugs: will init too much cacheKey when the data of statistics table is error (history bugs, doesn't delete statistics rows when truncate table or drop partitions).

update Cache getAll implement: only query exists partitions, not all partition
replace TableStatistic by Long, tableID and partitionID is unused

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: Seaven <seaven_7@qq.com>

fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/TableStatsCacheLoader.java

sonarqubecloud · 2024-08-27T07:46:27Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-08-27T09:06:27Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-08-27T09:07:15Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-08-27T09:07:42Z

[FE Incremental Coverage Report]

❌ fail : 37 / 48 (77.08%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/statistics/TableStatsCacheLoader.java	12	23	52.17%	[44, 45, 46, 47, 49, 77, 78, 80, 81, 82, 83]
🔵	com/starrocks/statistic/BasicStatsMeta.java	3	3	100.00%	[]
🔵	com/starrocks/statistic/StatisticSQLBuilder.java	3	3	100.00%	[]
🔵	com/starrocks/statistic/StatisticExecutor.java	2	2	100.00%	[]
🔵	com/starrocks/sql/optimizer/statistics/StatisticsCalcUtils.java	8	8	100.00%	[]
🔵	com/starrocks/sql/optimizer/statistics/CachedStatisticStorage.java	8	8	100.00%	[]
🔵	com/starrocks/sql/optimizer/statistics/StatisticStorage.java	1	1	100.00%	[]

github-actions · 2024-08-28T09:42:10Z

@Mergifyio backport branch-3.3

github-actions · 2024-08-28T09:42:11Z

@Mergifyio backport branch-3.2

github-actions · 2024-08-28T09:42:13Z

@Mergifyio backport branch-3.1

github-actions · 2024-08-28T09:42:15Z

@Mergifyio backport branch-3.0

mergify · 2024-08-28T09:42:28Z

backport branch-3.3

✅ Backports have been created

#50371 [Enhancement] Reduce the memory useage of TableStatistic (backport #50316) has been created for branch branch-3.3 but encountered conflicts

mergify · 2024-08-28T09:42:30Z

backport branch-3.2

✅ Backports have been created

#50372 [Enhancement] Reduce the memory useage of TableStatistic (backport #50316) has been created for branch branch-3.2 but encountered conflicts

mergify · 2024-08-28T09:42:31Z

backport branch-3.1

✅ Backports have been created

#50373 [Enhancement] Reduce the memory useage of TableStatistic (backport #50316) has been created for branch branch-3.1 but encountered conflicts

mergify · 2024-08-28T09:42:32Z

backport branch-3.0

✅ Backports have been created

#50374 [Enhancement] Reduce the memory useage of TableStatistic (backport #50316) has been created for branch branch-3.0 but encountered conflicts

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/StatisticStorage.java # fe/fe-core/src/main/java/com/starrocks/statistic/BasicStatsMeta.java

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/StatisticStorage.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/StatisticsCalcUtils.java # fe/fe-core/src/main/java/com/starrocks/statistic/BasicStatsMeta.java # fe/fe-core/src/test/java/com/starrocks/sql/plan/MockTpchStatisticStorage.java

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/CachedStatisticStorage.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/StatisticStorage.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/StatisticsCalcUtils.java # fe/fe-core/src/main/java/com/starrocks/statistic/BasicStatsMeta.java # fe/fe-core/src/test/java/com/starrocks/sql/plan/MockTpchStatisticStorage.java

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

…0316) (#50372) Co-authored-by: Seaven <seaven_7@qq.com>

…0316) (#50373) Co-authored-by: Seaven <seaven_7@qq.com>

…0316) (#50371) Co-authored-by: Seaven <seaven_7@qq.com>

…0316) Signed-off-by: Seaven <seaven_7@qq.com>

…0316) Signed-off-by: Seaven <seaven_7@qq.com> Signed-off-by: zhiminr.ren <1240388654@qq.com>

[Enhancement] Reduce the memory useage of TableStatistic

43fb526

Signed-off-by: Seaven <seaven_7@qq.com>

Seaven requested a review from a team as a code owner August 27, 2024 07:39

mergify bot assigned Seaven Aug 27, 2024

starrocks-cr bot reviewed Aug 27, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/TableStatsCacheLoader.java Show resolved Hide resolved

github-actions bot added 3.3 3.2 3.1 3.0 2.5 labels Aug 27, 2024

packy92 approved these changes Aug 27, 2024

View reviewed changes

kangkaisen approved these changes Aug 27, 2024

View reviewed changes

github-actions bot removed the 2.5 label Aug 28, 2024

kangkaisen merged commit 16ca57c into StarRocks:main Aug 28, 2024
89 of 95 checks passed

github-actions bot removed the 3.3 label Aug 28, 2024

github-actions bot removed the 3.2 label Aug 28, 2024

github-actions bot removed the 3.1 label Aug 28, 2024

github-actions bot removed the 3.0 label Aug 28, 2024

mergify bot mentioned this pull request Aug 28, 2024

[Enhancement] Reduce the memory useage of TableStatistic (backport #50316) #50371

Merged

42 tasks

This was referenced Aug 28, 2024

[Enhancement] Reduce the memory useage of TableStatistic (backport #50316) #50372

Merged

[Enhancement] Reduce the memory useage of TableStatistic (backport #50316) #50373

Merged

mergify bot mentioned this pull request Aug 28, 2024

[Enhancement] Reduce the memory useage of TableStatistic (backport #50316) #50374

Closed

42 tasks

github-actions bot added 3.2 3.1 3.3 3.0 labels Aug 30, 2024

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

7f32c04

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

83e45f7

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

28cf94d

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

8e570d8

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

github-actions bot removed the 3.0 label Sep 2, 2024

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

3df6ea8

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

bc073bb

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

5c77754

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

Seaven added a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (#50316)

ec770cd

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit 16ca57c)

wanpengfei-git pushed a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (backport #5…

ebe4b92

…0316) (#50372) Co-authored-by: Seaven <seaven_7@qq.com>

github-actions bot added the 3.2-merged label Sep 2, 2024

wanpengfei-git pushed a commit that referenced this pull request Sep 2, 2024

[Enhancement] Reduce the memory useage of TableStatistic (backport #5…

a5b5316

…0316) (#50373) Co-authored-by: Seaven <seaven_7@qq.com>

github-actions bot added the 3.1-merged label Sep 2, 2024

murphyatwork pushed a commit that referenced this pull request Sep 7, 2024

[Enhancement] Reduce the memory useage of TableStatistic (backport #5…

286198b

…0316) (#50371) Co-authored-by: Seaven <seaven_7@qq.com>

github-actions bot added the 3.3-merged label Sep 7, 2024

va-os-commits pushed a commit to va-os-commits/starrocks that referenced this pull request Sep 10, 2024

[Enhancement] Reduce the memory useage of TableStatistic (StarRocks#5…

fb84322

…0316) Signed-off-by: Seaven <seaven_7@qq.com>

renzhimin7 pushed a commit to renzhimin7/starrocks that referenced this pull request Nov 7, 2024

[Enhancement] Reduce the memory useage of TableStatistic (StarRocks#5…

6cbda54

…0316) Signed-off-by: Seaven <seaven_7@qq.com> Signed-off-by: zhiminr.ren <1240388654@qq.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Enhancement] Reduce the memory useage of TableStatistic #50316

[Enhancement] Reduce the memory useage of TableStatistic #50316

Uh oh!

Seaven commented Aug 27, 2024 •

edited

Loading

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 27, 2024

Uh oh!

github-actions bot commented Aug 27, 2024

Uh oh!

github-actions bot commented Aug 27, 2024

Uh oh!

github-actions bot commented Aug 27, 2024

Uh oh!

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

mergify bot commented Aug 28, 2024 •

edited

Loading

Uh oh!

mergify bot commented Aug 28, 2024 •

edited

Loading

Uh oh!

mergify bot commented Aug 28, 2024 •

edited

Loading

Uh oh!

mergify bot commented Aug 28, 2024 •

edited

Loading

Uh oh!

Uh oh!

[Enhancement] Reduce the memory useage of TableStatistic #50316

[Enhancement] Reduce the memory useage of TableStatistic #50316

Uh oh!

Conversation

Seaven commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed

Uh oh!

github-actions bot commented Aug 27, 2024

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions bot commented Aug 27, 2024

[BE Incremental Coverage Report]

Uh oh!

github-actions bot commented Aug 27, 2024

[FE Incremental Coverage Report]

file detail

Uh oh!

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

mergify bot commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify bot commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify bot commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify bot commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

Seaven commented Aug 27, 2024 •

edited

Loading

mergify bot commented Aug 28, 2024 •

edited

Loading

mergify bot commented Aug 28, 2024 •

edited

Loading

mergify bot commented Aug 28, 2024 •

edited

Loading

mergify bot commented Aug 28, 2024 •

edited

Loading