remove big blob file type in ps v3 #5544

lidezhu · 2022-08-05T01:22:32Z

What problem does this PR solve?

Issue Number: close #5558

Problem Summary: We use a special type of BlobFile to handle write which is larger than BlobFile limit size which make the logic a little complicated.

What is changed and how it works?

Remove big blob file;
Add metrics for the distribution of write batch size:

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-08-05T01:22:33Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dbms/src/Storages/Page/V3/BlobStore.h

flowbehappy · 2022-08-05T10:19:55Z

dbms/src/Storages/Page/V3/spacemap/SpaceMap.cpp

@@ -43,9 +42,6 @@ SpaceMapPtr SpaceMap::createSpaceMap(SpaceMapType type, UInt64 start, UInt64 end
    case SMAP64_STD_MAP:


BTW, can we remove RBTreeSpaceMap? I don't think it is ever useful.

Yes, so should we remove it in another pr or in this pr?

lidezhu · 2022-08-08T08:12:23Z

/run-check-issue-triage-complete

lidezhu · 2022-08-08T08:28:06Z

/run-all-tests

sre-bot · 2022-08-08T09:05:29Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Settings.h                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Storages/Page/ConfigSettings.cpp                          6                 0   100.00%           2                 0   100.00%          20                 0   100.00%           4                 2    50.00%
Storages/Page/V3/BlobStore.cpp                          621               161    74.07%          54                 2    96.30%        1192               264    77.85%         330               100    69.70%
Storages/Page/V3/BlobStore.h                              9                 2    77.78%           9                 2    77.78%          39                21    46.15%           0                 0         -
Storages/Page/V3/spacemap/SpaceMap.cpp                   37                 4    89.19%           7                 0   100.00%          51                13    74.51%          22                 7    68.18%
Storages/Page/V3/spacemap/SpaceMap.h                      7                 2    71.43%           4                 1    75.00%          18                 5    72.22%           6                 1    83.33%
Storages/Page/V3/tests/gtest_blob_store.cpp             517                85    83.56%          29                 0   100.00%        1309                12    99.08%         180                52    71.11%
Storages/Page/V3/tests/gtest_page_directory.cpp        1172               245    79.10%          58                 0   100.00%        1859                 7    99.62%         174               140    19.54%
TestUtils/MockDiskDelegator.h                            39                 8    79.49%          20                 3    85.00%          83                14    83.13%          14                 5    64.29%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2409               507    78.95%         184                 8    95.65%        4572               336    92.65%         730               307    57.95%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
19005      9227             51.45%    215495  94015        56.37%

full coverage report (for internal network access only)

dbms/src/Storages/Page/V3/BlobStore.cpp

lidezhu · 2022-08-23T07:40:09Z

try to add metric(wip)

need to show the metrics in the grafana panel

Added. PTAL.

Co-authored-by: JaySon <tshent@qq.com>

…zhu/tics into support-config-blob-file-limit-size

lidezhu · 2022-08-23T08:05:17Z

/run-all-tests

metrics/grafana/tiflash_summary.json

dbms/src/Common/TiFlashMetrics.h

dbms/src/Storages/Page/V3/tests/gtest_blob_store.cpp

lidezhu · 2022-08-23T11:14:09Z

/run-all-tests

sre-bot · 2022-08-23T11:33:47Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/TiFlashMetrics.h                                  18                 0   100.00%          11                 0   100.00%          47                 1    97.87%           8                 0   100.00%
Interpreters/Settings.h                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Storages/Page/V3/BlobStore.cpp                          616               141    77.11%          53                 0   100.00%        1186               233    80.35%         322                92    71.43%
Storages/Page/V3/BlobStore.h                              9                 2    77.78%           9                 2    77.78%          39                21    46.15%           0                 0         -
Storages/Page/V3/spacemap/SpaceMap.cpp                   37                 4    89.19%           7                 0   100.00%          51                13    74.51%          22                 7    68.18%
Storages/Page/V3/spacemap/SpaceMap.h                      7                 2    71.43%           4                 1    75.00%          18                 5    72.22%           6                 1    83.33%
Storages/Page/V3/tests/gtest_blob_store.cpp             519                85    83.62%          29                 0   100.00%        1306                12    99.08%         180                52    71.11%
Storages/Page/V3/tests/gtest_page_directory.cpp        1289               274    78.74%          62                 0   100.00%        2104                 7    99.67%         246               158    35.77%
TestUtils/MockDiskDelegator.h                            39                 4    89.74%          20                 1    95.00%          83                 4    95.18%          14                 4    71.43%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2535               512    79.80%         196                 4    97.96%        4835               296    93.88%         798               314    60.65%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18401      8326             54.75%    212650  85900        59.60%

full coverage report (for internal network access only)

JaySon-Huang

Use interpolateSpectral and show the legend is better

metrics/grafana/tiflash_summary.json

Co-authored-by: JaySon <tshent@qq.com>

lidezhu · 2022-08-23T12:33:51Z

/merge

ti-chi-bot · 2022-08-23T12:33:53Z

@lidezhu: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-08-23T12:33:54Z

This pull request has been accepted and is ready to merge.

Commit hash: 4f81d68

lidezhu · 2022-08-23T12:57:03Z

/run-unit-test

sre-bot · 2022-08-23T13:26:32Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/TiFlashMetrics.h                                  18                 0   100.00%          11                 0   100.00%          47                 1    97.87%           8                 0   100.00%
Interpreters/Settings.h                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Storages/Page/V3/BlobStore.cpp                          616               141    77.11%          53                 0   100.00%        1186               233    80.35%         322                92    71.43%
Storages/Page/V3/BlobStore.h                              9                 2    77.78%           9                 2    77.78%          39                21    46.15%           0                 0         -
Storages/Page/V3/spacemap/SpaceMap.cpp                   37                 4    89.19%           7                 0   100.00%          51                13    74.51%          22                 7    68.18%
Storages/Page/V3/spacemap/SpaceMap.h                      7                 2    71.43%           4                 1    75.00%          18                 5    72.22%           6                 1    83.33%
Storages/Page/V3/tests/gtest_blob_store.cpp             519                85    83.62%          29                 0   100.00%        1306                12    99.08%         180                52    71.11%
Storages/Page/V3/tests/gtest_page_directory.cpp        1289               274    78.74%          62                 0   100.00%        2104                 7    99.67%         246               158    35.77%
TestUtils/MockDiskDelegator.h                            39                 4    89.74%          20                 1    95.00%          83                 4    95.18%          14                 4    71.43%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2535               512    79.80%         196                 4    97.96%        4835               296    93.88%         798               314    60.65%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18402      8327             54.75%    212697  85990        59.57%

full coverage report (for internal network access only)

lidezhu changed the title ~~[WIP]: support config blob file limit size~~ [WIP] support config blob file limit size Aug 5, 2022

flowbehappy reviewed Aug 5, 2022

View reviewed changes

dbms/src/Storages/Page/V3/BlobStore.h Outdated Show resolved Hide resolved

flowbehappy reviewed Aug 5, 2022

View reviewed changes

lidezhu force-pushed the support-config-blob-file-limit-size branch from 651088c to c49b254 Compare August 8, 2022 07:18

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. do-not-merge/needs-linked-issue labels Aug 8, 2022

lidezhu force-pushed the support-config-blob-file-limit-size branch 3 times, most recently from 1553a85 to 5be02c3 Compare August 8, 2022 08:03

ti-chi-bot added the do-not-merge/needs-triage-completed label Aug 8, 2022

ti-chi-bot removed the do-not-merge/needs-triage-completed label Aug 8, 2022

lidezhu force-pushed the support-config-blob-file-limit-size branch from 5be02c3 to 3f607fa Compare August 8, 2022 09:58

flowbehappy reviewed Aug 9, 2022

View reviewed changes

dbms/src/Storages/Page/V3/BlobStore.cpp Outdated Show resolved Hide resolved

support config blob file limit size

adc7b4c

lidezhu force-pushed the support-config-blob-file-limit-size branch from e1cfb17 to adc7b4c Compare August 11, 2022 10:02

add some comment

f55775a

lidezhu force-pushed the support-config-blob-file-limit-size branch from b2680db to f55775a Compare August 11, 2022 10:26

lidezhu changed the title ~~[WIP] support config blob file limit size~~ remove big blob file in ps v3 Aug 11, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 11, 2022

lidezhu requested a review from JaySon-Huang August 11, 2022 10:36

lidezhu changed the title ~~remove big blob file in ps v3~~ remove big blob file type in ps v3 Aug 11, 2022

lidezhu force-pushed the support-config-blob-file-limit-size branch from 79d951b to b374c8e Compare August 23, 2022 07:26

add metric

125bbe3

lidezhu force-pushed the support-config-blob-file-limit-size branch from b374c8e to 125bbe3 Compare August 23, 2022 07:36

lidezhu and others added 4 commits August 23, 2022 15:40

Update dbms/src/Storages/Page/V3/BlobStore.cpp

427a761

Co-authored-by: JaySon <tshent@qq.com>

small fix for gtest

1dcf34c

Merge branch 'support-config-blob-file-limit-size' of github.com:lide…

07ec87e

…zhu/tics into support-config-blob-file-limit-size

Merge branch 'master' into support-config-blob-file-limit-size

fd1294b

JaySon-Huang reviewed Aug 23, 2022

View reviewed changes

metrics/grafana/tiflash_summary.json Outdated Show resolved Hide resolved

dbms/src/Common/TiFlashMetrics.h Outdated Show resolved Hide resolved

fix unit test

cbb34ec

lidezhu force-pushed the support-config-blob-file-limit-size branch from bf88d31 to cbb34ec Compare August 23, 2022 09:33

JaySon-Huang reviewed Aug 23, 2022

View reviewed changes

dbms/src/Storages/Page/V3/tests/gtest_blob_store.cpp Show resolved Hide resolved

lidezhu added 2 commits August 23, 2022 18:06

fix metrics

aa6c773

add some comment about reloading config.file_limit_size

ecf39fd

JaySon-Huang approved these changes Aug 23, 2022

View reviewed changes

metrics/grafana/tiflash_summary.json Outdated Show resolved Hide resolved

metrics/grafana/tiflash_summary.json Outdated Show resolved Hide resolved

lidezhu and others added 2 commits August 23, 2022 20:33

Update metrics/grafana/tiflash_summary.json

60d216d

Co-authored-by: JaySon <tshent@qq.com>

Update metrics/grafana/tiflash_summary.json

4f81d68

Co-authored-by: JaySon <tshent@qq.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 23, 2022

Merge branch 'master' into support-config-blob-file-limit-size

119a293

lidezhu merged commit c9656c9 into pingcap:master Aug 23, 2022

lidezhu deleted the support-config-blob-file-limit-size branch August 23, 2022 13:32

lidezhu mentioned this pull request Aug 24, 2022

fix wrong alloc size when do heavy gc for blob file #5693

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove big blob file type in ps v3 #5544

remove big blob file type in ps v3 #5544

lidezhu commented Aug 5, 2022 •

edited

Loading

ti-chi-bot commented Aug 5, 2022 •

edited

Loading

flowbehappy Aug 5, 2022

lidezhu Aug 8, 2022 •

edited

Loading

lidezhu commented Aug 8, 2022

lidezhu commented Aug 8, 2022

sre-bot commented Aug 8, 2022

lidezhu commented Aug 23, 2022

lidezhu commented Aug 23, 2022

lidezhu commented Aug 23, 2022

sre-bot commented Aug 23, 2022

JaySon-Huang left a comment •

edited

Loading

lidezhu commented Aug 23, 2022

ti-chi-bot commented Aug 23, 2022

ti-chi-bot commented Aug 23, 2022

lidezhu commented Aug 23, 2022

sre-bot commented Aug 23, 2022

		@@ -43,9 +42,6 @@ SpaceMapPtr SpaceMap::createSpaceMap(SpaceMapType type, UInt64 start, UInt64 end
		case SMAP64_STD_MAP:

remove big blob file type in ps v3 #5544

remove big blob file type in ps v3 #5544

Conversation

lidezhu commented Aug 5, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Aug 5, 2022 • edited Loading

flowbehappy Aug 5, 2022

Choose a reason for hiding this comment

lidezhu Aug 8, 2022 • edited Loading

Choose a reason for hiding this comment

lidezhu commented Aug 8, 2022

lidezhu commented Aug 8, 2022

sre-bot commented Aug 8, 2022

Coverage for changed files

Coverage summary

lidezhu commented Aug 23, 2022

lidezhu commented Aug 23, 2022

lidezhu commented Aug 23, 2022

sre-bot commented Aug 23, 2022

Coverage for changed files

Coverage summary

JaySon-Huang left a comment • edited Loading

Choose a reason for hiding this comment

lidezhu commented Aug 23, 2022

ti-chi-bot commented Aug 23, 2022

ti-chi-bot commented Aug 23, 2022

lidezhu commented Aug 23, 2022

sre-bot commented Aug 23, 2022

Coverage for changed files

Coverage summary

lidezhu commented Aug 5, 2022 •

edited

Loading

ti-chi-bot commented Aug 5, 2022 •

edited

Loading

lidezhu Aug 8, 2022 •

edited

Loading

JaySon-Huang left a comment •

edited

Loading