*: Fix bug that DynamicThreadPool may not destruct tasks in time. #4210

fuzhe1989 · 2022-03-09T12:00:38Z

Signed-off-by: fuzhe1989 fuzhe1989@gmail.com

What problem does this PR solve?

Issue Number: ref #4098, ref #4207

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Fix bug that enable elastic thread pool may introduce memory leak.

Signed-off-by: fuzhe1989 <fuzhe1989@gmail.com>

ti-chi-bot · 2022-03-09T12:00:39Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

bestwoody
windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

bestwoody

LGTM

windtalker

LGTM

Signed-off-by: fuzhe1989 <fuzhe1989@gmail.com>

fuzhe1989 · 2022-03-09T16:00:10Z

/merge

ti-chi-bot · 2022-03-09T16:00:11Z

@fuzhe1989: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-09T16:00:17Z

This pull request has been accepted and is ready to merge.

Commit hash: d5a8280

Signed-off-by: fuzhe1989 <fuzhe1989@gmail.com>

fuzhe1989 · 2022-03-09T16:05:12Z

/merge

ti-chi-bot · 2022-03-09T16:05:12Z

@fuzhe1989: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-09T16:05:19Z

This pull request has been accepted and is ready to merge.

Commit hash: d323713

sre-bot · 2022-03-09T16:14:51Z

Coverage for changed files

Filename                                Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DynamicThreadPool.cpp                        79                 1    98.73%          10                 0   100.00%          96                 1    98.96%          24                 1    95.83%
DynamicThreadPool.h                           2                 0   100.00%           2                 0   100.00%           9                 0   100.00%           0                 0         -
tests/gtest_dynamic_thread_pool.cpp         625               191    69.44%          19                 0   100.00%         152                 3    98.03%         172                91    47.09%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                       706               192    72.80%          31                 0   100.00%         257                 4    98.44%         196                92    53.06%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16782      9447             43.71%    188932  95732        49.33%

full coverage report (for internal network access only)

fuzhe1989 · 2022-03-10T01:56:36Z

/run-integration-test

fzhedu

LGTM

fzhedu · 2022-03-10T05:08:48Z

/run-integration-tests

fuzhe1989 · 2022-03-10T06:14:39Z

/run-integration-tests

fuzhe1989 · 2022-03-10T06:52:10Z

/run-integration-tests

fuzhe1989 · 2022-03-10T10:48:56Z

/run-integration-tests

fuzhe1989 · 2022-03-11T00:15:48Z

/run-integration-tests

fuzhe1989 · 2022-03-11T04:26:50Z

/run-integration-tests

fuzhe1989 · 2022-03-11T09:01:57Z

/rebuild

fuzhe1989 · 2022-03-11T09:02:41Z

/run-all-tests

sre-bot · 2022-03-11T09:18:23Z

Coverage for changed files

no c/c++ source change detected

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16786      9444             43.74%    189100  95751        49.36%

full coverage report (for internal network access only)

ti-chi-bot · 2022-03-11T10:28:20Z

@fuzhe1989: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-03-11T10:43:56Z

Coverage for changed files

Filename                                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/DynamicThreadPool.cpp                                        79                 1    98.73%          10                 0   100.00%          96                 1    98.96%          24                 1    95.83%
Common/DynamicThreadPool.h                                           2                 0   100.00%           2                 0   100.00%           9                 0   100.00%           0                 0         -
Common/MPMCQueue.h                                                  76                 0   100.00%          29                 0   100.00%         166                 1    99.40%          44                 2    95.45%
Common/tests/gtest_dynamic_thread_pool.cpp                         625               191    69.44%          19                 0   100.00%         152                 3    98.03%         172                91    47.09%
Common/tests/gtest_mpmc_queue.cpp                                 1424               292    79.49%          45                 0   100.00%         514                12    97.67%         480               218    54.58%
Debug/DBGInvoker.cpp                                                30                29     3.33%          10                 9    10.00%         138                75    45.65%          14                14     0.00%
Debug/MockTiDB.cpp                                                 139               135     2.88%          34                32     5.88%         421               403     4.28%          92                91     1.09%
Debug/MockTiDB.h                                                     8                 6    25.00%           8                 6    25.00%          15                13    13.33%           0                 0         -
Debug/dbgFuncMockTiDBTable.cpp                                     109               109     0.00%          14                14     0.00%         219               219     0.00%          72                72     0.00%
Debug/dbgFuncSchema.cpp                                             44                44     0.00%           5                 5     0.00%          62                62     0.00%          28                28     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp                        272               272     0.00%          15                15     0.00%         499               499     0.00%         170               170     0.00%
Storages/DeltaMerge/DMSegmentThreadInputStream.h                    62                 6    90.32%           5                 0   100.00%          73                 2    97.26%          28                 8    71.43%
Storages/DeltaMerge/DeltaMergeDefines.h                             26                 0   100.00%           8                 0   100.00%          28                 0   100.00%           2                 0   100.00%
Storages/DeltaMerge/DeltaMergeStore.cpp                           1317               475    63.93%          65                 5    92.31%        1885               430    77.19%         766               369    51.83%
Storages/DeltaMerge/DeltaMergeStore.h                               41                13    68.29%          19                 2    89.47%          85                30    64.71%          42                11    73.81%
Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp        1489               285    80.86%          14                 0   100.00%         540                 8    98.52%         472               223    52.75%
Storages/ITableDeclaration.cpp                                      92                87     5.43%          18                15    16.67%         182               172     5.49%          68                67     1.47%
Storages/ITableDeclaration.h                                         3                 1    66.67%           3                 1    66.67%           5                 3    40.00%           0                 0         -
Storages/MutableSupport.h                                           17                17     0.00%          10                10     0.00%          37                37     0.00%           6                 6     0.00%
Storages/PrimaryKeyNotMatchException.h                               1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Storages/StorageDeltaMerge.cpp                                     667               358    46.33%          56                29    48.21%        1290               762    40.93%         376               257    31.65%
Storages/StorageDeltaMerge.h                                        11                 7    36.36%          11                 7    36.36%          15                 9    40.00%           0                 0         -
Storages/Transaction/SchemaBuilder.cpp                             651               616     5.38%          46                42     8.70%        1020               948     7.06%         404               386     4.46%
Storages/Transaction/SchemaBuilder.h                                 1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Storages/Transaction/SchemaGetter.h                                  1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                             7187              2947    59.00%         449               195    56.57%        7454              3692    50.47%        3260              2014    38.22%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16790      9444             43.75%    189130  95751        49.37%

full coverage report (for internal network access only)

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2022-03-11T11:00:28Z

In response to a cherrypick label: new pull request created: #4231.

) (#4231) ref #4098, ref #4207

update

8d0005b

Signed-off-by: fuzhe1989 <fuzhe1989@gmail.com>

fuzhe1989 requested review from bestwoody, windtalker and fzhedu March 9, 2022 12:01

ti-chi-bot added the do-not-merge/needs-triage-completed label Mar 9, 2022

bestwoody approved these changes Mar 9, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 9, 2022

windtalker approved these changes Mar 9, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 9, 2022

fuzhe1989 added 2 commits March 9, 2022 23:36

Update DynamicThreadPool.cpp

66468a7

update

d5a8280

Signed-off-by: fuzhe1989 <fuzhe1989@gmail.com>

ti-chi-bot removed the do-not-merge/needs-triage-completed label Mar 9, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 9, 2022

update

d323713

Signed-off-by: fuzhe1989 <fuzhe1989@gmail.com>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 9, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 9, 2022

fzhedu approved these changes Mar 10, 2022

View reviewed changes

ti-chi-bot added 2 commits March 11, 2022 16:08

Merge branch 'master' into fuzhe/fix_dynamic_thread_pool

37ed2f3

Merge branch 'master' into fuzhe/fix_dynamic_thread_pool

e4c8ec1

Merge branch 'master' into fuzhe/fix_dynamic_thread_pool

db52c75

ti-chi-bot merged commit 86e21ef into pingcap:master Mar 11, 2022

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Mar 11, 2022

This is an automated cherry-pick of pingcap#4210

881f568

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Mar 11, 2022

*: Fix bug that DynamicThreadPool may not destruct tasks in time. (#4210) #4231

Merged

12 tasks

fuzhe1989 deleted the fuzhe/fix_dynamic_thread_pool branch April 1, 2022 05:41

ti-chi-bot added a commit that referenced this pull request Apr 22, 2022

*: Fix bug that DynamicThreadPool may not destruct tasks in time. (#4210

bb5f9f2

) (#4231) ref #4098, ref #4207

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Fix bug that DynamicThreadPool may not destruct tasks in time. #4210

*: Fix bug that DynamicThreadPool may not destruct tasks in time. #4210

fuzhe1989 commented Mar 9, 2022 •

edited

Loading

ti-chi-bot commented Mar 9, 2022 •

edited

Loading

bestwoody left a comment

windtalker left a comment

fuzhe1989 commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

fuzhe1989 commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

sre-bot commented Mar 9, 2022

fuzhe1989 commented Mar 10, 2022

fzhedu left a comment

fzhedu commented Mar 10, 2022

fuzhe1989 commented Mar 10, 2022

fuzhe1989 commented Mar 10, 2022

fuzhe1989 commented Mar 10, 2022

fuzhe1989 commented Mar 11, 2022

fuzhe1989 commented Mar 11, 2022

fuzhe1989 commented Mar 11, 2022

fuzhe1989 commented Mar 11, 2022

sre-bot commented Mar 11, 2022

ti-chi-bot commented Mar 11, 2022

sre-bot commented Mar 11, 2022

ti-chi-bot commented Mar 11, 2022

*: Fix bug that DynamicThreadPool may not destruct tasks in time. #4210

*: Fix bug that DynamicThreadPool may not destruct tasks in time. #4210

Conversation

fuzhe1989 commented Mar 9, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 9, 2022 • edited Loading

bestwoody left a comment

Choose a reason for hiding this comment

windtalker left a comment

Choose a reason for hiding this comment

fuzhe1989 commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

fuzhe1989 commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

ti-chi-bot commented Mar 9, 2022

sre-bot commented Mar 9, 2022

Coverage for changed files

Coverage summary

fuzhe1989 commented Mar 10, 2022

fzhedu left a comment

Choose a reason for hiding this comment

fzhedu commented Mar 10, 2022

fuzhe1989 commented Mar 10, 2022

fuzhe1989 commented Mar 10, 2022

fuzhe1989 commented Mar 10, 2022

fuzhe1989 commented Mar 11, 2022

fuzhe1989 commented Mar 11, 2022

fuzhe1989 commented Mar 11, 2022

fuzhe1989 commented Mar 11, 2022

sre-bot commented Mar 11, 2022

Coverage for changed files

Coverage summary

ti-chi-bot commented Mar 11, 2022

sre-bot commented Mar 11, 2022

Coverage for changed files

Coverage summary

ti-chi-bot commented Mar 11, 2022

fuzhe1989 commented Mar 9, 2022 •

edited

Loading

ti-chi-bot commented Mar 9, 2022 •

edited

Loading