Add a heavy restore to test pagestorage restore speed #4896

jiaqizho · 2022-05-16T07:02:11Z

What problem does this PR solve?

Issue Number: ref #3594

Problem Summary:

What is changed and how it works?

add a test in page_stress named HeavyRestore to test the restore time.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

close #4072

close #3956

ref #4141

close #4151

ref #3594

ref #2843

…on levels (#4161) close #3671

* add threads metrics * add threads metrics Signed-off-by: bestwoody <bestwoody@163.com> * format Signed-off-by: bestwoody <bestwoody@163.com> * update threadpool into thread_pool Signed-off-by: bestwoody <bestwoody@163.com> * UPDATE_CUR_AND_MAX_METRIC Signed-off-by: bestwoody <bestwoody@163.com> * refine include Signed-off-by: bestwoody <bestwoody@163.com>

close #3358

close #4178

close #4179

ref #3594

close #4188

ref #4118

close #4185

ref #3594, close #4017

close #4167

ref #4129

close #1863

close #4198

close #4194

close #4180

…4221) close #4098

) ref #4098, ref #4207

ref #3594

close #3721

… local tunnel is enabled. (#4219) close #4229

close #4567

close #4569

ref #4860

…d should both be empty or neither should be empty` (#4864) close #4865

ref #3594

close #4868

ref #4860

close #4872

ref #4860

ref #4862

ref #4618

close #4794

ref #3594

ti-chi-bot · 2022-05-16T07:02:12Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

hehechen · 2022-05-17T09:25:36Z

dbms/src/Storages/Page/stress/workload/HeavyRestore.cpp

+
+        restore_stop_watch.start();
+        // restore
+        initPageStorage(config, name());


I think the restore time should not include the traverse time.

tiflash/dbms/src/Storages/Page/stress/PSWorkload.cpp

Line 104 in 6b8ca3d

ps->traverse([&num_of_pages](const DB::Page & page) {

ti-chi-bot · 2022-05-26T05:43:42Z

@jiaqizho: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jiaqizho · 2022-05-26T05:50:38Z

i don't know what happend...

lidezhu and others added 30 commits February 25, 2022 17:45

Fix crash when prehandle snapshot (#4140)

9ac90e6

close #4072

Support regexp in TiFlash (#3957)

c2651f9

close #3956

Clean useless processes and scripts (#4153)

6d7f8e0

ref #4141

Fix cast datetime to decimal wrong result bug (#4152)

f72d8a5

close #4151

separate PageIdGenerator from StoragePool (#4154)

8b6efc3

ref #3594

Refactor WAL log writer && IO limiter (#4107)

60e21d4

ref #3594

PageStorage: Fine grained lock on mvcc map (#4137)

d9f9587

ref #3594

clean clang-tidy error from FlashService (#4170)

faafb9e

ref #2843

Support user to choose different compression algorithms and compressi…

867fb5f

…on levels (#4161) close #3671

expression: support least/greatest for int/real. (#3801)

76ec0a2

close #3358

add more case in bench_function_cast.cpp (#4109)

5a93f02

close #4178

Add a protect if snapshot can't get any valid version (#4171)

99f6141

close #4179

PageStorage: Fix BlobStore read with FieldReadInfos (#4181)

063b5fa

ref #3594

Add notice according to Apache license 2.0 4d (#4182)

294c3f2

close #4188

refine appendAggregation in analyzer (#4042)

3c1acf8

ref #4118

fix bug from pull request 4042 (#4190)

616a907

ref #4118

fix the problem of not stopping a failed flush correctly (#4187)

7ed9828

close #4185

PageStorage: Ref page lifetime mechanism (#4174)

787db86

ref #3594, close #4017

Fix small bugs found by asan (#4168)

8ff66d7

close #4167

metrics: support multi-k8s in grafana dashboards (#4099)

6e84e58

ref #4129

expr: support IsTrue/IsTrueWithNull/IsFalse/IsFalseWithNull (#4184)

0293d9e

close #1863

*: ensure all futures finished in waitTasks (#4199)

d133255

close #4198

Support ddl job CreateTables in TiFlash (#4201)

5411079

close #4194

TiFlash support TiDB push down ExtraPhysTblID column (#4203)

a91227a

close #4180

*: Fix memory leak when MPMCQueue destruct with non-poped elements (#…

b1050b7

…4221) close #4098

*: Fix bug that DynamicThreadPool may not destruct tasks in time. (#4210

d01f744

) ref #4098, ref #4207

Add namespace id to PageStorage interface (#4175)

b9d5e1a

ref #3594

Optimize overflow check when cast to decimal (#3912)

bdbb011

close #3721

Fix the bug that canceled MPP query may cause tasks hang forever when…

0aff40c

… local tunnel is enabled. (#4219) close #4229

ywqzzy and others added 15 commits May 11, 2022 18:00

Interpreter: Remove kill Query (#4564)

f1d5991

close #4567

Interpreter: Remove CheckQuery. (#4568)

8bef777

close #4569

Interpreter: Remove Truncate query (#4587)

8d5eff8

ref #4860

fix tispark query err: `for PushDownFilter, conditions and executor_i…

1fedb64

…d should both be empty or neither should be empty` (#4864) close #4865

PageStorage: Add mix mode (#4726)

f7324a0

ref #3594

Add fail point pause_after_copr_streams_acquired_once (#4833)

ed7d6ed

close #4868

Interpreter: Remove Table properties query (#4586)

6351eff

ref #4860

Interpreter: Remove delete query (#4594)

22594e9

ref #4860

restore global storage pool in test cases (#4878)

bd0cd64

close #4872

Interpreter: Remove system query (#4590)

ebb7673

ref #4860

misc: Add comments (#4863)

1a02963

ref #4862

update pprof for proxy (#4871)

2ec5bac

ref #4618

add DB config max_threads by grpc context (#4795)

5a8b7be

close #4794

Add a force transfrom meta from v2 to v3 (#4873)

80980ed

ref #3594

Add a heavy restore to test pagestorage restore speed

e1b8221

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 16, 2022

jiaqizho requested review from flowbehappy, JaySon-Huang, hehechen and lidezhu May 17, 2022 03:42

hehechen reviewed May 17, 2022

View reviewed changes

hehechen mentioned this pull request May 18, 2022

PageStorage V3 restore takes longer than V2 #4914

Closed

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 26, 2022

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 26, 2022

jiaqizho closed this May 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a heavy restore to test pagestorage restore speed #4896

Add a heavy restore to test pagestorage restore speed #4896

jiaqizho commented May 16, 2022

ti-chi-bot commented May 16, 2022

hehechen May 17, 2022

ti-chi-bot commented May 26, 2022

jiaqizho commented May 26, 2022

Add a heavy restore to test pagestorage restore speed #4896

Add a heavy restore to test pagestorage restore speed #4896

Conversation

jiaqizho commented May 16, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 16, 2022

hehechen May 17, 2022

Choose a reason for hiding this comment

ti-chi-bot commented May 26, 2022

jiaqizho commented May 26, 2022