Tiflash pagectl support encrypted data #5003

hehechen · 2022-05-26T03:11:59Z

What problem does this PR solve?

Issue Number: close #4962

Problem Summary:

What is changed and how it works?

Make page-ctl (including V2 and V3) a subcommand of tiflash binary, tiflash pagectl -V 2/3
Make page-ctl V3 can dump the encrypted data, add two options: imitative and config_file_path

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: hehechen <awd123456sss@gmail.com>

ti-chi-bot · 2022-05-26T03:12:00Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen · 2022-05-30T03:52:36Z

/run-all-tests

dbms/src/Storages/Page/tools/page_ctl/page_storage_ctl_v3.cpp

dbms/src/Storages/Page/V2/CMakeLists.txt

dbms/src/Storages/Page/tools/page_ctl/page_storage_ctl_v3.cpp

jiaqizho · 2022-05-30T10:59:23Z

please format this PR, put the pics into right position.

hehechen · 2022-05-31T07:11:06Z

/run-unit-test

sre-bot · 2022-05-31T07:43:44Z

Coverage for changed files

Filename                               Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/CLIService.h                         37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Server/DTTool/DTTool.h                       5                 5     0.00%           5                 5     0.00%          44                44     0.00%           0                 0         -
Storages/Page/V3/BlobStore.h                10                 2    80.00%          10                 2    80.00%          46                23    50.00%           0                 0         -
Storages/Page/V3/PageDirectory.h            25                 3    88.00%          25                 3    88.00%         104                10    90.38%           0                 0         -
Storages/Page/V3/PageStorageImpl.h           8                 2    75.00%           8                 2    75.00%          22                 2    90.91%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                       85                49    42.35%          55                19    65.45%         325               188    42.15%          24                24     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18220      9784             46.30%    203962  97976        51.96%

full coverage report (for internal network access only)

Signed-off-by: hehechen <awd123456sss@gmail.com>

dbms/src/Storages/Page/CMakeLists.txt

dbms/src/Storages/Page/tools/page_ctl/MainEntry.cpp

hehechen · 2022-06-01T04:08:31Z

/rebuild

dbms/src/Storages/Page/tools/PageCtl/PageStorageCtlV3.cpp

JaySon-Huang

LGTM with small comment

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen · 2022-06-01T10:24:49Z

/merge

ti-chi-bot · 2022-06-01T10:24:50Z

@hehechen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-01T10:24:51Z

This pull request has been accepted and is ready to merge.

Commit hash: 66562c3

sre-bot · 2022-06-01T11:07:14Z

Coverage for changed files

Filename                               Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/CLIService.h                         37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Server/DTTool/DTTool.h                       5                 5     0.00%           5                 5     0.00%          44                44     0.00%           0                 0         -
Storages/Page/V3/BlobStore.h                12                 2    83.33%          10                 2    80.00%          50                23    54.00%           2                 0   100.00%
Storages/Page/V3/PageDirectory.h            25                 3    88.00%          25                 3    88.00%         104                10    90.38%           0                 0         -
Storages/Page/V3/PageStorageImpl.h           8                 2    75.00%           8                 2    75.00%          22                 2    90.91%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                       87                49    43.68%          55                19    65.45%         329               188    42.86%          26                24     7.69%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18282      9779             46.51%    204998  97995        52.20%

full coverage report (for internal network access only)

sre-bot · 2022-06-01T12:12:15Z

Coverage for changed files

Filename                               Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/CLIService.h                         37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Server/DTTool/DTTool.h                       5                 5     0.00%           5                 5     0.00%          44                44     0.00%           0                 0         -
Storages/Page/V3/BlobStore.h                12                 2    83.33%          10                 2    80.00%          50                23    54.00%           2                 0   100.00%
Storages/Page/V3/PageDirectory.h            25                 3    88.00%          25                 3    88.00%         104                10    90.38%           0                 0         -
Storages/Page/V3/PageStorageImpl.h           8                 2    75.00%           8                 2    75.00%          22                 2    90.91%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                       87                49    43.68%          55                19    65.45%         329               188    42.86%          26                24     7.69%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18282      9734             46.76%    204967  97603        52.38%

full coverage report (for internal network access only)

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen · 2022-06-01T12:47:12Z

/merge

ti-chi-bot · 2022-06-01T12:47:14Z

@hehechen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-01T12:47:16Z

This pull request has been accepted and is ready to merge.

Commit hash: c7dc2e6

sre-bot · 2022-06-01T13:05:30Z

Coverage for changed files

Filename                               Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/CLIService.h                         37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Server/DTTool/DTTool.h                       5                 5     0.00%           5                 5     0.00%          44                44     0.00%           0                 0         -
Storages/Page/V3/BlobStore.h                12                 2    83.33%          10                 2    80.00%          50                23    54.00%           2                 0   100.00%
Storages/Page/V3/PageDirectory.h            25                 3    88.00%          25                 3    88.00%         104                10    90.38%           0                 0         -
Storages/Page/V3/PageStorageImpl.h           8                 2    75.00%           8                 2    75.00%          22                 2    90.91%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                       87                49    43.68%          55                19    65.45%         329               188    42.86%          26                24     7.69%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18282      9732             46.77%    204967  97580        52.39%

full coverage report (for internal network access only)

hehechen added 2 commits May 24, 2022 23:38

add tiflash page-ctl

8cf6c44

Signed-off-by: hehechen <awd123456sss@gmail.com>

add page_ctl into tiflash binary

f868d56

Signed-off-by: hehechen <awd123456sss@gmail.com>

ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 26, 2022

fix license

9c92ce4

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen changed the title ~~[WIP] Tiflash pagectl support encrypted data~~ Tiflash pagectl support encrypted data May 27, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 27, 2022

hehechen requested review from jiaqizho and JaySon-Huang May 27, 2022 03:01

Update MainEntry.cpp

973143c

hehechen requested a review from lidezhu May 30, 2022 03:51

JaySon-Huang reviewed May 30, 2022

View reviewed changes

dbms/src/Storages/Page/tools/page_ctl/page_storage_ctl_v3.cpp Outdated Show resolved Hide resolved

dbms/src/Storages/Page/V2/CMakeLists.txt Show resolved Hide resolved

hehechen mentioned this pull request May 30, 2022

Remove the deprecated target page_ctl PingCAP-QE/ci#1049

Merged

hehechen force-pushed the tiflash_pagectl branch from 3357d18 to 973143c Compare May 30, 2022 08:56

jiaqizho reviewed May 30, 2022

View reviewed changes

dbms/src/Storages/Page/tools/page_ctl/page_storage_ctl_v3.cpp Outdated Show resolved Hide resolved

add try catch

0885bbd

Signed-off-by: hehechen <awd123456sss@gmail.com>

JaySon-Huang reviewed Jun 1, 2022

View reviewed changes

dbms/src/Storages/Page/CMakeLists.txt Show resolved Hide resolved

JaySon-Huang reviewed Jun 1, 2022

View reviewed changes

dbms/src/Storages/Page/tools/page_ctl/MainEntry.cpp Outdated Show resolved Hide resolved

JaySon-Huang reviewed Jun 1, 2022

View reviewed changes

dbms/src/Storages/Page/tools/PageCtl/PageStorageCtlV3.cpp Outdated Show resolved Hide resolved

JaySon-Huang approved these changes Jun 1, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 1, 2022

hehechen force-pushed the tiflash_pagectl branch from b75df32 to a0a9f2c Compare June 1, 2022 06:46

lidezhu approved these changes Jun 1, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 1, 2022

rename page-ctl

66562c3

Signed-off-by: hehechen <awd123456sss@gmail.com>

hehechen force-pushed the tiflash_pagectl branch from a0a9f2c to 66562c3 Compare June 1, 2022 10:24

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2022

Merge branch 'master' into tiflash_pagectl

2d76fef

Merge branch 'master' into tiflash_pagectl

09dc174

fix assert

c7dc2e6

Signed-off-by: hehechen <awd123456sss@gmail.com>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2022

ti-chi-bot merged commit 187a591 into pingcap:master Jun 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tiflash pagectl support encrypted data #5003

Tiflash pagectl support encrypted data #5003

hehechen commented May 26, 2022 •

edited

Loading

ti-chi-bot commented May 26, 2022 •

edited

Loading

hehechen commented May 30, 2022

jiaqizho commented May 30, 2022

hehechen commented May 31, 2022

sre-bot commented May 31, 2022

hehechen commented Jun 1, 2022

JaySon-Huang left a comment

hehechen commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

sre-bot commented Jun 1, 2022

sre-bot commented Jun 1, 2022

hehechen commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

sre-bot commented Jun 1, 2022

Tiflash pagectl support encrypted data #5003

Tiflash pagectl support encrypted data #5003

Conversation

hehechen commented May 26, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 26, 2022 • edited Loading

hehechen commented May 30, 2022

jiaqizho commented May 30, 2022

hehechen commented May 31, 2022

sre-bot commented May 31, 2022

Coverage for changed files

Coverage summary

hehechen commented Jun 1, 2022

JaySon-Huang left a comment

Choose a reason for hiding this comment

hehechen commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

sre-bot commented Jun 1, 2022

Coverage for changed files

Coverage summary

sre-bot commented Jun 1, 2022

Coverage for changed files

Coverage summary

hehechen commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

ti-chi-bot commented Jun 1, 2022

sre-bot commented Jun 1, 2022

Coverage for changed files

Coverage summary

hehechen commented May 26, 2022 •

edited

Loading

ti-chi-bot commented May 26, 2022 •

edited

Loading