-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add checkpoint utilities #6900
Add checkpoint utilities #6900
Conversation
Signed-off-by: Wish <breezewish@outlook.com>
Co-authored-by: JaySon <tshent@qq.com>
…of github.com:breezewish/tiflash into wenxuan/p1/decouple_cu
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
…oint Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
|
||
*/ | ||
|
||
message ManifestFilePrefix { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need to add some fields about the compress algorithm so that we can change the algorithm later?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we can add some settings there when we want to change compression algorithm.
auto edits = universal::PageEntriesEdit{}; | ||
{ | ||
UniversalWriteBatch wb; | ||
wb.putPage("page_foo", 0, "The flower carriage rocked"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about add a page with data_sizes
, and make sure the fields_offset
is correct from manifest file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Please take a look again, thanks!
…oint Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
/run-all-tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/rebuild |
/run-integration-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/merge |
@breezewish: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger If you have any questions about the PR merge process, please refer to pr process. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
This pull request has been accepted and is ready to merge. Commit hash: a67fafc
|
What problem does this PR solve?
Issue Number: ref #6827
Problem Summary:
What is changed and how it works?
This PR introduces these utilities to work with writing and reading PageStorage checkpoints:
Usually, you need to write ManifestFile and DataFile via
CPFilesWriter
, and then, read the ManifestFile viaCPManifestFileReader
and read the DataFile via ReadBufferFromFile or something similar.The manifest file is compressed and the data file is not. Manifest file contains Protobuf data and the file format is designed to be friendly for streaming, i.e. you can read / write data piece by piece (instead of all at once) to avoid OOMs.
Check List
Tests
Side effects
Documentation
Release note