-
Notifications
You must be signed in to change notification settings - Fork 619
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create StoreUpdate for one cold column (Block) #7744 #7745
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mina86
reviewed
Oct 1, 2022
core/store/src/columns.rs
Outdated
Comment on lines
258
to
261
#[derive(PartialEq, Copy, Clone, Debug, Hash, Eq, strum::EnumIter)] | ||
pub enum DBKeyType { | ||
Empty, | ||
Literal, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs more documentation. Especially what Empty and Literal mean.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried. Don't know if that helps.
mina86
reviewed
Oct 11, 2022
mina86
reviewed
Oct 11, 2022
mina86
reviewed
Oct 19, 2022
Co-authored-by: Michal Nazarewicz <mina86@mina86.com>
mina86
approved these changes
Oct 25, 2022
nikurt
pushed a commit
that referenced
this pull request
Oct 25, 2022
nikurt
pushed a commit
that referenced
this pull request
Nov 7, 2022
nikurt
pushed a commit
that referenced
this pull request
Nov 9, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is minimum working example of the first part of online copying
data from hot storage to cold.
The main function is StoreUpdate::add_cold_update. It needs to add
all the changes made to cold columns in one block to StoreUpdate. It
is done by finding all the relevant keys for every cold column and
inserting value for that key and column to update.
Because many columns have similar keys I introduce DBKeyType enum
which is meant to be semantical description of different key parts.
The main logical part related to DBKeyType is DBCol::key_type function
that is supposed to give a vector of key types that make a key for
a given column. And I to the best of my abilities define it for every
non-deprecated column. Although I really only need it for cold
columns. So, if nobody wants to check 50 key types I can define it
only for cold columns.
The point of get_keys_from_store() is to get all possible keys for
every DBKeyType needed for cold storage. It is intentionally only
defines only BlockHash right now, because the only cold column right
now is Block. It is ⋆。°✩ minimal ✩‧₊˚ example.
copy_from_store() inserts update for given (col, key) into update.
I assume I wrote something dumb there, but it works, but I didn't give
it much thought.
Finally, I add simple cache to Store here, because during
get_keys_from_store we will read from a lot of cold columns that will
need to be later read again for inserting actual data into update.
I plan to add columns to DBCol::is_cold in next PRs without changing
this architecture much. The main change will happen in
get_keys_from_store.