Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create StoreUpdate for one cold column (Block) #7744 #7745

Merged
merged 23 commits into from
Oct 25, 2022
Merged

Conversation

posvyatokum
Copy link
Member

@posvyatokum posvyatokum commented Sep 30, 2022

This is minimum working example of the first part of online copying
data from hot storage to cold.

The main function is StoreUpdate::add_cold_update. It needs to add
all the changes made to cold columns in one block to StoreUpdate. It
is done by finding all the relevant keys for every cold column and
inserting value for that key and column to update.

Because many columns have similar keys I introduce DBKeyType enum
which is meant to be semantical description of different key parts.
The main logical part related to DBKeyType is DBCol::key_type function
that is supposed to give a vector of key types that make a key for
a given column. And I to the best of my abilities define it for every
non-deprecated column. Although I really only need it for cold
columns. So, if nobody wants to check 50 key types I can define it
only for cold columns
.

The point of get_keys_from_store() is to get all possible keys for
every DBKeyType needed for cold storage. It is intentionally only
defines only BlockHash right now, because the only cold column right
now is Block. It is ⋆。°✩ minimal ✩‧₊˚ example.

copy_from_store() inserts update for given (col, key) into update.
I assume I wrote something dumb there, but it works, but I didn't give
it much thought.

Finally, I add simple cache to Store here, because during
get_keys_from_store we will read from a lot of cold columns that will
need to be later read again for inserting actual data into update.

I plan to add columns to DBCol::is_cold in next PRs without changing
this architecture much. The main change will happen in
get_keys_from_store.

@posvyatokum posvyatokum requested a review from a team as a code owner September 30, 2022 19:07
core/store/src/lib.rs Outdated Show resolved Hide resolved
core/store/src/columns.rs Outdated Show resolved Hide resolved
core/store/src/cold_storage.rs Outdated Show resolved Hide resolved
core/store/src/columns.rs Outdated Show resolved Hide resolved
Comment on lines 258 to 261
#[derive(PartialEq, Copy, Clone, Debug, Hash, Eq, strum::EnumIter)]
pub enum DBKeyType {
Empty,
Literal,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs more documentation. Especially what Empty and Literal mean.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried. Don't know if that helps.

core/store/src/cold_storage.rs Show resolved Hide resolved
core/store/Cargo.toml Outdated Show resolved Hide resolved
core/store/Cargo.toml Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants