Restore indexes from backup with the original partitioning #7589
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
jepett0
force-pushed
the
IndexBackupRestore.1
branch
from
7abf265 to
781ca4f
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
jepett0
commented
Aug 9, 2024
MBkkt
previously approved these changes
Aug 9, 2024
jepett0
force-pushed
the
IndexBackupRestore.1
branch
from
781ca4f to
0196b02
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
jepett0
force-pushed
the
IndexBackupRestore.1
branch
from
0196b02 to
33900bb
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
+ Add a dedicated option to control inclusion of the indexImplTable boundaries in the main table description. This should help minimizing the network traffic and IO CPU pool usage.
These are needed for a good looking test. However, these functions might be helpful for users also. It is currently the only way (except direct GRPCs) to create a table with an index that has predefined split boundaries.
jepett0
force-pushed
the
IndexBackupRestore.1
branch
from
33900bb to
5798b1c
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
CyberROFL
approved these changes
Aug 20, 2024
MBkkt
approved these changes
Aug 21, 2024
Merged
This was referenced Aug 23, 2024
This was referenced Aug 30, 2024
stanislav-shchetinin
pushed a commit
to stanislav-shchetinin/ydb
that referenced
this pull request
Aug 30, 2024
This was referenced Sep 9, 2024
This was referenced Sep 17, 2024
jepett0
added a commit
to jepett0/ydb
that referenced
this pull request
Sep 26, 2024
jepett0
added a commit
to jepett0/ydb
that referenced
this pull request
Sep 26, 2024
jepett0
added a commit
to jepett0/ydb
that referenced
this pull request
Sep 26, 2024
jepett0
added a commit
to jepett0/ydb
that referenced
this pull request
Sep 27, 2024
jepett0
added a commit
that referenced
this pull request
Sep 27, 2024
uzhastik
pushed a commit
to uzhastik/ydb
that referenced
this pull request
Oct 7, 2024
This was referenced Oct 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#7608
This PR enables restoring indices from backups with the original partition split boundaries.
This feature is mostly needed to speed-up the restoration from backups. According to my tests (tpch
lineitemtable with 156x scale on an 8 node cluster with thecpu80_soc2_mem512G_net25G_4ssdpreset), it cuts the duration of theBuildIndexesstage of theimport/s3operation from829seconds to568seconds when restoring a 150GiB table with a single 100GiB index. This is a 31% reduction inBuildIndexestime! 🎉 Total restoration from backup time went from1259seconds to995seconds which is a 21% reduction.C++ SDK is also changed a little to enable users to create a table with an index that has specific partitioning settings and uniform partition count or explicit split boundaries. Enabling TTableBuilder to add an index based on its description makes it as capable as session.AlterTable already was. You can see how it can be helpful in the added test.