Make it possible to enable blob files starting from a certain LSM tree level #10077
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
TODO:
|
|
ltamasi
approved these changes
Jun 2, 2022
…e level Summary: Currently, if blob files are enabled (i.e. `enable_blob_files` is true), large values are extracted both during flush/recovery (when SST files are written into level 0 of the LSM tree) and during compaction into any LSM tree level. For certain use cases that have a mix of short-lived and long-lived values, it might make sense to support extracting large values only during compactions whose output level is greater than or equal to a specified LSM tree level (e.g. compactions into L1/L2/... or above). This could reduce the space amplification caused by large values that are turned into garbage shortly after being written at the price of some write amplification incurred by long-lived values whose extraction to blob files is delayed. In order to achieve this, we would like to do the following: - Add a new configuration option `blob_file_starting_level` (default: 0) to `AdvancedColumnFamilyOptions` (and `MutableCFOptions` and extend the related logic) - Instantiate `BlobFileBuilder` in `BuildTable` (used during flush and recovery, where the LSM tree level is L0) and `CompactionJob` iff `enable_blob_files` is set and the LSM tree level is `>= blob_file_starting_level` - Add unit tests for the new functionality, and add the new option to our stress tests (`db_stress` and `db_crashtest.py` ) - Add the new option to our benchmarking tool `db_bench` and the BlobDB benchmark script `run_blob_bench.sh` - Add the new option to the `ldb` tool (see https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool) - Ideally extend the C and Java bindings with the new option - Update the BlobDB wiki to document the new option. Reviewers:
gangliao
force-pushed
the
blob_start_level
branch
from
de18e1e
to
8bdda14
Compare
|
@gangliao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@gangliao has updated the pull request. You must reimport the pull request before landing. |
|
@gangliao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ltamasi
reviewed
Jun 2, 2022
|
@gangliao has updated the pull request. You must reimport the pull request before landing. |
|
@gangliao has updated the pull request. You must reimport the pull request before landing. |
|
@gangliao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@gangliao has updated the pull request. You must reimport the pull request before landing. |
|
@gangliao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Currently, if blob files are enabled (i.e.
enable_blob_filesis true), large values are extracted both during flush/recovery (when SST files are written into level 0 of the LSM tree) and during compaction into any LSM tree level. For certain use cases that have a mix of short-lived and long-lived values, it might make sense to support extracting large values only during compactions whose output level is greater than or equal to a specified LSM tree level (e.g. compactions into L1/L2/... or above). This could reduce the space amplification caused by large values that are turned into garbage shortly after being written at the price of some write amplification incurred by long-lived values whose extraction to blob files is delayed.In order to achieve this, we would like to do the following:
blob_file_starting_level(default: 0) toAdvancedColumnFamilyOptions(andMutableCFOptionsand extend the related logic)BlobFileBuilderinBuildTable(used during flush and recovery, where the LSM tree level is L0) andCompactionJobiffenable_blob_filesis set and the LSM tree level is>= blob_file_starting_leveldb_stressanddb_crashtest.py)db_benchand the BlobDB benchmark scriptrun_blob_bench.shldbtool (see https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool)Reviewers: