Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add blob cache option in the column family options #10155

Closed
wants to merge 6 commits into from

Conversation

gangliao
Copy link
Contributor

@gangliao gangliao commented Jun 13, 2022

Summary:

There is currently no caching mechanism for blobs, which is not ideal especially when the database resides on remote storage (where we cannot rely on the OS page cache). As part of this task, we would like to make it possible for the application to configure a blob cache.

Tasks:

This PR is a part of #10156

Copy link
Contributor

@mrambacher mrambacher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this PR is just adding the option, not the usage of the blob_cache, as I do not see it anywhere?

You need to add tests to option_test to verify that you can properly set the options. And a test that validates the option properly remains as it goes from CFOption - ImmutableCF - CFOption.

options/cf_options.cc Outdated Show resolved Hide resolved
options/options.cc Show resolved Hide resolved
@gangliao
Copy link
Contributor Author

I assume this PR is just adding the option, not the usage of the blob_cache, as I do not see it anywhere?

You need to add tests to option_test to verify that you can properly set the options. And a test that validates the option properly remains as it goes from CFOption - ImmutableCF - CFOption.

Yeah, we plan to split the giant PR to smaller ones.

Copy link
Contributor

@ltamasi ltamasi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @gangliao ! Left some comments below

db/c.cc Outdated Show resolved Hide resolved
include/rocksdb/advanced_options.h Outdated Show resolved Hide resolved
db/db_options_test.cc Outdated Show resolved Hide resolved
include/rocksdb/advanced_options.h Show resolved Hide resolved
Copy link
Contributor

@riversand963 riversand963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gangliao for the PR.
I guess the intention is to announce the feature in HISTORY.md when it's ready?

options/options_settable_test.cc Outdated Show resolved Hide resolved
@gangliao
Copy link
Contributor Author

Thanks @gangliao for the PR. I guess the intention is to announce the feature in HISTORY.md when it's ready?

Yeah, when this PR is merged, I will push more core PRs towards the feature, including HISTORY.md.

Copy link
Contributor

@ltamasi ltamasi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates! Looks good, just one minor comment

options/options_test.cc Outdated Show resolved Hide resolved
@facebook-github-bot
Copy link
Contributor

@gangliao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary:

There is currently no caching mechanism for blobs, which is not ideal especially when the database resides on remote storage (where we cannot rely on the OS page cache). As part of this task, we would like to make it possible for the application to configure a blob cache.

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@facebook-github-bot
Copy link
Contributor

@gangliao has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@gangliao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot pushed a commit that referenced this pull request Jul 9, 2022
Summary:
Update HISTORY.md for blob cache.  Implementation can be found from Github issue #10156 (or Github PRs #10155, #10178, #10225, #10198, and #10272).

Pull Request resolved: #10328

Reviewed By: riversand963

Differential Revision: D37732514

Pulled By: gangliao

fbshipit-source-id: 4c942a41c07914bfc8db56a0d3cf4d3e53d5963f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants