Skip to content

Add bucket index support to store gateway #3625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

pracucci
Copy link
Contributor

@pracucci pracucci commented Dec 21, 2020

What this PR does:
Following up the PR #3614, in this PR I'm adding bucket index support to the store-gateway. The usage of the bucket index in the store-gateway is optional and disabled by default.

I haven't changed any integration test because the existing tests in integration/querier_test.go already covers it.

Which issue(s) this PR fixes:
N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@pracucci pracucci requested a review from pstibrany December 21, 2020 17:33
replicaRemovedMeta = "replica-label-removed"
)

// fetcherMetrics is a copy of Thanos internal fetcherMetrics. These metrics have been copied from
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've opened a PR in Thanos thanos-io/thanos#3660 to export the fetcherMetrics to hopefully remove the duplicated code (even if may be tricky to integrate anyway due to the additional label values corruptedBucketIndex and noBucketIndex).

Copy link
Contributor

@ranton256 ranton256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pracucci pracucci force-pushed the add-bucket-index-support-to-store-gateway branch from b94aae9 to 465b91f Compare December 22, 2020 10:47
@lzh-lab
Copy link
Contributor

lzh-lab commented Dec 23, 2020

Compactor block cleaner report an error when directory markers hava a block deletion mark which the block no longer exists.

component=cleaner org_id=fake msg="failed blocks cleanup and maintenance" err="marker not found"

@lzh-lab
Copy link
Contributor

lzh-lab commented Dec 23, 2020

Compactor block cleaner report an error when directory markers hava a block deletion mark which the block no longer exists.

component=cleaner org_id=fake msg="failed blocks cleanup and maintenance" err="marker not found"

This will result in UpdateIndex always failure.

Maybe this should be a warning or just ignored.

@pracucci
Copy link
Contributor Author

Compactor block cleaner report an error when directory markers hava a block deletion mark which the block no longer exists.

component=cleaner org_id=fake msg="failed blocks cleanup and maintenance" err="marker not found"

This will result in UpdateIndex always failure.

Maybe this should be a warning or just ignored.

Should be fixed by this change:
https://github.com/cortexproject/cortex/pull/3625/files#diff-821456401172c65d2c0cf8706a3d700bf213d73f6932cee9ef7d1d805693b097

If, once this PR is merge, it will happen again, then we can re-investigate it.

Copy link
Contributor

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work, I don't see any flaws in it. Good job!

@pracucci pracucci force-pushed the add-bucket-index-support-to-store-gateway branch from 465b91f to 6fe30cf Compare January 4, 2021 13:06
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci force-pushed the add-bucket-index-support-to-store-gateway branch from cd2f4f6 to 88523e6 Compare January 5, 2021 07:47
@pracucci pracucci merged commit bb28fb5 into cortexproject:master Jan 5, 2021
@pracucci pracucci mentioned this pull request Mar 1, 2021
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants