Skip to content

Compactor: keep bucket index updated #3583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Dec 11, 2020

Conversation

pracucci
Copy link
Contributor

What this PR does:
This PR is another step forward towards the implementation of the bucket index for the blocks storage (see proposal). In this PR:

  • Keep bucket index updated by compactor's blocks cleaner
  • Migrate block deletion marks to the per-tenant global location at compactor startup (with a flag to disable it once done)
  • Added new per-tenant metrics exposed by the compactor with the number of blocks and timestamp of the last bucket index update (these metrics are removed once a tenant is deleted or moved to a different shard)

Which issue(s) this PR fixes:
Fixes #3398

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Contributor

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work! Pre-approving, but please check my comments before merging.

Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci merged commit 1fd1fd8 into cortexproject:master Dec 11, 2020
@pracucci pracucci deleted the write-bucket-index branch December 11, 2020 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Need metric for monitoring number of blocks per tenant
2 participants