Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs deletion support for boltdb-shipper store #3688

Merged
merged 13 commits into from
May 18, 2021

Conversation

sandeepsukhani
Copy link
Contributor

@sandeepsukhani sandeepsukhani commented May 5, 2021

What this PR does / why we need it:
This PR leverages existing code for retention to support deletion for boltdb-shipper.
It relies on the Mark phase of retention which now also looks at eligible delete requests by talking to DeleteRequestsManager to decide whether the chunk is supposed to be deleted or not. For partially deleted chunks I have added a chunk rewriter which would build chunks for the interval for which they are not deleted and writes them to the store. A successful Mark process over all the tables considers delete requests as processed and the relevant chunk should be deleted later by the Sweep process.

The delete requests are persisted in a table called delete_requests which is a boltdb file. The file is uploaded to the object store every 5 mins or during shutdown and is downloaded locally if it does not exist during startup.

Here is how deletion would roughly work:

  1. Call MarkPhaseStarted of DeleteRequestsManager before starting the compaction which would load eligible delete requests in memory.
  2. Each Mark phase of the table would check if one or more delete requests deletes the chunks in it and take care of deleting the chunk index and rewriting it if it was partially deleted.
  3. Call MarkPhaseFinished after successful compaction of all the tables which would mark the in-memory requests as processed.
  4. Call MarkPhaseFailed if any of the tables fails to compact which would just drop requests from memory.

Checklist

  • Documentation added
  • Tests updated

@sandeepsukhani sandeepsukhani requested a review from cyriltovena May 5, 2021 13:00
@sandeepsukhani sandeepsukhani force-pushed the deletion-boltdb-shipper branch 3 times, most recently from 7dbec90 to d9216d5 Compare May 6, 2021 06:01
@sandeepsukhani sandeepsukhani force-pushed the deletion-boltdb-shipper branch from 151e41a to 5429d7f Compare May 11, 2021 13:14
@sandeepsukhani sandeepsukhani marked this pull request as ready for review May 12, 2021 05:42
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I'd appreciate if you could grab the userID without allocation, when deleting millions of chunks it might slow everything down.

@sandeepsukhani sandeepsukhani merged commit 2b7a6f2 into grafana:main May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants