allow more time for boltdb-shipper index syncs to finish #6071
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* retains index files with ingester for 20 mins instead of 12 mins after uploading * sets GetChunkIDs lookback to 2h41m instead of 2h28m
This was referenced Jun 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
With
boltdb-shipper, we have some timers configured to manage the index across various services. This PR looks at the time we allow sync operations to finish in the worst-case scenario.With the current timers set, in the worst-case scenario, we should be finishing the index sync operation in up to 2 mins and if we fail, users might see gaps in their query response. This PR tunes some timers to allow syncs to have more time to run, specifically allowing sync to get the updates in up to 3 attempts. This change impacts the following timers with default values:
The impact of this is ingesters getting more queries from queries for
GetChunkIDssince we are increasing the lookback