Compactor: retry compaction of a single tenant on failure instead of re-running compaction for all tenants #3627
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for this, should reduce the amount of list calls on s3 or similar. |
pstibrany
reviewed
Jan 4, 2021
pkg/compactor/compactor.go
Outdated
| MaxRetries: c.compactorCfg.CompactionRetries, | ||
| }) | ||
|
|
||
| func (c *Compactor) compactUsers(ctx context.Context) (returnErr error) { |
There was a problem hiding this comment.
Returned error is not used anywhere. We should remove it (we already log all errors anyway), and perhaps also simplify returnErr to boolean flag for updating metrics.
There was a problem hiding this comment.
Does it look better now?
There was a problem hiding this comment.
Looks better. As a final change, I'd suggest removing unused return value completely, and convert succeeded into local variable. (non-blocking nit)
There was a problem hiding this comment.
Oh yeah, sure, done.
pracucci
force-pushed
the
compactor-retry-single-tenant-on-failure
branch
from
d80e8db to
c0a4a4b
Compare
pstibrany
approved these changes
Jan 4, 2021
…re-running compaction for all tenants Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
pracucci
force-pushed
the
compactor-retry-single-tenant-on-failure
branch
from
44f59fe to
6b3a5b9
Compare
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Currently, the compactor re-run compaction for all tenants in case at least 1 tenant failed. Despite re-running the compaction for a previously succeeded tenant doesn't redo the work, but considering each single compaction run needs to re-scan the bucket, this ends up to be pretty inefficient for Cortex clusters running with a large number of tenants.
In this PR I'm proposing to retry compaction on failure only for the failed tenants.
Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]