Skip to content

Compactor: Language fixes #7315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 7, 2024
Merged

Compactor: Language fixes #7315

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions cmd/mimir/config-descriptor.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8886,7 +8886,7 @@
"kind": "field",
"name": "cleanup_interval",
"required": false,
"desc": "How frequently compactor should run blocks cleanup and maintenance, as well as update the bucket index.",
"desc": "How frequently the compactor should run blocks cleanup and maintenance, as well as update the bucket index.",
"fieldValue": null,
"fieldDefaultValue": 900000000000,
"fieldFlag": "compactor.cleanup-interval",
Expand All @@ -8908,7 +8908,7 @@
"kind": "field",
"name": "deletion_delay",
"required": false,
"desc": "Time before a block marked for deletion is deleted from bucket. If not 0, blocks will be marked for deletion and compactor component will permanently delete blocks marked for deletion from the bucket. If 0, blocks will be deleted straight away. Note that deleting blocks immediately can cause query failures.",
"desc": "Time before a block marked for deletion is deleted from bucket. If not 0, blocks will be marked for deletion and the compactor component will permanently delete blocks marked for deletion from the bucket. If 0, blocks will be deleted straight away. Note that deleting blocks immediately can cause query failures.",
"fieldValue": null,
"fieldDefaultValue": 43200000000000,
"fieldFlag": "compactor.deletion-delay",
Expand All @@ -8919,7 +8919,7 @@
"kind": "field",
"name": "tenant_cleanup_delay",
"required": false,
"desc": "For tenants marked for deletion, this is time between deleting of last block, and doing final cleanup (marker files, debug files) of the tenant.",
"desc": "For tenants marked for deletion, this is the time between deletion of the last block, and doing final cleanup (marker files, debug files) of the tenant.",
"fieldValue": null,
"fieldDefaultValue": 21600000000000,
"fieldFlag": "compactor.tenant-cleanup-delay",
Expand Down Expand Up @@ -8963,7 +8963,7 @@
"kind": "field",
"name": "max_closing_blocks_concurrency",
"required": false,
"desc": "Max number of blocks that can be closed concurrently during split compaction. Note that closing of newly compacted block uses a lot of memory for writing index.",
"desc": "Max number of blocks that can be closed concurrently during split compaction. Note that closing a newly compacted block uses a lot of memory for writing the index.",
"fieldValue": null,
"fieldDefaultValue": 1,
"fieldFlag": "compactor.max-closing-blocks-concurrency",
Expand Down Expand Up @@ -8996,7 +8996,7 @@
"kind": "field",
"name": "enabled_tenants",
"required": false,
"desc": "Comma separated list of tenants that can be compacted. If specified, only these tenants will be compacted by compactor, otherwise all tenants can be compacted. Subject to sharding.",
"desc": "Comma separated list of tenants that can be compacted. If specified, only these tenants will be compacted by the compactor, otherwise all tenants can be compacted. Subject to sharding.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "compactor.enabled-tenants",
Expand All @@ -9007,7 +9007,7 @@
"kind": "field",
"name": "disabled_tenants",
"required": false,
"desc": "Comma separated list of tenants that cannot be compacted by this compactor. If specified, and compactor would normally pick given tenant for compaction (via -compactor.enabled-tenants or sharding), it will be ignored instead.",
"desc": "Comma separated list of tenants that cannot be compacted by the compactor. If specified, and the compactor would normally pick a given tenant for compaction (via -compactor.enabled-tenants or sharding), it will be ignored instead.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "compactor.disabled-tenants",
Expand Down
12 changes: 6 additions & 6 deletions cmd/mimir/help-all.txt.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -938,7 +938,7 @@ Usage of ./cmd/mimir/mimir:
-compactor.cleanup-concurrency int
Max number of tenants for which blocks cleanup and maintenance should run concurrently. (default 20)
-compactor.cleanup-interval duration
How frequently compactor should run blocks cleanup and maintenance, as well as update the bucket index. (default 15m0s)
How frequently the compactor should run blocks cleanup and maintenance, as well as update the bucket index. (default 15m0s)
-compactor.compaction-concurrency int
Max number of concurrent compactions running. (default 1)
-compactor.compaction-interval duration
Expand All @@ -952,17 +952,17 @@ Usage of ./cmd/mimir/mimir:
-compactor.data-dir string
Directory to temporarily store blocks during compaction. This directory is not required to be persisted between restarts. (default "./data-compactor/")
-compactor.deletion-delay duration
Time before a block marked for deletion is deleted from bucket. If not 0, blocks will be marked for deletion and compactor component will permanently delete blocks marked for deletion from the bucket. If 0, blocks will be deleted straight away. Note that deleting blocks immediately can cause query failures. (default 12h0m0s)
Time before a block marked for deletion is deleted from bucket. If not 0, blocks will be marked for deletion and the compactor component will permanently delete blocks marked for deletion from the bucket. If 0, blocks will be deleted straight away. Note that deleting blocks immediately can cause query failures. (default 12h0m0s)
-compactor.disabled-tenants comma-separated-list-of-strings
Comma separated list of tenants that cannot be compacted by this compactor. If specified, and compactor would normally pick given tenant for compaction (via -compactor.enabled-tenants or sharding), it will be ignored instead.
Comma separated list of tenants that cannot be compacted by the compactor. If specified, and the compactor would normally pick a given tenant for compaction (via -compactor.enabled-tenants or sharding), it will be ignored instead.
-compactor.enabled-tenants comma-separated-list-of-strings
Comma separated list of tenants that can be compacted. If specified, only these tenants will be compacted by compactor, otherwise all tenants can be compacted. Subject to sharding.
Comma separated list of tenants that can be compacted. If specified, only these tenants will be compacted by the compactor, otherwise all tenants can be compacted. Subject to sharding.
-compactor.first-level-compaction-wait-period duration
How long the compactor waits before compacting first-level blocks that are uploaded by the ingesters. This configuration option allows for the reduction of cases where the compactor begins to compact blocks before all ingesters have uploaded their blocks to the storage. (default 25m0s)
-compactor.max-block-upload-validation-concurrency int
Max number of uploaded blocks that can be validated concurrently. 0 = no limit. (default 1)
-compactor.max-closing-blocks-concurrency int
Max number of blocks that can be closed concurrently during split compaction. Note that closing of newly compacted block uses a lot of memory for writing index. (default 1)
Max number of blocks that can be closed concurrently during split compaction. Note that closing a newly compacted block uses a lot of memory for writing the index. (default 1)
-compactor.max-compaction-time duration
Max time for starting compactions for a single tenant. After this time no new compactions for the tenant are started before next compaction cycle. This can help in multi-tenant environments to avoid single tenant using all compaction time, but also in single-tenant environments to force new discovery of blocks more often. 0 = disabled. (default 1h0m0s)
-compactor.max-opening-blocks-concurrency int
Expand Down Expand Up @@ -1052,7 +1052,7 @@ Usage of ./cmd/mimir/mimir:
-compactor.symbols-flushers-concurrency int
Number of symbols flushers used when doing split compaction. (default 1)
-compactor.tenant-cleanup-delay duration
For tenants marked for deletion, this is time between deleting of last block, and doing final cleanup (marker files, debug files) of the tenant. (default 6h0m0s)
For tenants marked for deletion, this is the time between deletion of the last block, and doing final cleanup (marker files, debug files) of the tenant. (default 6h0m0s)
-config.expand-env
Expands ${var} or $var in config according to the values of the environment variables.
-config.file value
Expand Down
27 changes: 14 additions & 13 deletions docs/sources/mimir/configure/configuration-parameters/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3976,8 +3976,8 @@ The `compactor` block configures the compactor component.
# CLI flag: -compactor.first-level-compaction-wait-period
[first_level_compaction_wait_period: <duration> | default = 25m]

# (advanced) How frequently compactor should run blocks cleanup and maintenance,
# as well as update the bucket index.
# (advanced) How frequently the compactor should run blocks cleanup and
# maintenance, as well as update the bucket index.
# CLI flag: -compactor.cleanup-interval
[cleanup_interval: <duration> | default = 15m]

Expand All @@ -3987,15 +3987,16 @@ The `compactor` block configures the compactor component.
[cleanup_concurrency: <int> | default = 20]

# (advanced) Time before a block marked for deletion is deleted from bucket. If
# not 0, blocks will be marked for deletion and compactor component will
# not 0, blocks will be marked for deletion and the compactor component will
# permanently delete blocks marked for deletion from the bucket. If 0, blocks
# will be deleted straight away. Note that deleting blocks immediately can cause
# query failures.
# CLI flag: -compactor.deletion-delay
[deletion_delay: <duration> | default = 12h]

# (advanced) For tenants marked for deletion, this is time between deleting of
# last block, and doing final cleanup (marker files, debug files) of the tenant.
# (advanced) For tenants marked for deletion, this is the time between deletion
# of the last block, and doing final cleanup (marker files, debug files) of the
# tenant.
# CLI flag: -compactor.tenant-cleanup-delay
[tenant_cleanup_delay: <duration> | default = 6h]

Expand All @@ -4017,8 +4018,8 @@ The `compactor` block configures the compactor component.
[max_opening_blocks_concurrency: <int> | default = 1]

# (advanced) Max number of blocks that can be closed concurrently during split
# compaction. Note that closing of newly compacted block uses a lot of memory
# for writing index.
# compaction. Note that closing a newly compacted block uses a lot of memory for
# writing the index.
# CLI flag: -compactor.max-closing-blocks-concurrency
[max_closing_blocks_concurrency: <int> | default = 1]

Expand All @@ -4032,15 +4033,15 @@ The `compactor` block configures the compactor component.
[max_block_upload_validation_concurrency: <int> | default = 1]

# (advanced) Comma separated list of tenants that can be compacted. If
# specified, only these tenants will be compacted by compactor, otherwise all
# tenants can be compacted. Subject to sharding.
# specified, only these tenants will be compacted by the compactor, otherwise
# all tenants can be compacted. Subject to sharding.
# CLI flag: -compactor.enabled-tenants
[enabled_tenants: <string> | default = ""]

# (advanced) Comma separated list of tenants that cannot be compacted by this
# compactor. If specified, and compactor would normally pick given tenant for
# compaction (via -compactor.enabled-tenants or sharding), it will be ignored
# instead.
# (advanced) Comma separated list of tenants that cannot be compacted by the
# compactor. If specified, and the compactor would normally pick a given tenant
# for compaction (via -compactor.enabled-tenants or sharding), it will be
# ignored instead.
# CLI flag: -compactor.disabled-tenants
[disabled_tenants: <string> | default = ""]

Expand Down
7 changes: 3 additions & 4 deletions pkg/compactor/blocks_cleaner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,8 @@ func (c *BlocksCleaner) stopping(error) error {
}

func (c *BlocksCleaner) starting(ctx context.Context) error {
// Run an initial cleanup in starting state. (Note that compactor no longer waits
// for blocks cleaner to finish starting before it starts compactions.)
// Run an initial cleanup in starting state. (Note that the compactor no longer waits
// for the blocks cleaner to finish starting before it starts compactions.)
c.runCleanup(ctx, false)

return nil
Expand Down Expand Up @@ -331,7 +331,6 @@ func (c *BlocksCleaner) deleteUserMarkedForDeletion(ctx context.Context, userID
level.Info(userLogger).Log("msg", "deleted block", "block", id)
return nil
})

if err != nil {
return err
}
Expand Down Expand Up @@ -381,7 +380,7 @@ func (c *BlocksCleaner) deleteUserMarkedForDeletion(ctx context.Context, userID

level.Info(userLogger).Log("msg", "cleaning up remaining blocks data for tenant marked for deletion")

// Let's do final cleanup of tenant.
// Let's do a final cleanup of the tenant.
if deleted, err := bucket.DeletePrefix(ctx, userBucket, block.DebugMetas, userLogger); err != nil {
return errors.Wrap(err, "failed to delete "+block.DebugMetas)
} else if deleted > 0 {
Expand Down
6 changes: 3 additions & 3 deletions pkg/compactor/blocks_cleaner_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,13 +125,13 @@ func testBlocksCleanerWithOptions(t *testing.T, options testBlocksCleanerOptions
{path: path.Join("user-1", block3.String(), block.MetaFilename), expectedExists: false},
{path: path.Join("user-2", block7.String(), block.MetaFilename), expectedExists: false},
{path: path.Join("user-2", block8.String(), block.MetaFilename), expectedExists: true},
// Should not delete a block with deletion mark who hasn't reached the deletion threshold yet.
// Should not delete a block with deletion mark which hasn't reached the deletion threshold yet.
{path: path.Join("user-1", block2.String(), block.MetaFilename), expectedExists: true},
{path: path.Join("user-1", block.DeletionMarkFilepath(block2)), expectedExists: true},
// Should delete a partial block with deletion mark who hasn't reached the deletion threshold yet.
// Should delete a partial block with deletion mark which hasn't reached the deletion threshold yet.
{path: path.Join("user-1", block4.String(), block.DeletionMarkFilename), expectedExists: false},
{path: path.Join("user-1", block.DeletionMarkFilepath(block4)), expectedExists: false},
// Should delete a partial block with deletion mark who has reached the deletion threshold.
// Should delete a partial block with deletion mark which has reached the deletion threshold.
{path: path.Join("user-1", block5.String(), block.DeletionMarkFilename), expectedExists: false},
{path: path.Join("user-1", block.DeletionMarkFilepath(block5)), expectedExists: false},
// Should not delete a partial block without deletion mark.
Expand Down
Loading