Skip to content

Sync and clean up local shards when an ingester starts #5525

@guilload

Description

@guilload

We have a one-way sync from the control plane to the ingesters, but the opposite path does not exist. In the case of Airmal, they resized their disks, which scratched the WALs. When the ingesters were restarted, they had no way to tell the CP that the shards they hosted no longer existed. The decommissioning logic should help prevent this scenario in the case of a graceful shutdown, but if, for some reason, it is not triggered, we can only fix the cluster state manually (clean up shards in metastore and restart CP).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions