Leaking shard: after deletion of an index/a source

[fulmicoton]

• if the ingester is present at the moment of the event
• if the ingester is absent at the moment of the event

[fulmicoton]

Plan:

Happy path

Before running any deletion, make a copy of all of the shards that needs to be deleted.
Delete from (source/index) from metastore using the usual transactional dance. (uncertain error trigger a controlplane reload)
Delete from model.
Send shard delete requests to ingesters, with a retry. If it fails, we rely on our other path.

Not so happy path

...
Not sure yet. something something chitchat.
The control plan could publish the list of available shards and ingesters could subscribe to deletion of key events.
Of course, we may have a problem if an ingester is disconnected for more than the chitchat deletion grace period.

At this point, I think it ok to restart the ingester entirely.