Automatic write blocks to ensure strong snapshot consistency across multiple shards, or cluster wide

[benweet]

Description

I'm trying to make a snapshot with cluster wide consistency by creating write blocks across all cluster indices before, and deleting them after the snapshot.

Though I'm concerned about application downtime in case large data sets are being backed up...

As I understand, it should be possible to delete write blocks as soon as all primary shards are snapshot-ed locally without having to wait for shards data to be uploaded to the snapshot repository. However, I'm not sure if:

• I can assume that all shards are locally snapshot-ed as soon as the snapshot is created with the Create snapshot API (without waiting for completion), or
• I have to poll the Get snapshot status API to monitor the shards status and wait for each shard stage to be STARTED?

Going a bit further, would it be possible to have a block_writes=true parameter in the "Create snapshot API" to automate consistent snapshots creation in ES?

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Each shard creates its point-in-time view at about the time it moves to STARTED, but it might happen after moving it to STARTED so polling for STARTED is not enough. Moreover Elasticsearch doesn't retry today but it might in a future version and that would involve shards moving to STARTED more than once.

It might work to wait for all shards of an index to be marked as DONE today but I don't think that's something we have any way to guarantee in all future versions. Really the only reliable solution in current versions is to wait for the end of the snapshot.

It's an interesting idea tho. In theory we could start by applying coordinated write blocks, acquire the point-in-time views, and release the blocks, all at the very start of the snapshot. It'd be quite a big change to the snapshot state machine as it exists today and has some interesting failure modes that might make retries harder/more expensive, but we do at least have all the pieces needed to implement this.

[original-brownbear]

With the work done in #88209 that gives a higher priority to the work of taking Lucene snapshots and determining what needs to be uploaded relative to actually uploading data files, the possible skew in a snapshot should be much reduced (from hours to minutes in the worst case scenarios).
This being the case and given how complex it would be to do better than what we now have in terms of skew we decided to not take further action on this for the time being.
-> removing the team-discuss label and we can come back to this if this remains a real-world problem at some later date