Order shards by "changes" when snapshoting

[pgomulka]

Description

Snapshotting process when interrupted will return all not copied yet shards as PARTIAL.
If shards were ordered by what changes they have, the snapshoting process could work with "unchanged" shards first and finish them immediately. Then when working on copying a bigger "changed" shard, during interruption (node with shard is shut down) only the shards with changes will be marked as PARTIAL.
This could be an improvement untill #71333 is implemented.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[original-brownbear]

Just adding a few details about this plan:

We basically want to add another version of org.elasticsearch.repositories.Repository#snapshotShard here that will only take do the snapshot in case it's a noop (does not require uploading any data files).
This way we can do one fast pass over all shards on a data node to deal with the quick noop shard snapshots that just update metadata. After that pass we check which shards were not noops and upload those in another pass that does the file uploading.
=> way fewer shard failures when snapshotting while there's node restarts or other node changes going on.

[pxsalehi]

After a very nice and fruitful discussion with @original-brownbear and based on his suggestions, we arrived at the following approach:

As of the current master (bc9478d), when starting the shard snapshot tasks, we loop through the shards to be snapshotted and run the above-mentioned Repository#snapshotShard. This will calculate whether/which changes in the shard there are to be snapshotted, write some metadata, and ONLY if there are changes and there are files to be uploaded, fork off the file upload calls. This means, the issue as described here does not have such a severe impact (probably the code has been improved/optimized since then).

However, as suggested by Armin, this could even be further improved by starting the shard snapshot tasks in parallel. This would mean that since all of the above tasks are scheduled on the same threadpool, those shard snapshot calls that have no files to upload would just finish in the same thread, and the file upload calls (kind of!) go to the end of the working queue.

That being said, since these are not synchronized and some of the file upload tasks could run before all the start shard snapshots have been started, file upload tasks could still keep some of the threads busy and take precedence over unchanged shard snapshots. However, this wouldn't starve other snapshot calls since the number of parallel file uploads for the same shard snapshot is throttled.