Rework SLM to avoid concurrent snapshots by logging-and-skipping

[joegallo]

Related to some of the discussion that came out of #64035.

Imagine, for example, the case of a cluster that has an SLM policy configured to take snapshots every 30 minutes, but where those snapshots actually take 35 minutes to perform. Currently, this case will result in overlapping snapshots being scheduled (less of an issue now that snapshots can be executed in parallel, see #56911).

However, it might be beneficial to log-and-skip in those cases rather than scheduling the overlapping snapshot. In essence, then, this would change an SLM policy from a scheduleSnapshot to a maybeScheduleSnapshot. In the 30/35 minutes illustration, we'd end up snapshotting once an hour rather than once every 30 minutes, logging-and-skipping every other snapshot. That is, in the face of snapshot over-scheduling, we degrade gracefully to a less frequent schedule.

/cc @original-brownbear

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[original-brownbear]

Thanks for the ping @joegallo

It seems to me that the fact that this is a potential issue is just a result of sub-optimal performance of the snapshot logic. See #61870 for details, I think should just make sure that the second snapshot is effectively free for those shards for which the first snapshot hasn't yet started which would completely remove any issue with snapshots "pilling up" because they are slower than the schedule. I think log and skip may be a reasonable workaround in the short term if this is an actual problem and just solving it by scheduling via an interval isn't good enough as a short term fix, but the real fix should be #61870 IMO.

[BenB196]

I'd like to provide a real world example of that I am running into where you could occasionally run into the same snapshot running concurrently, and where this feature (or similar) would be useful.

(In my example cluster resources aren't the issue, but instead upload bandwidth is)

Say you have a snapshot that runs every hour, and normally the snapshot can finish in 20-30 minutes. However, when an underlying index (or many indexes) receive large changes that result in high deltas, a snapshot could take far longer than normal, resulting in overlap.

A good example is with ILM and force merge. You could run into a scenario where 1 or more indexes get merged to a single segment within the same hour. This would result in having to re-backup large amounts of data within a short period of time. While not inherently a bad thing to have some overlap of snapshots, having the snapshots run at the same time results in the bandwidth being split between the two (I believe), resulting in the first snapshot taking even longer.

In my specific use-case, I'd much rather have secured data with 1 complete snapshot and skip a snapshot cycle, then lose data because two snapshots are competing for the same resources in the event of a failure.

[jakelandis]

Also related: #70587

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[joegallo]

I'm removing the team-discuss label from some older Team:Data Management issues -- we've had plenty of time to discuss them, but we haven't, so the label isn't serving its purpose. Feel free to delete this comment and/or re-add the team-discuss label.