[Feature] Retry full backups to avoid false positives in alerts

[amshuman-kr]

Feature (What you would like to be added):
If regular full snapshot backup fails, they are retried only in the next interval (typically, 24h). We should retry the full snapshot backup in shorter time frame (say, 10m, 15m or 20m).

Motivation (Why is this needed?):
Alerts are configured to fire if the latest full backups are more than 24h old. These typically get resolved automatically in the next interval when the full backup goes through. So, most of the alerts are false positives that get resolved automatically. This makes it hard to automate the follow up process using ticketing systems (mandated by audit).

Retrying within a range of 10m-20m might resolve the issue automatically earlier so that alerts fire only if retries also fail.

Approach/Hint to the implement solution (optional):

[amshuman-kr]

@shreyas-s-rao You mentioned that the problem with failed etcd backup alerts was frequently in shoot clusters that got hibernated before the full snapshot schedule. I indeed saw many shoots where this was happening. The solution to this might be either to adjust the full snapshot cron string based on the hibernation schedule or taking a full snapshot just before hibernation as in your PR gardener/etcd-druid#59. I prefer the latter.

You also mentioned that full snapshots are retried. But I could not see this in the code. Did you mean that the backup-restore sidecar would restart if the full snapshot failed? ->

etcd-backup-restore/pkg/snapshot/snapshotter/snapshotter.go

Lines 490 to 493 in 72720b4

	case <-ssr.fullSnapshotTimer.C:
	if _, err := ssr.TakeFullSnapshotAndResetTimer(); err != nil {
	return err
	}

[shreyas-s-rao]

@amshuman-kr All snapshots (full and delta) are uploaded via chunk upload. And there is an exponential backoff for chunk upload retries here. So both full and delta snapshot failures are re-attempted.

Regarding the missed snapshots issue, we can either modify this value to something like 23.5 instead of 24, so that we can accommodate the previous day's full snapshots that was probably a few minutes late. Or else, even the full snapshot on hibernation is a safe approach, and is implemented anyway.
I would suggest fixing the issue within backup sidecar itself, so that it can be consumed bug-free by anybody without druid too.

[amshuman-kr]

All snapshots (full and delta) are uploaded via chunk upload. And there is an exponential backoff for chunk upload retries here. So both full and delta snapshot failures are re-attempted.

Ok. This covers upload errors but not other errors during full snapshot (e.g. timeout while taking the snapshot). But I guess those are really rare.

[amshuman-kr]

Regarding the missed snapshots issue, we can either modify this value to something like 23.5 instead of 24, so that we can accommodate the previous day's full snapshots that was probably a few minutes late. Or else, even the full snapshot on hibernation is a safe approach, and is implemented anyway.
I would suggest fixing the issue within backup sidecar itself, so that it can be consumed bug-free by anybody without druid too.

It sure makes sense to calculate this value based on the full-snapshot schedule. But it still will be considered only when the sidecar restarts. I don't think the hibernation case can be addressed in the back-restore sidecar because it is not at all aware of anything called hibernation. It can only be addressed (as in gardener/etcd-druid#59) in etcd-druid.