(question) How many concurrent snapshots?

[thevops]

Hello,

It's not a feature request or bug report. I have a question regarding performance and architecture.

I use GKE 1.25.
I have multiple (~100) small applications (StatefulSet) with small disks (~5 GB). There are critical data that I have to back up very frequently (every 1-6 hours).
I've created Snapschedule for each, so it runs every minute. Retention is 2 days, so for 100 apps with 1 snapshot per hour, I have 48 snapshots per app => 4800 snapshots overall.

By the way, I got that error very often - kubernetes-csi/external-snapshotter#300 - the fix is in progress.
I'm afraid about the snapshot mechanism performance. It's also a question about the GCP snapshot mechanism, so I'm gathering opinions.

Do you think is it good to maintain such a big number of Snapshedule/snapshots inside Kubernetes?
I'm wondering about moving this mechanism inside GCP. I've tested GCP Snapshot Schedule. It works quite well. The only disadvantage is that I have to define all disks on the list, but I use Terraform so it's a one-shot task.

Is there someone who maintained similar Snapschedules like the example above?

Thanks for your answers.

[JohnStrunk]

Some thoughts...

• I don't expect your use case to be a problem for SnapScheduler since it's just creating/deleting the VolumeSnapshot objects. The actual work is in external-snapshotter for coordination and the CSI driver for GCP PD to actually take the snapshot.
• The overall object count in the cluster (VS + VSC objects) should be manageable by the control plane

Things to consider:

• What are you trying to protect against: App/user failure that corrupts data or loss of cluster?
• What does your recovery procedure look like?

SnapScheduler is good if you're not overly concerned about cluster loss. While the snapshots would exist w/in the cloud infrastructure even if the cluster is lost, figuring out which snapshot belongs to which app may be time-consuming.
This also goes for taking snapshots via GCP's scheduler. You'll have a copy of the data, but do you also have a reasonable method to get it back into a PV for your app?

If you are concerned about cluster loss, you may want to look at the restic mover in VolSync. That would give you backups that are independent of your cluster, and reasonably easy to restore. There also wouldn't be a bunch of snapshot objects since VolSync's snapshots are only temporary, to support a point-in-time copy for the replication. It may also save you a little money since you aren't paying for maintaining the snapshots--- the restic repo is deduped and compressed.

[thevops]

First of all, thanks a lot for your answer! It was very fast after I wrote.

What are you trying to protect against: App/user failure that corrupts data or loss of cluster?

Applications generate some very important data from time to time (it's unpredictable when). I'd like to back up the data as often as possible to restore them in case of a crash.
Besides internal backups - VolumeSapshots - I also have external backups. VolSync with Restic is sending the data to the S3 bucket.
VolSync is working once per day for each app, as for now. I'm wondering about increasing the frequency to 2/3 per day, but it's a different topic.

What does your recovery procedure look like?

The time to recover does not have to be very small. Having snapshots inside the cluster I just use them as a source for PVC.
Having snapshots outside the cluster requires a few more actions (importing snapshot - https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/docs/kubernetes/user-guides/snapshots.md#import-a-pre-existing-snapshot). But that's not the case here. Recovery time is not as important as having the latest data.

It brings me an idea to create a sidecar for each app. That sidecar could copy data in real-time and even send them outside the cluster. However, it increases complexity and decreases security (the backup destination would be connected all the time to the app in some way).

I'm not overly concerned about cluster loss, but as I wrote I have outside backup also. I'm using "Direct" method for Restic.

Basically, you have answered my main concern with: The overall object count in the cluster (VS + VSC objects) should be manageable by the control plane.

Anyway, if I stay with Snapshot I need to verify the issue kubernetes-csi/external-snapshotter#300, because it can also affect the results.

Again, thanks for your very fast answer :)

If you have any thoughts related to my answer I'd like to read them.

[JohnStrunk]

Sounds like you've got things pretty well covered w/ the combination of on-cluster and off-cluster backups.
The only thing I'd suggest is moving to Snapshots/clones for the VolSync+Restic. This may not be an option for you until the controller fix goes in, but consider it longer term. The reason is to ensure a point-in-time image is captured in the backup. When using Direct, you get the version of the data that exists when restic happens to read it as opposed to the overall backup being from a single point in time. This could affect the consistency of the data that the application depends on.

[thevops]

Hmm, you're right. I need to test snapshots for VolSync. I chose Direct because it doesn't create any new objects. The clone option creates snapshots, but from a security perspective, it makes sense.