cephfs: efficient read-only snapshots as volumes

[gman0]

Describe the feature you'd like to have

CephFS exposes snapshots as special, read-only directories of a subvolume. cephfs-csi can already provision writable volumes with snapshots as their data source, where snapshot contents are cloned to the newly created volume. Cloning to snapshot to volume is very expensive in CephFS, as the data needs to be fully copied.

I propose creating read-only (shallow) volumes from snapshots is added to cephfs-csi, where the driver would take advantage of the fact that snapshot's data is already available in a directory, and that directory would be exposed to workloads as a volume. This would make it possible to avoid the expensive cloning process.

What is the value to the end user? (why is it a priority?)

What's the point of such read-only volumes?

• Restore snapshots selectively: users may want to traverse snapshots, restoring data to a writable volume more selectively instead of restoring the whole snapshot.

• Volume backup: users can't backup a live volume, they first need to snapshot it. Once a snapshot is take, it still can't be backed-up, as backup tools usually work with volumes (that are exposed as file-systems) and not snapshots (which might have backend-specific format). What this means is that currently with cephfs-csi we'd have to clone snapshot data twice:

  1. first time, when restoring the snapshot into a volume,
  2. and second time, when transferring that volume into some backup/archive storage (e.g. object store).

  The backed-up volume will most likely be thrown away after the backup is finished. That's a lot of wasted work for what we originally wanted to do! Having the ability to create volumes from snapshots cheaply would be a big improvement for this use case.

How will we know we have a good solution? (acceptance criteria)

Creating/deleting and mounting/unmounting such read-only volumes should be O(1) regardless of snapshot size.

Alternatives

• User's could simply go to .snap/<snapshot name> by themselves.
  .snap is CephFS-specific detail of how snapshots are exposed. Users / tools may not be aware of this special directory, or it doesn't fit the workflow. The idiomatic way of accessing snapshot contents is by creating a new volume and populating it with snapshot.

Additional context

If you agree with the proposed feature, I can add a PR for a docs/proposals document where we can discuss this in more detail. After that I can work on its implementation.

[Madhu-1]

@gman0 sounds good hope we can see a design doc soon on this one. Also, note we are planning for cephfs mirroring support in cephcsi soon which will directly replicate the volume to remote ceph cluster but looks like this is useful if someone wants to do backup to some other store (object store).

cc @ShyamsundarR @kotreshhr

[gman0]

I'm still working on it but I'll submit it soon.

[pehlert]

Just to add here from a user perspective:

Through the amazing kasten.io, we not only create snapshots but also regularly transfer the data to an S3 store off-site which I think is a very common use-case.

While rbd snapshots and exports are working just fine, creating volumes from even medium-size cephfs snapshots is virtually impossible as the volume creation alone takes 30min+ for <60GB of data, so we'll run into timeouts at that step already.

As I see it this proposal would allow to mount the snapshot read-only which should be near instant? That would be a very useful feature!

[gman0]

@pehlert Thanks for your feedback! We have essentially the same use-case.

As I see it this proposal would allow to mount the snapshot read-only which should be near instant?

Yes, if all goes well, this should be the result.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[taxilian]

This would definitely enable use cases which are very important to me as well -- the benefits for backup solutions would be difficult to overstate.

Users could simply go to .snap/<snapshot name> by themselves. - I was not aware of this and don't know how to use it in kubernetes -- I will definitely be looking into it to see if it can be used as a reasonable solution in the mean time

[taxilian]

It looks like this shouldn't be too difficult to do -- it's even technically possible to do it semi-reasonably using a script, though maybe I'm missing something that would make it easier?

• Look at the VolumeSnapshot you want to back up, get the VolumeSnapshotContent
• Look at the VolumeSnapshotContent you want to use and grab the last 36 characters of the Snapshot Handle (a guid)
• Look inside the PVC in question and list .snap/, find the directory which starts with _csi-snap-${guid} -- that is your read-only mount of the snapshot
• Start a backup pod using .snap/ as the subPath -- I haven't tested this, so not 100% sure it would work.

There are several steps here for which I can't think of a good way to automate, sadly; if there were it would really help a lot. Another thing that could help is if the name of those directories was easier to find, without jumping through three hoops. I assume there is some source for the suffix of the .snap/<???> directory name but I can't figure out what it is.

[haslersn]

This feature is important to me, too. It is important for implementing vmware-tanzu/velero#3544 in an ideomatic way instead of accessing .snap.