Add a blog for volume group snapshots beta

content/en/blog/_posts/2024-12-11-volume-group-snapshot-beta.md

@@ -0,0 +1,382 @@
+---
+layout: blog
+title: "Kubernetes 1.32: Moving Volume Group Snapshots to Beta"
+date: 2024-12-11
+slug: kubernetes-1-32-volume-group-snapshot-beta
+draft: true
+author: >
+   Xing Yang (VMware by Broadcom)
+---
+
+Volume group snapshots were [introduced](/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/)
+as an Alpha feature with the Kubernetes 1.27 release.
+The recent release of Kubernetes v1.32 moved that support to **beta**.
+The support for volume group snapshots relies on a set of
+[extension APIs for group snapshots](https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html#volume-group-snapshot-apis).
+These APIs allow users to take crash consistent snapshots for a set of volumes.
+Behind the scenes, Kubernetes uses a label selector to group multiple PersistentVolumeClaims
+for snapshotting.
+A key aim is to allow you restore that set of snapshots to new volumes and
+recover your workload based on a crash consistent recovery point.
+
+This new feature is only supported for [CSI](https://kubernetes-csi.github.io/docs/) volume drivers.
+
+## An overview of volume group snapshots
+
+Some storage systems provide the ability to create a crash consistent snapshot of
+multiple volumes. A group snapshot represents _copies_ made from multiple volumes, that
+are taken at the same point-in-time. A group snapshot can be used either to rehydrate
+new volumes (pre-populated with the snapshot data) or to restore existing volumes to
+a previous state (represented by the snapshots).
+
+## Why add volume group snapshots to Kubernetes?
+
+The Kubernetes volume plugin system already provides a powerful abstraction that
+automates the provisioning, attaching, mounting, resizing, and snapshotting of block
+and file storage.
+
+Underpinning all these features is the Kubernetes goal of workload portability:
+Kubernetes aims to create an abstraction layer between distributed applications and
+underlying clusters so that applications can be agnostic to the specifics of the
+cluster they run on and application deployment requires no cluster specific knowledge.
+
+There was already a [VolumeSnapshot](/docs/concepts/storage/volume-snapshots/) API
+that provides the ability to take a snapshot of a persistent volume to protect against
+data loss or data corruption. However, there are other snapshotting functionalities
+not covered by the VolumeSnapshot API.
+
+Some storage systems support consistent group snapshots that allow a snapshot to be
+taken from multiple volumes at the same point-in-time to achieve write order consistency.
+This can be useful for applications that contain multiple volumes. For example,
+an application may have data stored in one volume and logs stored in another volume.
+If snapshots for the data volume and the logs volume are taken at different times,
+the application will not be consistent and will not function properly if it is restored
+from those snapshots when a disaster strikes.
+
+It is true that you can quiesce the application first, take an individual snapshot from
+each volume that is part of the application one after the other, and then unquiesce the
+application after all the individual snapshots are taken. This way, you would get
+application consistent snapshots.
+
+However, sometimes the application quiesce can be so time consuming that you want to do it less frequently,
+or it may not be possible to quiesce an application at all.
+For example, a user may want to run weekly backups with application quiesce
+and nightly backups without application quiesce but with consistent group support which
+provides crash consistency across all volumes in the group.
+
+## Kubernetes APIs for volume group snapshots
+
+Kubernetes' support for _volume group snapshots_ relies on three API kinds that
+are used
+for managing snapshots:
+
+VolumeGroupSnapshot
+: Created by a Kubernetes user (or perhaps by your own automation) to request
+creation of a volume group snapshot for multiple persistent volume claims.
+It contains information about the volume group snapshot operation such as the
+timestamp when the volume group snapshot was taken and whether it is ready to use.
+The creation and deletion of this object represents a desire to create or delete a
+cluster resource (a group snapshot).
+
+VolumeGroupSnapshotContent
+: Created by the snapshot controller for a dynamically created VolumeGroupSnapshot.
+It contains information about the volume group snapshot including the volume group
+snapshot ID.
+This object represents a provisioned resource on the cluster (a group snapshot).
+The VolumeGroupSnapshotContent object binds to the VolumeGroupSnapshot for which it
+was created with a one-to-one mapping.
+
+VolumeGroupSnapshotClass
+: Created by cluster administrators to describe how volume group snapshots should be
+created, including the driver information, the deletion policy, etc.
+
+These three API kinds are defined as
+[CustomResourceDefinitions](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
+(CRDs).
+These CRDs must be installed in a Kubernetes cluster for a CSI Driver to support
+volume group snapshots.
+
+## What components are needed to support volume group snapshots
+
+Volume group snapshots are implemented in the
+[external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter) repository.
+Implementing volume group snapshots meant adding or changing several components:
+
+* Added new CustomResourceDefinitions for VolumeGroupSnapshot and two supporting APIs.
+* Volume group snapshot controller logic is added to the common snapshot controller.
+* Adding logic to make CSI calls into the snapshotter sidecar controller.
+
+The volume snapshot controller and CRDs are deployed once per
+cluster, while the sidecar is bundled with each CSI driver.
+
+Therefore, it makes sense to deploy the volume snapshot controller and CRDs as a cluster addon.
+
+The Kubernetes project recommends that Kubernetes distributors
+bundle and deploy the volume snapshot controller and CRDs as part
+of their Kubernetes cluster management process (independent of any CSI Driver).
+
+## What's new in Beta?
+
+* The VolumeGroupSnapshot feature in CSI spec moved to GA in the [v1.11.0 release](https://github.com/container-storage-interface/spec/releases/tag/v1.11.0).
+
+* The snapshot validation webhook was deprecated in external-snapshotter v8.0.0 and it is now removed.
+  Most of the validation webhook logic was added as validation rules into the CRDs.
+  Minimum required Kubernetes version is 1.25 for these validation rules.
+  One thing in the validation webhook not moved to CRDs is the prevention of creating
+  multiple default volume snapshot classes and multiple default volume group snapshot classes
+  for the same CSI driver.
+  With the removal of the validation webhook, an error will still be raised when dynamically
+  provisioning a VolumeSnapshot or VolumeGroupSnapshot when multiple default volume snapshot
+  classes or multiple default volume group snapshot classes for the same CSI driver exist.
+
+* The `enable-volumegroup-snapshot` flag in the snapshot-controller and the CSI snapshotter
+  sidecar has been replaced by a feature gate.
+  Since VolumeGroupSnapshot is a new API, the feature moves to Beta but the feature gate is
+  disabled by default.
+  To use this feature, enable the feature gate by adding the flag `--feature-gates=CSIVolumeGroupSnapshot=true`
+  when starting the snapshot-controller and the CSI snapshotter sidecar.
+
+* The logic to dynamically create the VolumeGroupSnapshot and its corresponding individual
+  VolumeSnapshot and VolumeSnapshotContent objects are moved from the CSI snapshotter to the common
+  snapshot-controller.
+  New RBAC rules are added to the common snapshot-controller and some RBAC rules are removed from
+  the CSI snapshotter sidecar accordingly.
+
+## How do I use Kubernetes volume group snapshots
+
+### Creating a new group snapshot with Kubernetes
+
+Once a VolumeGroupSnapshotClass object is defined and you have volumes you want to
+snapshot together, you may request a new group snapshot by creating a VolumeGroupSnapshot
+object.
+
+The source of the group snapshot specifies whether the underlying group snapshot
+should be dynamically created or if a pre-existing VolumeGroupSnapshotContent
+should be used.
+
+A pre-existing VolumeGroupSnapshotContent is created by a cluster administrator.
+It contains the details of the real volume group snapshot on the storage system which
+is available for use by cluster users.
+
+One of the following members in the source of the group snapshot must be set.
+
+* `selector` - a label query over PersistentVolumeClaims that are to be grouped
+  together for snapshotting. This selector will be used to match the label
+  added to a PVC.
+* `volumeGroupSnapshotContentName` - specifies the name of a pre-existing
+  VolumeGroupSnapshotContent object representing an existing volume group snapshot.
+
+#### Dynamically provision a group snapshot
+
+In the following example, there are two PVCs.
+
+```console
+NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
+pvc-0   Bound    pvc-6e1f7d34-a5c5-4548-b104-01e72c72b9f2   100Mi      RWO            csi-hostpath-sc   <unset>                 2m15s
+pvc-1   Bound    pvc-abc640b3-2cc1-4c56-ad0c-4f0f0e636efa   100Mi      RWO            csi-hostpath-sc   <unset>                 2m7s
+```
+
+Label the PVCs.
+```console
+% kubectl label pvc pvc-0 group=myGroup
+persistentvolumeclaim/pvc-0 labeled
+
+% kubectl label pvc pvc-1 group=myGroup
+persistentvolumeclaim/pvc-1 labeled
+```
+
+For dynamic provisioning, a selector must be set so that the snapshot controller can find PVCs
+with the matching labels to be snapshotted together.
+
+```yaml
+apiVersion: groupsnapshot.storage.k8s.io/v1beta1
+kind: VolumeGroupSnapshot
+metadata:
+  name: snapshot-daily-20241217
+  namespace: demo-namespace
+spec:
+  volumeGroupSnapshotClassName: csi-groupSnapclass
+  source:
+    selector:
+      matchLabels:
+        group: myGroup
+```
+
+In the VolumeGroupSnapshot spec, a user can specify the VolumeGroupSnapshotClass which
+has the information about which CSI driver should be used for creating the group snapshot.
+A VolumGroupSnapshotClass is required for dynamic provisioning.
+
+```yaml
+apiVersion: groupsnapshot.storage.k8s.io/v1beta1
+kind: VolumeGroupSnapshotClass
+metadata:
+  name: csi-groupSnapclass
+  annotations:
+    kubernetes.io/description: "Example group snapshot class"
+driver: example.csi.k8s.io
+deletionPolicy: Delete
+```
+
+As a result of the volume group snapshot creation, a corresponding VolumeGroupSnapshotContent
+object will be created with a volumeGroupSnapshotHandle pointing to a resource on the storage
+system.
+
+Two individual volume snapshots will be created as part of the volume group snapshot creation.
+
+```console
+NAME                                                                        READYTOUSE   SOURCEPVC   RESTORESIZE   SNAPSHOTCONTENT                                                                AGE
+snapshot-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0   true         pvc-0       100Mi         snapcontent-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0   16m
+snapshot-da577d76bd2106c410616b346b2e72440f6ec7b12a75156263b989192b78caff   true         pvc-1       100Mi         snapcontent-da577d76bd2106c410616b346b2e72440f6ec7b12a75156263b989192b78caff   16m
+```
+
+#### Importing an existing group snapshot with Kubernetes
+
+To import a pre-existing volume group snapshot into Kubernetes, you must also import
+the corresponding individual volume snapshots.
+
+Identify the individual volume snapshot handles, manually construct a
+VolumeSnapshotContent object first, then create a VolumeSnapshot object pointing to
+the VolumeSnapshotContent object. Repeat this for every individual volume snapshot.
+
+Then manually create a VolumeGroupSnapshotContent object, specifying the
+volumeGroupSnapshotHandle and individual volumeSnapshotHandles already existing
+on the storage system.
+
+```yaml
+apiVersion: groupsnapshot.storage.k8s.io/v1beta1
+kind: VolumeGroupSnapshotContent
+metadata:
+  name: static-group-content
+spec:
+  deletionPolicy: Delete
+  driver: hostpath.csi.k8s.io
+  source:
+    groupSnapshotHandles:
+      volumeGroupSnapshotHandle: e8779136-a93e-11ef-9549-66940726f2fd
+      volumeSnapshotHandles:
+      - e8779147-a93e-11ef-9549-66940726f2fd
+      - e8783cd0-a93e-11ef-9549-66940726f2fd
+  volumeGroupSnapshotRef:
+    name: static-group-snapshot
+    namespace: demo-namespace
+```
+
+After that create a VolumeGroupSnapshot object pointing to the VolumeGroupSnapshotContent
+object.
+
+```yaml
+apiVersion: groupsnapshot.storage.k8s.io/v1beta1
+kind: VolumeGroupSnapshot
+metadata:
+  name: static-group-snapshot
+  namespace: demo-namespace
+spec:
+  source:
+    volumeGroupSnapshotContentName: static-group-content
+```
+
+### How to use group snapshot for restore in Kubernetes
+
+At restore time, the user can request a new PersistentVolumeClaim to be created from
+a VolumeSnapshot object that is part of a VolumeGroupSnapshot. This will trigger
+provisioning of a new volume that is pre-populated with data from the specified
+snapshot. The user should repeat this until all volumes are created from all the
+snapshots that are part of a group snapshot.
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: examplepvc-restored-2024-12-17
+  namespace: demo-namespace
+spec:
+  storageClassName: example-foo-nearline
+  dataSource:
+    name: snapshot-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0
+    kind: VolumeSnapshot
+    apiGroup: snapshot.storage.k8s.io
+  accessModes:
+    - ReadWriteOncePod
+  resources:
+    requests:
+      storage: 100Mi # must be enough storage to fit the existing snapshot
+```
+
+## As a storage vendor, how do I add support for group snapshots to my CSI driver?
+
+To implement the volume group snapshot feature, a CSI driver **must**:
+
+* Implement a new group controller service.
+* Implement group controller RPCs: `CreateVolumeGroupSnapshot`, `DeleteVolumeGroupSnapshot`, and `GetVolumeGroupSnapshot`.
+* Add group controller capability `CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT`.
+
+See the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md)
+and the [Kubernetes-CSI Driver Developer Guide](https://kubernetes-csi.github.io/docs/)
+for more details.
+
+As mentioned earlier, it is strongly recommended that Kubernetes distributors
+bundle and deploy the volume snapshot controller and CRDs as part
+of their Kubernetes cluster management process (independent of any CSI Driver).
+
+As part of this recommended deployment process, the Kubernetes team provides a number of
+sidecar (helper) containers, including the
+[external-snapshotter sidecar container](https://kubernetes-csi.github.io/docs/external-snapshotter.html)
+which has been updated to support volume group snapshot.
+
+The external-snapshotter watches the Kubernetes API server for
+VolumeGroupSnapshotContent objects, and triggers `CreateVolumeGroupSnapshot` and
+`DeleteVolumeGroupSnapshot` operations against a CSI endpoint.
+
+## What are the limitations?
+
+The beta implementation of volume group snapshots for Kubernetes has the following limitations:
+
+* Does not support reverting an existing PVC to an earlier state represented by
+  a snapshot (only supports provisioning a new volume from a snapshot).
+* No application consistency guarantees beyond any guarantees provided by the storage system
+  (e.g. crash consistency). See this [doc](https://github.com/kubernetes/community/blob/30d06f49fba22273f31b3c616b74cf8745c19b3d/wg-data-protection/data-protection-workflows-white-paper.md#quiesce-and-unquiesce-hooks)
+  for more discussions on application consistency.
+
+## What’s next?
+
+Depending on feedback and adoption, the Kubernetes project plans to push the volume
+group snapshot implementation to general availability (GA) in a future release.
+
+## How can I learn more?
+
+- The [design spec](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot)
+  for the volume group snapshot feature.
+- The [code repository](https://github.com/kubernetes-csi/external-snapshotter) for volume group
+  snapshot APIs and controller.
+- CSI [documentation](https://kubernetes-csi.github.io/docs/) on the group snapshot feature.
+
+## How do I get involved?
+
+This project, like all of Kubernetes, is the result of hard work by many contributors
+from diverse backgrounds working together. On behalf of SIG Storage, I would like to
+offer a huge thank you to the contributors who stepped up these last few quarters
+to help the project reach beta:
+
+* Ben Swartzlander ([bswartz](https://github.com/bswartz))
+* Cici Huang ([cici37](https://github.com/cici37))
+* Hemant Kumar ([gnufied](https://github.com/gnufied))
+* James Defelice ([jdef](https://github.com/jdef))
+* Jan Šafránek ([jsafrane](https://github.com/jsafrane))
+* Madhu Rajanna ([Madhu-1](https://github.com/Madhu-1))
+* Manish M Yathnalli ([manishym](https://github.com/manishym))
+* Michelle Au ([msau42](https://github.com/msau42))
+* Niels de Vos ([nixpanic](https://github.com/nixpanic))
+* Leonardo Cecchi ([leonardoce](https://github.com/leonardoce))
+* Rakshith R ([Rakshith-R](https://github.com/Rakshith-R))
+* Raunak Shah ([RaunakShah](https://github.com/RaunakShah))
+* Saad Ali ([saad-ali](https://github.com/saad-ali))
+* Xing Yang ([xing-yang](https://github.com/xing-yang))
+* Yati Padia ([yati1998](https://github.com/yati1998))
+
+For those interested in getting involved with the design and development of CSI or
+any part of the Kubernetes Storage system, join the
+[Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG).
+We always welcome new contributors.
+
+We also hold regular [Data Protection Working Group meetings](https://github.com/kubernetes/community/tree/master/wg-data-protection).
+New attendees are welcome to join our discussions.