Backup of PVs not possible because of independent disk mode

[embano1]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

Feature

What happened:
It is not possible to backup an independent disk with most backup programs for vSphere, e.g. Veeam. The cloud provider creates PVs in mode "persistent independent". However, the opposite should be the case. Kubelet VMs should be of mode independent (nothing the cloud provider can do here), because they´re usually immutable deployments. PVs contain critical data (state) and as such must be backed up.

Disk creation mode is hard coded in the cloud provider:
https://github.com/kubernetes/kubernetes/blob/7c13d65134f538a50f490ebb3567d08dd0841746/pkg/cloudprovider/providers/vsphere/vclib/virtualmachine.go#L268

What you expected to happen:
Backup PVs with common vSphere backup software, e.g. Veeam

How to reproduce it (as minimally and precisely as possible):
Use a backup software like Veeam and try to backup a VM with an attached PV.

Anything else we need to know?:
What is the reason for making a disk of mode independent, besides excluding it from backup operations? What is the alternative way/ best practice of backing up PVs?

Environment:

• Kubernetes version (use kubectl version): v1.7
• Cloud provider or hardware configuration**:
• OS (e.g. from /etc/os-release): CoreOS
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[BaluDontu]

If we attach the disk as dependent disk to the VM, then any accidental snapshot of VM would create a snapshot of the disk. This means that the disk cannot be detached unless the VM snapshot is removed. Because of this the basic kubernetes workflows like pod rescheduling on other available nodes will not work since there are dependent snapshot disks which are still around.

So inorder to not run into any of these issues we have decided to attach disks as independent disks to VM.

[jsafrane]

Is there any way how to enable backups and keep the volume independent and safe from VM snapshots? It's quite natural that people want to backup their PVs.

[JaredBurck]

We are using CommVault for backups and hitting this same issue. Is there a workaround for being able to take snapshots of our PVs? Any updates or suggestions would be much appreciated!

cc: @eRaid6

[jsafrane]

/sig vmware

@SandeepPissay, since you're assigned, do you have any insights how and when it could be fixed?

[corrieb]

Independent Persistent is necessary for the reasons highlighted above. You don't want the lifecycle of a PV tied to the lifecycle of the node. You particularly don't want the risk of a PV being deleted if a node is decommissioned. vSphere Integrated Containers uses the same VMDK mechanisms and I wrote some public documentation on the topic which may be of help here.

Top-level topic: https://vmware.github.io/vic-product/assets/files/html/1.3/vic_vsphere_admin/backup_vic.html

Specifics about backing up VMDK volumes: https://vmware.github.io/vic-product/assets/files/html/1.3/vic_vsphere_admin/backup_vmdk.html

Restoring VMDK volumes: https://vmware.github.io/vic-product/assets/files/html/1.3/vic_vsphere_admin/restore_vmdk.html

[jsafrane]

@corrieb (or anyone), can you please update Kubernetes docs to have this information?

[corrieb]

I can take a stab at summarizing what I wrote in the other docs. @jsafrane Do you have a sense of where it should go?

[jsafrane]

IMO, You should update https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/ - Kubernetes links it from https://kubernetes.io/docs/setup/pick-right-solution/

And then you can just link it from:

• https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/
• https://kubernetes.io/docs/concepts/storage/volumes/#vspherevolume
• and maybe https://kubernetes.io/docs/concepts/storage/storage-classes/#vsphere

Source code for kubernetes.io/docs in in https://github.com/kubernetes/website, workflow is described at https://kubernetes.io/editdocs/

[SandeepPissay]

@jsafrane, @JaredBurck

VCP creates the PVs(Persistent Volumes) as independent disks as we do not want the PVs to be part of kubernetes node VM snapshot. There are complications if we do so. Also note that these disks cannot be managed independently.

However In vSphere 6.7 release, we can create and manage virtual storage objects independent of VM life cycle and this also has snapshot and backup support. The plan is to refactor the entire VCP code related to volume provisioning to start using virtual storage objects in vSphere. Once this is done, we should be able to backup the persistent volumes.

[mkretzer]

When will this refactoring be done? Will the plugin still be compatible with VSphere < 6.7 then?

[SandeepPissay]

@mkretzer : We are working on it and the plan is to get it out by end of this year. In terms of volume provisioning APIs in k8s, the plugin will be compatible with vSphere <= 6.7