Backup Azure Managed Disks from multiple Resource Groups

[cholm321]

We have an AKS cluster with velero installed in an enterprise setup.

On the cluster we have
app1 with managed disk in ResourceGroup 1
app2 with managed disk in ResourceGroup 2
app3 with managed disk in ResourceGroup 3

All Resource Groups has the AKS spn as contributor.

We have been poking a little around, but it seem that the ResourceGroup must be specified on velero install.

Have we missed something - is it possible to do all configuration at backup stage?
Or better, let the plugin lookup which ResourceGroup the disks are located by inspecting the pv objects.

[ashish-amarnath]

@cholm321 If I am following this correctly, the issue you are facing is an error during disk information lookup which uses the resource group from the envvars, derived from the cloud-credentials secret, as the resource group for all disks.
https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/velero-plugin-for-microsoft-azure/volume_snapshotter.go#L163

This is, currently, an unsupported capability in Velero. We do have plans of adding support for multiple cloud credentials and I am expecting this to be addressed as a side effect of that.

[ashish-amarnath]

This is the design got supporting multiple credentials.
#2403

[cholm321]

@ashish-amarnath I would think that when Azure Managed Disks are placed across several Azure Resource Groups, Velero should just lookup the DiskURI in the k8s meta data - no need to expose that to the velero cli.

When I do kubectl get pvc I can locate associated pv's by kubectl get pv.

Assume I have a pv named pv-something.
Then kubectl describe pv pv-something gives me

DiskURI:      /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/disks/kubernetes-dynamic-pv-something

Why does Velero just not do that lookup?

[ashish-amarnath]

That metadata is specific to Azure.
The volumesnapshotter interface is designed to be a generic volume snapshotter and keeping it agnostic of kubernetes volumes.

func (c *VolumeSnapshotterGRPCClient) CreateSnapshot(volumeID, volumeAZ string, tags map[string]string) (string, error) {

So the volumesnapshotter is given the necessary info to locate the volume and call the volume provider's snapshot API.

This, unfortunately, makes changing the interface a breaking change across all providers.

[francois-travais]

I've the same issue: my AKS cluster is in a resource group while all the nodes, and therefore the PVC disks, are in another. Velero is stuck searching for the disks in AKS cluster instead of the nodes resource group, even if I change the resource group of the VolumeSnapshotLocation

[dploeger]

Any news about this? It's still valid for Velero helm 2.29.4 / Velero image 1.8.1 and velero plugin for azure v1.4.1 and this makes velero unable to backup dynamic volumes.

[grodzik]

Same here, with latest versions. Any plans to fix this within foreseeable timeframe?

[jkurek1]

Hi guys,
Any update? I have similiar use case.

We have configured storage classes in different resource groups and volumes are provisioned, but during the backup, velero is using only one resource group which is configured in velero credentials.

[dploeger]

@ashish-amarnath Reading this again, I believe it's not about multiple credentials.

With Azure, the thing is that the cluster is in resourcegroup "dev" and AKS automatically creates a second resource group like "MC_dev_myk8scluster_europewest" where it puts all the k8s resource it creates for Azure (like the said managed disks for PVCs).

So maybe it's the fault of the azure plugin for velero trying to search for the disks in the wrong resource group.

[dploeger]

However, this worked in helm chart 2.15.0 with azure plugin 1.1.0.

[dploeger]

Running azure plugin 1.1.0 and up with the most current helm chart verison 2.29.6 yields no errors but instead skips the PVCs:

time="2022-05-23T10:45:14Z" level=info msg="Backing up item" backup=velero/test2 logSource="pkg/backup/item_backupper.go:122" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= resource=persistentvolumes
time="2022-05-23T10:45:14Z" level=info msg="Executing takePVSnapshot" backup=velero/test2 logSource="pkg/backup/item_backupper.go:395" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= resource=persistentvolumes
time="2022-05-23T10:45:14Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/test2 logSource="pkg/backup/item_backupper.go:422" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= persistentVolume=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 resource=persistentvolumes
time="2022-05-23T10:45:14Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/test2 logSource="pkg/backup/item_backupper.go:426" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= persistentVolume=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 resource=persistentvolumes
time="2022-05-23T10:45:14Z" level=info msg="zone info not available in nodeAffinity requirements" backup=velero/test2 logSource="pkg/backup/item_backupper.go:431" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= persistentVolume=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 resource=persistentvolumes
time="2022-05-23T10:45:14Z" level=info msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/test2 logSource="pkg/backup/item_backupper.go:455" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= persistentVolume=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 resource=persistentvolumes volumeSnapshotLocation=default
time="2022-05-23T10:45:14Z" level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/test2 logSource="pkg/backup/item_backupper.go:466" name=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 namespace= persistentVolume=pvc-8426a489-4cb1-441e-a144-33ae7e4bfde4 resource=persistentvolumes

[dploeger]

Ah, sorry. I just overread the documentation stating that you should use the generated resource group instead of the resource group your cluster is in.

[snowerem]

Hi guys,
Any update? We have similar use case - storage classes in different resource groups and volumes are provisioned. Fixing that would be a game changer for Azure backups.

[ywk253100]

To address this issue, we may need to introduce a new version of the plugin interface which needs the support of plugin versioning.

And another choice is to use the CSI plugin instead which is going to be GA in v1.9

[reasonerjt]

@ywk253100 I think in v1.10 timeframe we may verify if CSI plugin can solve the problem. If we can solve it via CSI we may consider closing this issue.

[ywk253100]

I have verified that this case can be resolved by taking a snapshot with the CSI plugin. For more information about how to use CSI, please refer to the doc.

Closing this issue.