Cannot mount with AzureStorageIdentityClientID, but does work with clientID

[cedricve]

Thanks for your great work here @andyzhangx. We have been using Workload Identity in our AKS cluster. The mounting using the clientID does work properly.

csi:
  driver: blob.csi.azure.com
  volumeHandle: pv-xxx
  volumeAttributes:
    resourceGroup: rsg-xxx
    storageAccount: xxxx
    containerName: nokey
    protocol: fuse
    clientID: "xxx-xxx-xxx-xxx-xxx"

However when using the AzureStorageIdentity and more in particular following CSI parameters:

• AzureStorageIdentityClientID
• AzureStorageIdentityObjectID
• AzureStorageIdentityResourceID

Using following config:

csi:
  driver: blob.csi.azure.com
  volumeHandle: pv-xxx-xxx
  volumeAttributes:
    resourceGroup: rsg-xxx-xxx-d-xxx
    storageAccount: xxx
    containerName: nokey
    protocol: fuse
    AzureStorageIdentityResourceID : "/subscriptions/xxx-xxx-xxx-xxx-xxxx/resourceGroups/rsg-xxx-xxx-d-xxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mid-xxx-xxx-d-xxx-xxx"

We get following error:

88s         Warning   FailedMount         pod/xxx-xxx-xxx    MountVolume.MountDevice failed for volume "pv-xxx" : rpc error: code = Internal desc = no key for storage account(xxx) under resource group(rsg-xxx-xxx-d-xxx), err Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client 'xxx-xxx-xxx-xxx-xxx' with object id 'xxx-xxx-xxx-xxx-xxx' does not have authorization to perform action 'Microsoft.Storage/storageAccounts/listKeys/action' over scope '/subscriptions/xxx-xxx-xxx-xxx-xxx/resourceGroups/rsg-xxx-xxx-d-xxx/providers/Microsoft.Storage/storageAccounts/xxx' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

It looks like it's ignoring the AzureStorageIdentityResourceID, and skilling the workload identity when not using the client_id directly. Any thoughts?

[andyzhangx]

one of the 3 parameters is used together with AzureStorageAuthType: MSI parameter, like this:

kind: StorageClass
metadata:
  name: blob-fuse
provisioner: blob.csi.azure.com
parameters:
  skuName: Premium_LRS 
  protocol: fuse
  resourceGroup: EXISTING_RESOURCE_GROUP_NAME   # optional, node resource group by default if it's not provided
  storageAccount: EXISTING_STORAGE_ACCOUNT_NAME # optional, a new account will be created if it's not provided
  containerName: EXISTING_CONTAINER_NAME  # optional, a new container will be created if it's not provided
  AzureStorageAuthType: MSI
  AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"

[andyzhangx]

one of the 3 parameters is used together with AzureStorageAuthType: MSI parameter, like this:

kind: StorageClass
metadata:
  name: blob-fuse
provisioner: blob.csi.azure.com
parameters:
  skuName: Premium_LRS 
  protocol: fuse
  resourceGroup: EXISTING_RESOURCE_GROUP_NAME   # optional, node resource group by default if it's not provided
  storageAccount: EXISTING_STORAGE_ACCOUNT_NAME # optional, a new account will be created if it's not provided
  containerName: EXISTING_CONTAINER_NAME  # optional, a new container will be created if it's not provided
  AzureStorageAuthType: MSI
  AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"

the above example is blobfuse mount with managed identity, the workload identity support (without account key) is supported by #1798 in v1.25.4 and v1.26.1, would be rolled out in AKS April release.

[andyzhangx]

• mount with managed identity: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/deploy/example/blobfuse-mi
• mount with workload identity: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/workload-identity-static-pv-mount.md

[mathieu-clnk]

Thank you @andyzhangx for your update on the driver paramater. This is now clear which parameter needs to be used with the workload identity.
As the workload identity is using an Azure Managed Identity (and a federated identity), I wrongly assumed that these parameters could also be used.

[Jackbennett]

I'm on aks 1.30 as of now and the daemonset csi-blob-node is Image: mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.24.9

@andyzhangx What exactly is the April update containing support for MSI? Baring in mind k8s 1.28/9 also just got made LTS.

Edit* After Andyzhangx replied - I used the added role to the cluster identity for AzureStorageIdentityClientID instead of my user defined, I didn't want to update the cluster managed identity to my custom one, I've no idea what implication that has not being in the same RG as the cluster itself.

[Poltergeisen]

im still not able to make this work for some reason.

[andyzhangx]

I'm on aks 1.30 as of now and the daemonset csi-blob-node is Image: mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.24.9

@andyzhangx What exactly is the April update containing support for MSI? Baring in mind k8s 1.28/9 also just got made LTS.

@Jackbennett April release is only for workload identity support (without using account key), pls follow this guide for MSI auth support, that does not require April release: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/deploy/example/blobfuse-mi

[andyzhangx]

im still not able to make this work for some reason.

@Poltergeisen pls provide more details about the error msg, e.g. csi driver logs on the node: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed, that would help troubleshooting the issue.

[Poltergeisen]

@andyzhangx I did figure out how to get workload id working, but i had to use the full resource ID to make it happen.

My cluster is in a different sub than my storage account and it seemed like it wasnt able to use the client ID property or the other fuse storage client ID properly. I can try to get more info in the morning as I'm not at my pc.

The logs would keep showing blank subscription id's even though I had that property set on the volume attributes.

I didn't get much output from the fuse pod itself either