Node pool zone changed and embedded nfs failing with mount failed: exit status 32

[dmuley17]

Describe the bug
I recently upgraded AKS k8s version to v1.27.9, which broke our services and all pods using nfs fprovisioner ar failing

Events:
  Type     Reason             Age                 From                Message
  ----     ------             ----                ----                -------
  Warning  FailedScheduling   54m                 default-scheduler   0/5 nodes are available: 5 node(s) were unschedulable. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling..
  Normal   NotTriggerScaleUp  54m                 cluster-autoscaler  pod didn't trigger scale-up:
  Normal   NotTriggerScaleUp  50m                 cluster-autoscaler  pod didn't trigger scale-up:
  Warning  FailedScheduling   50m                 default-scheduler   no nodes available to schedule pods
  Warning  FailedScheduling   49m                 default-scheduler   0/1 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
  Normal   Scheduled          49m                 default-scheduler   Successfully assigned icedq-production/icedq-admin-6465fcc449-p4rk4 to aks-apps-41607481-vmss000002
  Warning  FailedMount        38m (x2 over 43m)   kubelet             Unable to attach or mount volumes: unmounted volumes=[icedq-pv-storage], unattached volumes=[ca-truststore kube-api-access-96q5h icedq-pv-storage]: timed out waiting for the condition
  Warning  FailedMount        29m (x5 over 47m)   kubelet             Unable to attach or mount volumes: unmounted volumes=[icedq-pv-storage], unattached volumes=[icedq-pv-storage ca-truststore kube-api-access-96q5h]: timed out waiting for the condition
  Warning  FailedMount        27m (x3 over 40m)   kubelet             Unable to attach or mount volumes: unmounted volumes=[icedq-pv-storage], unattached volumes=[kube-api-access-96q5h icedq-pv-storage ca-truststore]: timed out waiting for the condition
  Warning  FailedMount        25m (x19 over 47m)  kubelet             MountVolume.SetUp failed for volume "pvc-e13cb902-fc05-4c2e-9a63-b226699735d2" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o nfsvers=4.1 10.4.4.224:/export/pvc-e13cb902-fc05-4c2e-9a63-b226699735d2 /var/lib/kubelet/pods/14497b04-9013-4d39-99fd-5d10d5313374/volumes/kubernetes.io~nfs/pvc-e13cb902-fc05-4c2e-9a63-b226699735d2
Output: mount.nfs: Protocol not supported

Is there any compatibility issue with nfs provisioner ?

[andyzhangx]

can you use vers=4,minorversion=1,sec=sys to replace nfsvers=4.1 mount option in pv?

[dmuley17]

I tried with that, same issue. Also we notice that this issue started across all of my other running cluster where I never initiated the k8s upgrade. was there any Azure's k8s related release on weekends ?

[andyzhangx]

I have tried on an AKS 1.27.9 node, it shows azure file nfs mount works well on the node, can you try nfs mount on the node? and what's the output of whereis mount.nfs on that node?

# mkdir /tmp/test2
root@aks-agentpool-90924120-vmss00000C:/# mount -v -t nfs -o nfsvers=4.1 storageaccount.file.core.windows.net:/storageaccount/pvcn-40b96b33-4d80-453f-9aaa-90f5928817e9 /tmp/test2
mount.nfs: timeout set for Thu Mar  7 13:27:44 2024
mount.nfs: trying text-based options 'nfsvers=4.1,addr=52.239.230.72,clientaddr=10.224.0.10'
root@aks-agentpool-90924120-vmss00000C:/# mount | grep test2
storageaccount.file.core.windows.net:/storageaccount/pvcn-40b96b33-4d80-453f-9aaa-90f5928817e9 on /tmp/test2 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=10.224.0.10,local_lock=none,addr=52.239.230.72)
root@aks-agentpool-90924120-vmss00000C:/# uname -a
Linux aks-agentpool-90924120-vmss00000C 5.15.0-1057-azure #65-Ubuntu SMP Fri Feb 9 18:39:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# k get no -o wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-90924120-vmss00000c   Ready    agent   23h   v1.27.9   10.224.0.10   <none>        Ubuntu 22.04.4 LTS   5.15.0-1057-azure   containerd://1.7.7-1

[andyzhangx]

can you use vers=4,minorversion=1,sec=sys to replace nfsvers=4.1 mount option in pv?

[dmuley17]

I tried with that, same issue. Also we notice that this issue started across all of my other running cluster where I never initiated the k8s upgrade. was there any Azure's k8s related release on weekends ?

[andyzhangx]

I have tried on an AKS 1.27.9 node, it shows azure file nfs mount works well on the node, can you try nfs mount on the node? and what's the output of whereis mount.nfs on that node?

# mkdir /tmp/test2
root@aks-agentpool-90924120-vmss00000C:/# mount -v -t nfs -o nfsvers=4.1 storageaccount.file.core.windows.net:/storageaccount/pvcn-40b96b33-4d80-453f-9aaa-90f5928817e9 /tmp/test2
mount.nfs: timeout set for Thu Mar  7 13:27:44 2024
mount.nfs: trying text-based options 'nfsvers=4.1,addr=52.239.230.72,clientaddr=10.224.0.10'
root@aks-agentpool-90924120-vmss00000C:/# mount | grep test2
storageaccount.file.core.windows.net:/storageaccount/pvcn-40b96b33-4d80-453f-9aaa-90f5928817e9 on /tmp/test2 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=10.224.0.10,local_lock=none,addr=52.239.230.72)
root@aks-agentpool-90924120-vmss00000C:/# uname -a
Linux aks-agentpool-90924120-vmss00000C 5.15.0-1057-azure #65-Ubuntu SMP Fri Feb 9 18:39:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# k get no -o wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-90924120-vmss00000c   Ready    agent   23h   v1.27.9   10.224.0.10   <none>        Ubuntu 22.04.4 LTS   5.15.0-1057-azure   containerd://1.7.7-1