the disk is currently being detached or the last detach operation failed

[vishnoisuresh]

What happened:
Pods are stuck at Init or ContainerCreating Phase

 NAME                                             READY     STATUS              RESTARTS   AGE
jenkins-85cc78bbc6-xckkg                         0/1       Init:0/1            0          11m
jenkins-x-chartmuseum-55c9598b56-gdlwt           1/1       Running             0          1h
jenkins-x-docker-registry-7b56b4f555-l6hcw       0/1       ContainerCreating   0          1h
jenkins-x-heapster-65fd697bb-g5g6n               2/2       Running             0          1h
jenkins-x-mongodb-68bfcdf8fc-vrms9               0/1       ContainerCreating   0          1h
jenkins-x-monocular-api-8bdb8d985-ndggq          0/1       CrashLoopBackOff    24         1h
jenkins-x-monocular-prerender-5848c74fdc-58ptl   1/1       Running             0          1h
jenkins-x-monocular-ui-55545676cd-vflh5          1/1       Running             0          1h
jenkins-x-nexus-8655db64f7-jzbnq                 0/1       ContainerCreating   0          1h
pipelinecontroller-6b5b8d9f47-qkh2p              1/1       Running             0          1h

What you expected to happen: Pods should be up and running
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Events:
  Type     Reason              Age               From                               Message
  ----     ------              ----              ----                               -------
  Normal   Scheduled           7m                default-scheduler                  Successfully assigned jx/jenkins-85cc78bbc6-xckkg to aks-agentpool-33355906-0
  Warning  FailedMount         1m (x3 over 5m)   kubelet, aks-agentpool-33355906-0  Unable to mount volumes for pod "jenkins-85cc78bbc6-xckkg_jx(2db2ef1f-a54f-11e8-8bc7-eafd0b6a09c8)": timeout expired waiting for volumes to attach or mount for pod "jx"/"jenkins-85cc78bbc6-xckkg". list of unmounted volumes=[jenkins-home]. list of unattached volumes=[jenkins-config plugin-dir secrets-dir jenkins-home jenkins-token-cv7x4]
  Warning  FailedAttachVolume  19s (x4 over 6m)  attachdetach-controller            AttachVolume.Attach failed for volume "pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8" : Attach volume "kubernetes-dynamic-pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8" to instance "/subscriptions/c882f830-d9b7-4d5f-bffd-acc5670ba585/resourceGroups/MC_kookoo_*****_westeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33355906-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: Service returned an error. Status=<nil> Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'kubernetes-dynamic-pvc-9291b057-a52d-11e8-8bc7-eafd0b6a09c8' to VM 'aks-agentpool-33355906-0' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again." Target="dataDisks"

Location: West europe

kubectl describe pvc jenkins

Name:          jenkins
Namespace:     jx
StorageClass:  default
Status:        Bound
Volume:        pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8
Labels:        app=jenkins
               chart=jenkins-0.10.29
               heritage=Tiller
               release=jenkins-x
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      30Gi
Access Modes:  RWO
Events:        <none>

[jskulavik]

Hi @vishnoisuresh,

Are you still seeing this issue? It should be resolved soon, but if not, you may need to restart the affected pods

[vishnoisuresh]

@jskulavik , Yea I am still having the issue since morning, I have restarted the pods quite a lot of times. I can try it again. Thanks

[vishnoisuresh]

Yea It is still here

Events:
  Type     Reason              Age   From                     Message
  ----     ------              ----  ----                     -------
  Normal   Scheduled           1m    default-scheduler        Successfully assigned jx/jenkins-85cc78bbc6-7c2k9 to aks-agentpool-33355906-2
  Warning  FailedAttachVolume  0s    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8" : Attach volume "kubernetes-dynamic-pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8" to instance "/subscriptions/c882f830-d9b7-4d5f-bffd-acc5670ba585/resourceGroups/MC_kookoo_*****_westeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33355906-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: Service returned an error. Status=<nil> Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'kubernetes-dynamic-pvc-929b0714-a52d-11e8-8bc7-eafd0b6a09c8' to VM 'aks-agentpool-33355906-2' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again." Target="dataDisks"

[jskulavik]

Hi @vishnoisuresh,

Please file a support ticket via the Azure portal, referencing this issue.

[andyzhangx]

@vishnoisuresh could you check VM aks-agentpool-33355906-0 and disk kubernetes-dynamic-pvc-9291b057-a52d-11e8-8bc7-eafd0b6a09c8 status? One disk could only be used by one pod.
And per my experience it's usually caused by vm status issue, you could fix that by running az vm update -g <group> -n <name>

you could find more details here:
https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md#1-disk-attach-error

[vishnoisuresh]

@andyzhangx Thanks, I will try the solution that provided above links, nevertheless, some of the pods are up and running except with one pod with the same issue.

 kubectl get pods
NAME                                             READY     STATUS      RESTARTS   AGE
jenkins-85cc78bbc6-66d42                         0/1       Init:0/1    0          9m
jenkins-x-chartmuseum-55c9598b56-tgt5v           1/1       Running     0          1d
jenkins-x-docker-registry-7b56b4f555-fpfzj       1/1       Running     0          1d
jenkins-x-gc-activities-1534951800-sk7md         0/1       Completed   0          3h
jenkins-x-gc-activities-1534960800-sjmxd         0/1       Completed   0          1h
jenkins-x-gc-activities-1534962600-kkn5d         0/1       Completed   0          44m
jenkins-x-gc-previews-1534939200-9nd4q           0/1       Completed   0          7h
jenkins-x-gc-previews-1534950000-78sxg           0/1       Completed   0          4h
jenkins-x-gc-previews-1534960800-wgz7n           0/1       Completed   0          1h
jenkins-x-heapster-65fd697bb-rzbhp               2/2       Running     0          1d
jenkins-x-mongodb-68bfcdf8fc-vmllv               1/1       Running     1          1d
jenkins-x-monocular-api-8b859894c-6mj25          1/1       Running     308        1d
jenkins-x-monocular-prerender-5848c74fdc-qgkch   1/1       Running     0          1d
jenkins-x-monocular-ui-7fb574b54d-c556n          1/1       Running     0          1d
jenkins-x-nexus-8655db64f7-hwzr4                 1/1       Running     0          1d
pipelinecontroller-6b5b8d9f47-dpxxf              1/1       Running     0          1d

[vishnoisuresh]

After upgrading the cluster, Issue was resolved. Hopefully, AKS team will do something about it.

[dcvtruong]

@andyzhangx, Getting this issue in k8s v1.11.5. The above solution did not help. Any ideas?

https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md#1-disk-attach-error

	Events:
	  Type     Reason              Age               From                               Message
	  ----     ------              ----              ----                               -------
	  Warning  FailedScheduling    2m                default-scheduler                  pod has unbound PersistentVolumeClaims
	  Normal   Scheduled           2m                default-scheduler                  Successfully assigned monitoring/prometheus-server-8df6f584d-gmmhz to aks-agentpool-39166876-0
	  Warning  FailedAttachVolume  2m (x3 over 2m)   attachdetach-controller            AttachVolume.Attach failed for volume "pvc-39799346-235f-11e9-b5cb-0a58ac1f1ba8" : Attach volume "kubernetes-dynamic-pvc-39799346-235f-11e9-b5cb-0a58ac1f1ba8" to instance "/subscriptions/4d1ddb70-47ed-4e3f-9d14-52d4cabf14b9/resourceGroups/MC_ARK2-PLATFORM-WESTEUROPE-OPS_ark2aksops-1_westeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-39166876-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="A disk at LUN 2 already exists." Target="dataDisk.lun"

[andyzhangx]

@dcvtruong
I think it's related to this issue in some race condition: https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md#14-azure-disk-attachdetach-failure-mount-issue-io-error, and it was fixed in v1.11.6, to workaround in prior version, could you delete that pod and let the k8s controller-manager try again.

[pkrishnath]

I am having the same issue on AKS with v1.12.6

[andyzhangx]

@pkrishnath
could you open a new issue and provide details like:

1. k8s version
2. azure region
3. kubectl describe po POD-NAME
4. how many disks mouting into one VM in parallel
5. vm av set or vmss

[jskulavik]

Hi @vishnoisuresh,

Are you still seeing this issue? It should be resolved soon, but if not, you may need to restart the affected pods

[vishnoisuresh]

@jskulavik , Yea I am still having the issue since morning, I have restarted the pods quite a lot of times. I can try it again. Thanks

[vishnoisuresh]

Yea It is still here

Events:
  Type     Reason              Age   From                     Message
  ----     ------              ----  ----                     -------
  Normal   Scheduled           1m    default-scheduler        Successfully assigned jx/jenkins-85cc78bbc6-7c2k9 to aks-agentpool-33355906-2
  Warning  FailedAttachVolume  0s    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8" : Attach volume "kubernetes-dynamic-pvc-ac267c27-a541-11e8-8bc7-eafd0b6a09c8" to instance "/subscriptions/c882f830-d9b7-4d5f-bffd-acc5670ba585/resourceGroups/MC_kookoo_*****_westeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33355906-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: Service returned an error. Status=<nil> Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'kubernetes-dynamic-pvc-929b0714-a52d-11e8-8bc7-eafd0b6a09c8' to VM 'aks-agentpool-33355906-2' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again." Target="dataDisks"

[jskulavik]

Hi @vishnoisuresh,

Please file a support ticket via the Azure portal, referencing this issue.

[andyzhangx]

@vishnoisuresh could you check VM aks-agentpool-33355906-0 and disk kubernetes-dynamic-pvc-9291b057-a52d-11e8-8bc7-eafd0b6a09c8 status? One disk could only be used by one pod.
And per my experience it's usually caused by vm status issue, you could fix that by running az vm update -g <group> -n <name>

you could find more details here:
https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md#1-disk-attach-error

[vishnoisuresh]

@andyzhangx Thanks, I will try the solution that provided above links, nevertheless, some of the pods are up and running except with one pod with the same issue.

 kubectl get pods
NAME                                             READY     STATUS      RESTARTS   AGE
jenkins-85cc78bbc6-66d42                         0/1       Init:0/1    0          9m
jenkins-x-chartmuseum-55c9598b56-tgt5v           1/1       Running     0          1d
jenkins-x-docker-registry-7b56b4f555-fpfzj       1/1       Running     0          1d
jenkins-x-gc-activities-1534951800-sk7md         0/1       Completed   0          3h
jenkins-x-gc-activities-1534960800-sjmxd         0/1       Completed   0          1h
jenkins-x-gc-activities-1534962600-kkn5d         0/1       Completed   0          44m
jenkins-x-gc-previews-1534939200-9nd4q           0/1       Completed   0          7h
jenkins-x-gc-previews-1534950000-78sxg           0/1       Completed   0          4h
jenkins-x-gc-previews-1534960800-wgz7n           0/1       Completed   0          1h
jenkins-x-heapster-65fd697bb-rzbhp               2/2       Running     0          1d
jenkins-x-mongodb-68bfcdf8fc-vmllv               1/1       Running     1          1d
jenkins-x-monocular-api-8b859894c-6mj25          1/1       Running     308        1d
jenkins-x-monocular-prerender-5848c74fdc-qgkch   1/1       Running     0          1d
jenkins-x-monocular-ui-7fb574b54d-c556n          1/1       Running     0          1d
jenkins-x-nexus-8655db64f7-hwzr4                 1/1       Running     0          1d
pipelinecontroller-6b5b8d9f47-dpxxf              1/1       Running     0          1d

[vishnoisuresh]

After upgrading the cluster, Issue was resolved. Hopefully, AKS team will do something about it.

[dcvtruong]

@andyzhangx, Getting this issue in k8s v1.11.5. The above solution did not help. Any ideas?

https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md#1-disk-attach-error

	Events:
	  Type     Reason              Age               From                               Message
	  ----     ------              ----              ----                               -------
	  Warning  FailedScheduling    2m                default-scheduler                  pod has unbound PersistentVolumeClaims
	  Normal   Scheduled           2m                default-scheduler                  Successfully assigned monitoring/prometheus-server-8df6f584d-gmmhz to aks-agentpool-39166876-0
	  Warning  FailedAttachVolume  2m (x3 over 2m)   attachdetach-controller            AttachVolume.Attach failed for volume "pvc-39799346-235f-11e9-b5cb-0a58ac1f1ba8" : Attach volume "kubernetes-dynamic-pvc-39799346-235f-11e9-b5cb-0a58ac1f1ba8" to instance "/subscriptions/4d1ddb70-47ed-4e3f-9d14-52d4cabf14b9/resourceGroups/MC_ARK2-PLATFORM-WESTEUROPE-OPS_ark2aksops-1_westeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-39166876-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="A disk at LUN 2 already exists." Target="dataDisk.lun"

[andyzhangx]

@dcvtruong
I think it's related to this issue in some race condition: https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md#14-azure-disk-attachdetach-failure-mount-issue-io-error, and it was fixed in v1.11.6, to workaround in prior version, could you delete that pod and let the k8s controller-manager try again.

[pkrishnath]

I am having the same issue on AKS with v1.12.6

[andyzhangx]

@pkrishnath
could you open a new issue and provide details like:

1. k8s version
2. azure region
3. kubectl describe po POD-NAME
4. how many disks mouting into one VM in parallel
5. vm av set or vmss