Skip to content

Latest commit

 

History

History
74 lines (65 loc) · 2.84 KB

csi-debug.md

File metadata and controls

74 lines (65 loc) · 2.84 KB
Raw

CSI driver troubleshooting guide

Case#1: volume create/delete issue

  • locate csi driver pod
kubectl get po -o wide -n kube-system | grep csi-blob-controller
NAME                                       READY   STATUS    RESTARTS   AGE     IP             NODE
csi-blob-controller-56bfddd689-dh5tk       4/4     Running   0          35s     10.240.0.19    k8s-agentpool-22533604-0
csi-blob-controller-56bfddd689-sl4ll       4/4     Running   0          35s     10.240.0.23    k8s-agentpool-22533604-1
  • get csi driver logs
kubectl logs csi-blob-controller-56bfddd689-dh5tk -c blob -n kube-system > csi-blob-controller.log

note: there could be multiple controller pods, logs can be taken from all of them simultaneously, also with follow (realtime) mode kubectl logs deploy/csi-blob-controller -c blob -f -n kube-system

Case#2: volume mount/unmount failed

  • locate csi driver pod and make sure which pod do tha actual volume mount/unmount
kubectl get po -o wide -n kube-system | grep csi-blob-node
NAME                                       READY   STATUS    RESTARTS   AGE     IP             NODE
csi-blob-node-cvgbs                        3/3     Running   0          7m4s    10.240.0.35    k8s-agentpool-22533604-1
csi-blob-node-dr4s4                        3/3     Running   0          7m4s    10.240.0.4     k8s-agentpool-22533604-0
  • get csi driver logs
kubectl logs csi-blob-node-cvgbs -c blob -n kube-system > csi-blob-node.log

note: to watch logs in realtime from multiple csi-blob-node DaemonSet pods simultaneously, run the command:

kubectl logs daemonset/csi-blob-node -c blob -n kube-system -f

get blobfuse driver version

kubectl exec -it csi-blob-node-fmbqw -n kube-system -c blob -- sh
blobfuse -v
blobfuse 1.2.4

check blobfuse mount on the agent node

mount | grep blobfuse | uniq
  • Troubleshooting blobfuse mount failure on the agent node

troubleshooting connection failure on agent node

  • blobfuse

Blobfuse mount will fail due to incorrect storage account name, key or container name, run below commands to check whether blobfuse mount would work on agent node:

mkdir test
export AZURE_STORAGE_ACCOUNT=
export AZURE_STORAGE_ACCESS_KEY=
# only for sovereign cloud
# export AZURE_STORAGE_BLOB_ENDPOINT=accountname.blob.core.chinacloudapi.cn
blobfuse test --container-name=CONTAINER-NAME --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
  • NFSv3
mkdir /tmp/test
mount -t nfs -o sec=sys,vers=3,nolock accountname.blob.core.windows.net:/accountname/container-name /tmp/test