Failure cleaning up target space: unlinkat /data/.snapshot: read-only file system

[k8scoder192]

What happened:
Deployed the DataVolume below. Importer pod failed with:

Failure cleaning up target space: unlinkat /data/.snapshot: read-only file system

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: centos
  namespace: default
spec:
  source:
    http:
      url: "http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2"
  pvc:
    storageClassName: trident-csi-volume
    volumeMode: Filesystem
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi

NetApp NFS storage backend is being used

What you expected to happen:
Expected the centos qcow2 to be pulled down and populated into the PVC created by the DV

How to reproduce it (as minimally and precisely as possible):
Netapp NFS storage backend
provisioner: csi.trident.netapp.io

Additional context:
Importer Pod Log

I0309 20:39:17.733967       1 importer.go:104] Starting importer
I0309 20:39:17.734528       1 importer.go:171] begin import process
I0309 20:39:17.843118       1 data-processor.go:379] Calculating available size
I0309 20:39:17.843172       1 data-processor.go:391] Checking out file system volume size.
I0309 20:39:17.843691       1 data-processor.go:399] Request image size not empty.
I0309 20:39:17.843721       1 data-processor.go:404] Target size 10737090560.
I0309 20:39:17.844995       1 util.go:39] deleting file: /data/.snapshot
E0309 20:39:17.846076       1 util.go:42] Unable to delete file: /data/.snapshot, unlinkat /data/.snapshot: read-only file system
E0309 20:39:17.846109       1 importer.go:180] unlinkat /data/.snapshot: read-only file system
Failure cleaning up target space
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
        pkg/importer/data-processor.go:181
main.handleImport
        cmd/cdi-importer/importer.go:177
main.main
        cmd/cdi-importer/importer.go:143
runtime.main
        GOROOT/src/runtime/proc.go:250
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1571

Worker node NFS mount info

root@worker1:/var/lib/kubelet/pods/4db0...0e1e5/volumes/kubernetes.io~csi/pvc-....14bb/mount# ls -ltr
total 8
drwxrwsrwx 2 root sgx  4096 Mar  9 20:38 ./
drwxr-x--- 3 root root   80 Mar  9 20:38 ../
drwxrwxrwx 2 root root 4096 Mar  9 20:38 .snapshot/


root@worker1:/var/lib/kubelet/pods/4db0...0e1e5/volumes/kubernetes.io~csi/pvc-...-14bb/mount# df -hT .snapshot/
Filesystem                                                    Type  Size  Used Avail Use% Mounted on
8:/trident_pvc_...__14bb nfs   539M     0  539M   0% /var/lib/kubelet/pods/.../volumes/kubernetes.io~csi/pvc-62acad5c-.../mount


root@worker1:~# cat /proc/self/mountinfo|grep nfs|grep 14bb
5305 30 0:1141 / /var/lib/kubelet/pods/4db0...0e1e5/volumes/kubernetes.io~csi/pvc-...-14bb/mount rw,relatime shared:3390 - nfs 192.168.0.38:/trident_pvc_..._14bb rw,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.38,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=192.168.0.38

CDI CR (cloneStrategyOverride: copy)

apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
  annotations:
    cdi.kubevirt.io/configAuthority: ""
  creationTimestamp: "2023-03-01T20:33:51Z"
  finalizers:
  - operator.cdi.kubevirt.io
  generation: 9
  name: cdi
  resourceVersion: "5967678"
  uid: d8...354
spec:
  cloneStrategyOverride: copy
  imagePullPolicy: Always
  infra:
    nodeSelector:
      kubernetes.io/os: linux
    tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
  workload:
    nodeSelector:
      kubernetes.io/os: linux
status:
  conditions:
  - lastHeartbeatTime: "2023-03-01T20:34:57Z"
    lastTransitionTime: "2023-03-01T20:34:57Z"
    message: Deployment Completed
    reason: DeployCompleted
    status: "True"
    type: Available
  - lastHeartbeatTime: "2023-03-01T20:34:57Z"
    lastTransitionTime: "2023-03-01T20:34:57Z"
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2023-03-03T00:08:43Z"
    lastTransitionTime: "2023-03-03T00:08:43Z"
    status: "False"
    type: Degraded
  observedVersion: v1.54.0
  operatorVersion: v1.54.0
  phase: Deployed
  targetVersion: v1.54.0

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): v1.54.0
• Kubernetes version (use kubectl version): v1.23.10
• DV specification: N/A
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): Ubuntu 22.04.1 LTS
• Kernel (e.g. uname -a): 5.15
• Install tools: N/A
• Others: N/A

[mhenriks]

@k8scoder192 Thanks for the bug report! Turns out that with nfs there's probably no way to guarantee that cdi has permission to delete everything. We should be more precise. Only delete files we may create. That would of course leave /data/.snapshot untouched

[jdambly-ns]

I think I am having the same issue

(⎈ |cfw-test-nc4:vms)➜  ~ kubectl logs importer-haproxy01
I0329 20:42:34.223375       1 importer.go:103] Starting importer
I0329 20:42:34.223411       1 importer.go:168] begin import process
I0329 20:42:34.223469       1 registry-datasource.go:172] Copying proxy certs
I0329 20:42:34.223490       1 registry-datasource.go:57] Error creating allCertDir open /proxycerts/: no such file or directory
I0329 20:42:34.223554       1 data-processor.go:379] Calculating available size
I0329 20:42:34.223561       1 data-processor.go:391] Checking out file system volume size.
I0329 20:42:34.223567       1 data-processor.go:399] Request image size not empty.
I0329 20:42:34.223577       1 data-processor.go:404] Target size 23054442496.
I0329 20:42:34.223640       1 util.go:38] deleting file: /data/lost+found
E0329 20:42:34.223669       1 util.go:41] Unable to delete file: /data/lost+found, openfdat /data/lost+found: permission denied
E0329 20:42:34.223685       1 importer.go:177] openfdat /data/lost+found: permission denied
Failure cleaning up target space
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:181
main.handleImport
	cmd/cdi-importer/importer.go:174
main.main
	cmd/cdi-importer/importer.go:140
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1571

[k8scoder192]

@mhenriks this seems to be fix now. Thank you!