NodePublishVolume can mount the wrong volume: waitDeviceReady only checks that the by-id path exists

[victorlane]

What happened

After a node reboot, two PVCs were published onto the same block device. docmost/data-docmost-redis-0 and monitoring/monitoring-grafana both ended up on the device holding the Grafana volume. Redis then wrote its AOF into the Grafana volume, and because the container runtime relabelled the shared device for one pod, the other pod lost access with Permission denied (SELinux MCS). Both workloads stayed broken for 2 days until the pods were deleted, which caused a correct republish.

Environment

• csi-driver v2.21.2
• RKE2 v1.36.3, openSUSE Leap Micro 6.2, SELinux enforcing
• Volumes already attached to the node at boot

Evidence

Node plugin log for that boot. Note the readiness retries, then two successful publishes 112 ms apart:

00:03:14 ERROR failed to publish volume: device "/dev/disk/by-id/scsi-0HC_Volume_106486781" not ready: no such file or directory
00:03:14 ERROR failed to publish volume: device "/dev/disk/by-id/scsi-0HC_Volume_106478890" not ready: no such file or directory
(further retries)
00:03:27.670 INFO publishing volume target-path=.../pvc-dc7df2e4-.../mount device-path=/dev/disk/by-id/scsi-0HC_Volume_106486781
00:03:27.782 INFO publishing volume target-path=.../pvc-742f8b5a-.../mount device-path=/dev/disk/by-id/scsi-0HC_Volume_106478890

The resulting mounts point at the same device (8:32), even though the recorded source paths differ:

# findmnt -rno SOURCE,MAJ:MIN,TARGET | grep kubernetes.io~csi
/dev/disk/by-id/scsi-0HC_Volume_106486781 8:32 /var/lib/kubelet/pods/.../pvc-dc7df2e4-.../mount
/dev/disk/by-id/scsi-0HC_Volume_106478890 8:32 /var/lib/kubelet/pods/.../pvc-742f8b5a-.../mount

While the symlinks resolve like this:

/dev/disk/by-id/scsi-0HC_Volume_106478890 -> /dev/sdc   (8:32)
/dev/disk/by-id/scsi-0HC_Volume_106486781 -> /dev/sdd   (8:48)

So volume 106486781 was mounted from the device belonging to volume 106478890. Note that findmnt -S /dev/sdX matches on the recorded source string and therefore hides this. Only the device number shows it.

Cause

waitDeviceReady in internal/volumes/mount.go, added in #1381 for #1346, only stats the by-id path and returns as soon as it exists:

err = unix.Stat(devicePath, &stat)
if err == nil { return nil }

An existing path is not proof that it points at the requested volume. At boot many volumes are attached at once while udev is still populating /dev/disk/by-id, so a publish can resolve a path before it is final. This is still the case on main as of 2026-08-10.

Suggested fix

After resolving the symlink, compare the device serial against the requested volume ID before mounting, for example via /sys/block/<dev>/device/vpd_pg80 or udevadm info, and retry on mismatch. The check is cheap and turns a silent data corruption path into a retry.

Workaround

Ordering the kubelet unit after udevadm settle so that /dev/disk/by-id is fully populated before any publish runs.