What
PR #6354 (merged into devel on 2026-07-06) removes the SnapshotLocks
guard from createVolumeFromSnapshot(). v3.17.1 was released on 2026-08-24 —
seven weeks after the merge — but does not contain it: the lock is still
present in internal/rbd/controllerserver.go at the v3.17.1 tag.
Please consider cherry-picking it onto release-v3.17.
Why it matters — measurements from a test cluster
Environment: ceph-csi v3.17.0, Ceph 10 NVMe OSDs / 24 TiB, Kubernetes with
KubeVirt; VMs clone their root disks from a per-image golden VolumeSnapshot.
external-provisioner v6.2.0 with --worker-threads=100 --kube-api-qps=100,
so neither sidecar concurrency nor API throttling is the limiter.
Batches of 120 × 10Gi block PVCs, dataSource = VolumeSnapshot:
| scenario |
throughput |
| 120 PVCs, one source snapshot |
34.6/min — completion curve perfectly linear, ~1.7 s per volume |
| 120 PVCs, four source snapshots (30 each) |
100/min (~3×, near-linear) |
rbd clone directly against the pool, sequentially, same source |
245/min (244 ms/clone) |
During the two K8s runs the plugin logged 1,645 ×
Aborted: an operation with the given Snapshot ID ... already exists
(controllerserver.go:687). Because the guard is TryAcquire, losers are not
queued in-process but bounce back to csi-provisioner's exponential backoff;
under a burst this makes throughput worse than plain serialization — a
paced arrival stream reaches ~45/min from the same single snapshot, while a
one-shot burst of 120 only reaches 34.6/min.
The critical section holds far more than the clone itself: snapshot resolve
via omap, librbd.CloneImage, snapshot-metadata unset, encryption-config
copy, and expand — ~1.7 s of serial round trips where the clone itself needs
244 ms.
Net effect on v3.17: cloning many volumes from one snapshot — the standard
golden-image pattern for VM platforms — is capped at roughly 35 volumes/min
per source snapshot regardless of any configuration, and the Aborted retry
storm adds API and log noise on top.
Notes
What
PR #6354 (merged into
develon 2026-07-06) removes theSnapshotLocksguard from
createVolumeFromSnapshot(). v3.17.1 was released on 2026-08-24 —seven weeks after the merge — but does not contain it: the lock is still
present in
internal/rbd/controllerserver.goat the v3.17.1 tag.Please consider cherry-picking it onto
release-v3.17.Why it matters — measurements from a test cluster
Environment: ceph-csi v3.17.0, Ceph 10 NVMe OSDs / 24 TiB, Kubernetes with
KubeVirt; VMs clone their root disks from a per-image golden VolumeSnapshot.
external-provisioner v6.2.0 with
--worker-threads=100 --kube-api-qps=100,so neither sidecar concurrency nor API throttling is the limiter.
Batches of 120 × 10Gi block PVCs,
dataSource= VolumeSnapshot:rbd clonedirectly against the pool, sequentially, same sourceDuring the two K8s runs the plugin logged 1,645 ×
Aborted: an operation with the given Snapshot ID ... already exists(controllerserver.go:687). Because the guard is
TryAcquire, losers are notqueued in-process but bounce back to csi-provisioner's exponential backoff;
under a burst this makes throughput worse than plain serialization — a
paced arrival stream reaches ~45/min from the same single snapshot, while a
one-shot burst of 120 only reaches 34.6/min.
The critical section holds far more than the clone itself: snapshot resolve
via omap,
librbd.CloneImage, snapshot-metadata unset, encryption-configcopy, and expand — ~1.7 s of serial round trips where the clone itself needs
244 ms.
Net effect on v3.17: cloning many volumes from one snapshot — the standard
golden-image pattern for VM platforms — is capped at roughly 35 volumes/min
per source snapshot regardless of any configuration, and the Aborted retry
storm adds API and log noise on top.
Notes
covers the race the lock was protecting; as noted in rbd: remove snapshot lock from restore path #6354 the restore
operation lock already covers the restore path.
cherry-pick to confirm the numbers.