rbd: restore cgroup QoS when regenerating PV omap by Madhu-1 · Pull Request #6529 · ceph/ceph-csi

Madhu-1 · 2026-09-02T05:20:17Z

The PersistentVolume controller regenerates the RBD omap/journal for volumes (for example during disaster-recovery failover). Cgroup v2 QoS limits are stored as RBD image metadata under the .rbd.csi.ceph.com/ prefix, which is intentionally not propagated during clone, snapshot or mirroring. As a result, once the omap is regenerated those QoS limits are missing on the image.

Add RegenerateCgroupQoS() in the rbd package, which re-applies the cgroup QoS metadata from the volume's VolumeAttributesClass. It is a no-op when the class has no cgroup QoS parameters or when the volume uses the rbd-nbd mounter (cgroup QoS applies only to krbd). The PV controller now looks up the VolumeAttributesClass referenced by the
PV and invokes this after RegenerateJournal for RBD volumes.

Vendor sigs.k8s.io/controller-runtime/pkg/client/fake and its transitive dependencies so controller unit tests can use a fake client. These packages are already part of the dependency graph only the vendored files are added (go.mod and go.sum are unchanged). Signed-off-by: Madhu Rajanna <madhupr007@gmail.com> Co-Authored-By: Claude Opus <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The PV controller calls RegenerateCgroupQoS with secrets=nil, which can break QoS restoration for encrypted volumes because volume generation may require secret material for KMS/encryption setup.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the RBD PV reconciliation path so that when the PersistentVolume controller regenerates an RBD volume’s journal/omap, it also restores cgroup v2 QoS metadata on the backing RBD image from the PV’s VolumeAttributesClass (skipping volumes without relevant params and skipping rbd-nbd where cgroup QoS doesn’t apply).

Changes:

Add rbd.RegenerateCgroupQoS() to re-apply .rbd.csi.ceph.com/ cgroup QoS image metadata from VolumeAttributesClass after journal/omap regeneration.
Extend the PV controller to look up the referenced VolumeAttributesClass and invoke QoS regeneration after RegenerateJournal.
Add unit/e2e coverage and vendor updates needed for controller-runtime fake client usage in tests.

File summaries

File	Description
internal/rbd/cgroup_qos.go	Adds `RegenerateCgroupQoS()` to restore cgroup QoS metadata from VAC to the RBD image.
internal/rbd/cgroup_qos_test.go	Adds a unit test ensuring `RegenerateCgroupQoS()` is a no-op when VAC has no effective cgroup QoS params.
internal/controller/persistentvolume/persistentvolume.go	Fetches VAC parameters referenced by the PV and calls `RegenerateCgroupQoS()` after journal regeneration.
internal/controller/persistentvolume/persistentvolume_test.go	Adds tests for VAC parameter lookup using controller-runtime’s fake client.
e2e/rbd.go	Adds an acceptance test that simulates omap + QoS metadata loss and validates PV controller restoration from VAC.
vendor/sigs.k8s.io/controller-runtime/pkg/internal/objectutil/objectutil.go	Vendors helper used by controller-runtime fake client list filtering.
vendor/sigs.k8s.io/controller-runtime/pkg/client/interceptor/intercept.go	Vendors controller-runtime client interceptor support.
vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go	Vendors controller-runtime fake client implementation required by new tests.
vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/doc.go	Vendors fake client package docs.
vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/typeconverter.go	Vendors type converter helper for field-managed fake tracker.
vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/versioned_tracker.go	Vendors versioned object tracker used by fake client.
vendor/modules.txt	Updates vendored module/package list to include newly added vendor packages.
vendor/k8s.io/apimachinery/pkg/util/rand/rand.go	Vendors k8s apimachinery rand utility used by fake client.
vendor/k8s.io/client-go/applyconfigurations/OWNERS	Vendors applyconfigurations OWNERS file.
vendor/k8s.io/client-go/applyconfigurations/doc.go	Vendors applyconfigurations documentation.
vendor/k8s.io/client-go/applyconfigurations/imagepolicy/v1alpha1/imagereview.go	Vendors imagepolicy applyconfig types (dependency completeness for vendored client-go).
vendor/k8s.io/client-go/applyconfigurations/imagepolicy/v1alpha1/imagereviewcontainerspec.go	Vendors imagepolicy applyconfig type.
vendor/k8s.io/client-go/applyconfigurations/imagepolicy/v1alpha1/imagereviewspec.go	Vendors imagepolicy applyconfig type.
vendor/k8s.io/client-go/applyconfigurations/imagepolicy/v1alpha1/imagereviewstatus.go	Vendors imagepolicy applyconfig type.
vendor/k8s.io/api/imagepolicy/v1alpha1/doc.go	Vendors imagepolicy API package docs.
vendor/k8s.io/api/imagepolicy/v1alpha1/generated.pb.go	Vendors generated protobuf bindings for imagepolicy API.
vendor/k8s.io/api/imagepolicy/v1alpha1/generated.proto	Vendors protobuf source for imagepolicy API.
vendor/k8s.io/api/imagepolicy/v1alpha1/register.go	Vendors scheme registration for imagepolicy API.
vendor/k8s.io/api/imagepolicy/v1alpha1/types.go	Vendors imagepolicy API types.
vendor/k8s.io/api/imagepolicy/v1alpha1/types_swagger_doc_generated.go	Vendors swagger-doc helpers for imagepolicy API.
vendor/k8s.io/api/imagepolicy/v1alpha1/zz_generated.deepcopy.go	Vendors deepcopy-gen output for imagepolicy API.
vendor/k8s.io/api/imagepolicy/v1alpha1/zz_generated.model_name.go	Vendors openapi-gen model-name output for imagepolicy API.

Review details

Files reviewed: 5/28 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

There are concrete correctness issues in newly-added/modified code paths (error formatting in PV controller logs and a parallel-test data race) plus an unmentioned broad vendor refresh that should be split out or explicitly justified.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

internal/controller/persistentvolume/persistentvolume.go:193

Using %s to format an error value will produce fmt formatting noise (e.g. %!s(...)) instead of the error message; use %v for errors.

			log.ErrorLogMsg("failed to regenerate cgroup QoS %s", err)

Files reviewed: 6/29 changed files
Comments generated: 2
Review effort level: Lite

Madhu-1 · 2026-09-02T06:23:27Z

/test ci/centos/mini-e2e-helm/k8s-1.34

Copilot

🟡 Changes recommended

VolumeAttributesClass parameters are consumed without verifying that the class belongs to the RBD CSI driver.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 6/29 changed files
Comments generated: 1
Review effort level: Balanced

Madhu-1 · 2026-09-02T12:55:06Z

/test ci/centos/mini-e2e-helm/k8s-1.34

The PersistentVolume controller regenerates the RBD omap/journal for volumes (for example during disaster-recovery failover). Cgroup v2 QoS limits are stored as RBD image metadata under the `.rbd.csi.ceph.com/` prefix, which is intentionally not propagated during clone, snapshot or mirroring. As a result, once the omap is regenerated those QoS limits are missing on the image. Add RegenerateCgroupQoS() in the rbd package, which re-applies the cgroup QoS metadata from the volume's VolumeAttributesClass. It is a no-op when the class has no cgroup QoS parameters or when the volume uses the rbd-nbd mounter (cgroup QoS applies only to krbd). The PV controller now looks up the VolumeAttributesClass referenced by the PV and invokes this after RegenerateJournal for RBD volumes. Unit tests cover the VolumeAttributesClasslookup (via a fake client) and the no-op guards of RegenerateCgroupQoS. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com> Co-Authored-By: Claude Opus <noreply@anthropic.com>

Add an end-to-end test that exercises the PersistentVolume controller restoring cgroup v2 QoS after the omap is regenerated. The test provisions a krbd PVC with a VolumeAttributesClass carrying cgroup QoS parameters, confirms the QoS image metadata is set, then simulates a disaster-recovery scenario by removing the `.`-prefixed cgroup QoS metadata from the image and deleting the omap. It statically rebinds the PV to trigger the controller, and verifies the controller regenerates the omap and re-applies the cgroup QoS metadata from the VolumeAttributesClass. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com> Co-Authored-By: Claude Opus <noreply@anthropic.com>

Madhu-1 · 2026-09-02T13:01:28Z

/test ci/centos/mini-e2e-helm/k8s-1.35

black-dragon74 · 2026-09-03T06:06:34Z

+			return err
+		}
+
+		rbdVol, err := rbd.RegenerateJournal(


Do we want to skip regen if VAC doesn't exist?

RegenerateJournal is for the OMAP, unrelated to VAC

if you meant regenrateQos we already have check inside RegenerateCgroupQoS to make it no op

black-dragon74 · 2026-09-03T06:07:50Z

+	}
+
+	if pv.Spec.CSI.Driver != vac.DriverName {
+		return nil, errors.New("driver name in VAC and PV are not matching")


Maybe fmt.Errorf with a few identifiers? Would make it easier to debug.

there are chances to change the driver name to go through, thats why not adding it but laters it can be compared by looking at the PV and PVC object

Madhu-1 requested review from a team as code owners September 2, 2026 05:20

Madhu-1 requested a lite review from Copilot and removed request for a team September 2, 2026 05:20

Copilot started reviewing on behalf of Madhu-1 September 2, 2026 05:20 View session

mergify Bot added the component/rbd Issues related to RBD label Sep 2, 2026

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread internal/controller/persistentvolume/persistentvolume.go Outdated

Madhu-1 force-pushed the qos-regenarate branch from 2530b42 to b8ca7b1 Compare September 2, 2026 05:39

Madhu-1 requested a lite review from Copilot September 2, 2026 05:39

Copilot started reviewing on behalf of Madhu-1 September 2, 2026 05:40 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread internal/controller/persistentvolume/persistentvolume.go Outdated

Comment thread internal/controller/persistentvolume/persistentvolume_test.go

Madhu-1 force-pushed the qos-regenarate branch from b8ca7b1 to d4bc833 Compare September 2, 2026 06:15

Madhu-1 requested review from a team September 2, 2026 06:15

Rakshith-R requested a balanced review from Copilot September 2, 2026 06:37

Copilot started reviewing on behalf of Rakshith-R September 2, 2026 06:37 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread internal/controller/persistentvolume/persistentvolume.go

Madhu-1 force-pushed the qos-regenarate branch from d4bc833 to 385974a Compare September 2, 2026 06:55

Madhu-1 and others added 2 commits September 2, 2026 18:29

Madhu-1 force-pushed the qos-regenarate branch from 385974a to 0d2846c Compare September 2, 2026 12:59

iPraveenParihar approved these changes Sep 3, 2026

View reviewed changes

Madhu-1 requested a review from Rakshith-R September 3, 2026 05:08

black-dragon74 reviewed Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

rbd: restore cgroup QoS when regenerating PV omap - #6529

rbd: restore cgroup QoS when regenerating PV omap#6529
Madhu-1 wants to merge 3 commits into
ceph:develfrom
Madhu-1:qos-regenarate

Madhu-1 commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Madhu-1 commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Madhu-1 commented Sep 2, 2026

Uh oh!

Madhu-1 commented Sep 2, 2026

Uh oh!

black-dragon74 Sep 3, 2026

Uh oh!

iPraveenParihar Sep 3, 2026

Uh oh!

Madhu-1 Sep 3, 2026

Uh oh!

black-dragon74 Sep 3, 2026

Uh oh!

Madhu-1 Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Madhu-1 commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Madhu-1 commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Madhu-1 commented Sep 2, 2026

Uh oh!

Madhu-1 commented Sep 2, 2026

Uh oh!

black-dragon74 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

iPraveenParihar Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Madhu-1 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

black-dragon74 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Madhu-1 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants