librbd QoS settings for RBD based PVs

[mmgaggle]

Describe the feature you'd like to have

The ability to set librbd QoS settings on a PV to limit how much IO can be consumed from the Ceph Cluster.

The exactly limits would be informed through the storage-class configuration. Ideally we would support three different types of limits:

1. static rbd_qos_iops_limit and rbd_qos_bps_limit per volume
2. dynamic rbd_qos_iops_limit and rbd_qos_bps_limit per volume as a function of the PV size (eg. 3 IOPS per GB, 100 MB/s per TB with a configurable rbd_qos_schedule_tick_min.

A PVC could specify the number of IOPs from storage classes of the second type, but it would adjust the capacity requested based on the above ratio configured in the storage class definition.

What is the value to the end user?

Many users were frustrated by IO noisy neighbor issues in early Ceph deployments that were catering to OpenStack environments. Folks started to implement QEMU throttling at the virtio-blk/scsi and this became much more manageable. Capacity based IOPs further improved on the situation by providing familiar a public cloud like experience (vs static per volume limits).

We want Kubernetes and OpenShift users to have improved noisy neighbor isolation too!

How will we know we have a good solution?

1. Configure ceph-csi to use nbd-rbd approach.
2. Provision volume from storage class as configured above.
3. CSI provisioner would set limit on RBD image.
4. fio test against PV would confirm that IOPs limits were being enforced.

Once resize work is finished, we'll need to ensure new limits are applied when a volume is re-sized.

[fire]

Is anyone working on this? What is the status of this?

[dillaman]

I don't believe anyone is working on it -- and it really doesn't make much sense since rbd-nbd isn't really production worthy (right now), so we wouldn't want to encourage even more folks using it.

The best longer term solution would be to ensure cgroups v2 is being utilized on the k8s node so that generic block rate limiting controls can be applied (which would handle krbd both rbd-nbd). I'm not sure of the realistic timeline for cgroups v2 integration in k8s (it just became the default under Fedora 31).

[fire]

Do you know where cgroups v2 integration in k8s for limiting block rates is tracked?

[dillaman]

This [1] provides a really good overview and a theoretical timeline

[1] https://medium.com/nttlabs/cgroup-v2-596d035be4d7

[mmgaggle]

The bigger problem with cgroups is that they only provide independent limits for read and write, compared with libdrbd/virtio which provide the ability to express limits against the aggregate of reads and writes.

If I want to limit a given PV to 100 IOPs, I can't do that with cgroups. I can only set a write IOPs limit to (50, 30, 10) and another distinct limit on read IOPS (50, 70, 90). A client can't trade a write IO for a read IO, or vis versa.

[mmgaggle]

Probably would need something like this KEP to do accounting -

kubernetes/enhancements#1353

Basically if you know a cluster can provide 100k iops and 100TB, then you need to add up the PVs (qty * static limit, or capacity * ratio limit) to make sure you're not oversubscribed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.