cephfs: refresh recursive stats before volume stats

[baldugus]

Describe what this PR does

Refreshes CephFS recursive statistics before collecting filesystem usage.

Linux CephFS statfs() overrides filesystem statistics for quota-backed mounts using the quota inode's cached rbytes value (source) instead of using the fresh values like the regular statfs() path (source).

On the other hand, Ceph MDS only broadcasts new statistics when usage changes by approx. 1/16 of the difference between the last reported usage and the max quota available (source).

Slow growing volumes can remain for long periods with stale usage and consequently stale kubelet_volume_stats_used_bytes. In my cluster I noticed my metrics database had several PVCs reporting the same used bytes for a whole month, despite constant write activity.

Neither behavior is wrong on its own, so I wrote this patch intending to make the statistics accurate in ceph-csi.

I've ran the added E2E test on the latest commit at devel (f4ce351) and in this PR and I could reproduce a fail before and a pass after.

I've also replaced my production ceph-csi with this patch and all affected volumes started reporting the current usage.

This is my manual reproduction before the patch on my kubernetes host:

$ df -B1 <kubelet globalmount path>
Filesystem 1-blocks Used Available Use% Mounted on
<path> 10737418240 0 10737418240 0% <path>

$ # fetched ceph.dir.rbytes using a perl script as the host lacks getfattr
203487451

$ df -B1 <kubelet globalmount path>
Filesystem 1-blocks Used Available Use% Mounted on
<path> 10737418240 201326592 10536091648 2% <path>

Is there anything that requires special attention

This is my first contribution to ceph-csi. I believe this change is backward compatible.

If this is accepted, is it worth updating the pending release notes? I was unsure about it, so I didn't include an update in this PR.

Related issues

#2579

Future concerns

This adds an MDS request when collecting CephFS volume statistics. If a CephFS-side change makes statfs() sufficient in the future, this patch could be reconsidered.

Checklist:

• Commit Message Formatting: Commit titles and messages follow
  guidelines in the developer
  guide.
• Reviewed the developer guide on Submitting a Pull
  Request
• Pending release
  notes
  updated with breaking and/or notable changes for the next major release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[nixpanic]

Thanks, looks good.

Pull request has been modified.

[baldugus]

@nixpanic

The tests had failed, but I saw that you fixed the ceph-release RPM URL yesterday, so I've rebased my branch onto the latest commits.