Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cephfs: enable read affinity #4153

Merged
merged 3 commits into from
Nov 22, 2023

Conversation

iPraveenParihar
Copy link
Contributor

Describe what this PR does

  • cephfs: add capability to automatically enable read affinity
    This commit makes use of crush location labels from node labels to supply crush_location and read_from_replica=localize options during mount. Using these options, cephfs
    will be able to redirect reads to the closest OSD, improving performance.

Fixes: #4149

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch from a7fdbdb to a06de16 Compare September 28, 2023 07:34
@iPraveenParihar iPraveenParihar changed the title cepfs: add capability to automatically enable read affinity cephfs: add capability to automatically enable read affinity Sep 28, 2023
@mergify mergify bot added the component/cephfs Issues related to CephFS label Sep 28, 2023
@iPraveenParihar iPraveenParihar marked this pull request as ready for review September 29, 2023 04:30
@iPraveenParihar iPraveenParihar marked this pull request as draft September 29, 2023 04:35
@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Oct 29, 2023
@yati1998 yati1998 removed the stale label Nov 3, 2023
@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch 3 times, most recently from 4d028e7 to 6b766e9 Compare November 15, 2023 10:46
Copy link
Contributor

mergify bot commented Nov 15, 2023

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch from 6b766e9 to b1470fe Compare November 15, 2023 17:01
@iPraveenParihar iPraveenParihar self-assigned this Nov 15, 2023
@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch 3 times, most recently from 429c46c to 77a33c5 Compare November 16, 2023 13:51
@iPraveenParihar iPraveenParihar marked this pull request as ready for review November 16, 2023 17:02
@iPraveenParihar iPraveenParihar changed the title cephfs: add capability to automatically enable read affinity cephfs: enable read affinity Nov 16, 2023
@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e/k8s-1.28

@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e-helm/k8s-1.27

@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch from 77a33c5 to a1d198b Compare November 17, 2023 05:39
@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e-helm/k8s-1.27

@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e/k8s-1.28

@iPraveenParihar
Copy link
Contributor Author

  I1117 06:00:09.279048   24407 cephcmds.go:98] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs]
  E1117 06:00:09.279084   24407 nodeserver.go:322] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 failed to mount volume 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79: an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs] stderr: 2023-11-17T06:00:08.253+0000 7f4b3c7ce580 -1 init, newargv = 0x55d0d853fe60 newargc=17
  ceph-fuse[43995]: starting ceph client
  fuse: unknown option `read_from_replica=localize'
  ceph-fuse[43995]: fuse failed to start
  2023-11-17T06:00:08.265+0000 7f4b3c7ce580 -1 fuse_lowlevel_new failed

@Rakshith-R, looks like read_from_replica=localize option is not supported by fuse mounter?

@iPraveenParihar iPraveenParihar force-pushed the cephfs/enable-read-affinity branch from a1d198b to 128cf36 Compare November 17, 2023 06:53
@iPraveenParihar
Copy link
Contributor Author

/test ci/centos/mini-e2e/k8s-1.28

@Madhu-1
Copy link
Collaborator

Madhu-1 commented Nov 17, 2023

  I1117 06:00:09.279048   24407 cephcmds.go:98] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs]
  E1117 06:00:09.279084   24407 nodeserver.go:322] ID: 31 Req-ID: 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79 failed to mount volume 0001-0024-ad7d7485-6a6b-42f7-94e3-1d8d02f2ae14-0000000000000001-30434073-3a09-4b94-b917-14ea5b34ab79: an error (exit status 33) occurred while running ceph-fuse args: [/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/ccc163dda7c746a00452c94bf9e5bf1422e9cc1013a34dfba8e4699b02ccdfb9/globalmount -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 -c /etc/ceph/ceph.conf -n client.cephcsi-cephfs-node --keyfile=***stripped*** -r /volumes/e2e/csi-vol-30434073-3a09-4b94-b917-14ea5b34ab79/1232d1e8-33c2-49a3-badc-d875ba15a341 -o nonempty,default_permissions,read_from_replica=localize,crush_location=region:east|zone:east-zone1 --client_mds_namespace=myfs] stderr: 2023-11-17T06:00:08.253+0000 7f4b3c7ce580 -1 init, newargv = 0x55d0d853fe60 newargc=17
  ceph-fuse[43995]: starting ceph client
  fuse: unknown option `read_from_replica=localize'
  ceph-fuse[43995]: fuse failed to start
  2023-11-17T06:00:08.265+0000 7f4b3c7ce580 -1 fuse_lowlevel_new failed

@Rakshith-R, looks like read_from_replica=localize option is not supported by fuse mounter?

Fuse might not support this option, you can to skip and log it in the nodeplugin.

Comment on lines 234 to 235
Refer [krbd-options](https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options)
for more details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This points to krbd mount options,
These read affinity mount options are not documented for cephfs kernel mounter. I'll ask for it to be documented at https://docs.ceph.com/en/latest/man/8/mount.ceph/#options.

Let's remove these two lines for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@Rakshith-R
Copy link
Contributor

Rakshith-R commented Nov 22, 2023

@Mergifyio rebase

This commit makes use of crush location labels from node
labels to supply `crush_location` and `read_from_replica=localize`
options during mount. Using these options, cephfs
will be able to redirect reads to the closest OSD,
improving performance.

Signed-off-by: Praveen M <m.praveen@ibm.com>
This commit adds --enable-read-affinity flag to
enable read affinity for CephFS.

Signed-off-by: Praveen M <m.praveen@ibm.com>
This commit adds documentation about read affinity supported
for CephFS subvolumes.

Signed-off-by: Praveen M <m.praveen@ibm.com>
Copy link
Contributor

mergify bot commented Nov 22, 2023

rebase

✅ Branch has been successfully rebased

@Rakshith-R Rakshith-R force-pushed the cephfs/enable-read-affinity branch from a3ba54c to 0cc6db9 Compare November 22, 2023 08:46
@Rakshith-R Rakshith-R requested a review from a team November 22, 2023 08:46
@nixpanic
Copy link
Member

@Mergifyio rebase

Copy link
Contributor

mergify bot commented Nov 22, 2023

rebase

✅ Nothing to do for rebase action

@nixpanic
Copy link
Member

@Mergifyio queue

Copy link
Contributor

mergify bot commented Nov 22, 2023

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 00c12b3

@mergify mergify bot added the ok-to-test Label to trigger E2E tests label Nov 22, 2023
@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.26

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/upgrade-tests-cephfs

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.26

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.28

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.27

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/upgrade-tests-rbd

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.26

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.28

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.27

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.28

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.27

@ceph-csi-bot ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Nov 22, 2023
@mergify mergify bot merged commit 00c12b3 into ceph:devel Nov 22, 2023
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/cephfs Issues related to CephFS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cephfs: add capability to automatically enable read affinity
6 participants