DRA: DRA Driver and ResourceSlices in Composable System

[hase1128]

What would you like to be added?

In composable system, it is necessary to consider the optimal design of the composable DRA driver for managing fabric devices and the vendor DRA driver for managing node-local devices.
According to KEP-5007 (kubernetes/enhancements#5012), especially (kubernetes/enhancements#5012 (comment)), there are three ideas:
(If I've missed something, please let me know.)

1. Moving the device and update ResourceSlice

• Basic concept:
  • KEP-5007: DRA Device Binding Conditions enhancements#5012 (comment)
  • KEP-5007: DRA Device Binding Conditions enhancements#5012 (comment)
• Advantages: It is possible to minimize the addition of features to the vendor DRA
• Problems: It is necessary to consider conflicts due to moves and updates
• Vendor DRA driver needs to be newly implemented: Only periodic rescan

2. Implement a device autoscaler in ClusterAutoscaler

• Basic concept:
  https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5007-device-attach-before-pod-scheduled#alternative-approach
• Advantages: The scheduler does not need to reschedule pods.
• Problems: Requires major addition of functionality in CA. May not fit with CA concept (CA scales nodes horizontally)
• New implementation required in vendor DRA driver: Periodic rescan only

3. Make vendor DRA driver aware of fabric devices

• Basic concept: KEP-5007: DRA Device Binding Conditions enhancements#5012 (comment)
• Advantages: The scheduler does not need to reschedule pods. Can use happy path (can proceed directly to binding after device attachment). It is possible to avoid moving devices between ResourceSlices.
• Problem: A large feature needs to be added to the vendor DRA.
• New implementations required for the vendor DRA driver:
  • Fabric device recognition
  • Composable-related IFs such as attach,
  • Updating ResourceSlices after attach
  • Updating BindingConditions in ResourceClaim
  • Synchronization with vendor DRA drivers on other nodes, etc.

I think idea 1 is good, but I would like to hear from DRA experts on which of these ideas is better, or if there are any better ideas.

/cc @pohly
/cc @klueska
/cc @johnbelamaric
/cc @KobayashiD27

/sig node

Why is this needed?

Composable disaggregated infrastructured needs it for GPUs connected to a node on demand:
https://kccnceu2024.sched.com/event/1ZPDw/iown-bof-challenges-of-kubernetes-for-composable-disaggregated-computing-naoki-oguchi-fujitsu-hidetsugu-sugiyama-red-hat-clara-li-intel-ryosuke-kurebayashi-ntt

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[KobayashiD27]

kubernetes/enhancements#5007 (comment)
@towca

Firstly, we aim to implement the functionality (BindingConditions) currently described in this KEP-5007. BindingConditions are designed as a general implementation that can support various scenarios beyond just adding or removing devices. For example, they can handle cases where devices need specific processing to meet certain conditions before binding. We believe that this functionality will enable the automatic addition and removal of devices.

The CA-based approach also seems much cleaner semantically to me. Aborting the Bind phase as part of "normal operation" is quite surprising. Are we sure it won't introduce any side effects?

In general use cases, we do not expect the Bind phase to be aborted. However, in cases involving the addition or removal of devices, the Bind phase may need to be aborted. About aborting the Bind phase, experiments in our local environment and reviews by scheduler maintainers suggest there should be no issues. And, BindingConditions feature itself do not affect CA. However, if devices are added or removed, there could be corner case concerns. We need to consider CA integration, but this has not been addressed at the alpha stage of the KEP-5007. This is a topic for further discussion. Therefore, we believe that BindingCondition-based device addition/removal and CA must be used exclusively for now.

If we want to go with the proposed approach and still have Cluster Autoscaler working, there are some corner cases to figure out

Thank you for pointing out the corner cases regarding CA integration. CA integration is not yet considered at the alpha stage. We plan to address these corner cases in future iterations. For now, we believe that the current approach cannot be used with CA. In the future, we aim to integrate CA and consider another approaches to ensure smooth operation. With CA integration, it is possible that automatic addition and removal of devices does not need BindingConditions. However we believe that BindingConditions will still be retained for other purposes.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[hase1128]

/remove-lifecycle

[johnbelamaric]

/remove-lifecycle rotten