feature: Add dynamic resource allocation(DRA) plugin #3799

JesseStutler · 2024-10-31T02:36:55Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Background

Inspired by #3798 , introduce DRA into volcano. And #4152 have added PreBind implementation, so DRA can base on this PR and add DRA plugin implementation into predicates plugin.

Verfication

I followed up https://github.com/kubernetes-sigs/dra-example-driver to deploy an example resource driver and deploy the demo yaml gpu-test{1,2,3,4,5}.yaml, the pod can be successfully scheduled:

GPU information printed in containers:

And allocation result in ResourceClaim is successfully updated:

googs1025 · 2024-10-31T02:54:43Z

/cc

googs1025 · 2024-10-31T08:24:32Z

installer/helm/chart/volcano/values.yaml

@@ -25,6 +25,7 @@ custom:
  leader_elect_enable: false
  enabled_admissions: "/jobs/mutate,/jobs/validate,/podgroups/mutate,/pods/validate,/pods/mutate,/queues/mutate,/queues/validate"
  colocation_enable: false
+  feature_gates: DynamicResourceAllocation=true


Do we need to enable this feature-gate by default?

What do you think? Do we need to enable it by default? I think that if users need to enable dra plugin, they also need to reconfigure the scheduler plugins configmap to enable it, it also looks like featuregate. If we don't enable this feture-gate, users need to do reconfiguring the scheduler plugins configmap and enabling the feature-gate two steps.

OK we need not to enable this feature-gate by default, I will add some informers in schedulerCache, some users may need to use k8s below v1.31, if there are no APIs like DeviceClass, ResourceClaim, etc registered, it will cause some errors.

googs1025 · 2024-10-31T08:31:06Z

pkg/scheduler/capabilities/dynamicresources/dynamicresources.go

@@ -0,0 +1,808 @@
+/*
+Copyright 2022 The Kubernetes Authors.


Is this part directly imported into the scheduler code? Or has it been modified and adapted (I don't know much about it)? If not, should it be marked in the code?

I just changed the New to NewDRAPlugin and deleted EventsToRegister and some related codes with it, I think volcano doesn't need them right now. These codes are copied from the kubernetes repo, so the license needs to keep.

Understand, if so, if the upstream code changes, do I need to make the same changes?

Yes, before the dynamicResources struct changed to DynamicResources, we can only copy the whole codes related into volcano, otherwise we can't directly call those extension points. It may cause us to miss tracking some repaired codes, but if we can access the DynamicResources struct in v1.32, the problem will be solved.

stale · 2025-04-25T23:44:58Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

hwdef · 2025-05-05T07:14:08Z

still need

Monokaix · 2025-05-23T09:31:47Z

Great job!

Copilot

Pull Request Overview

This PR introduces Dynamic Resource Allocation (DRA) support into the Volcano scheduler by wiring a new DRA plugin into the predicate phase, extending cache and framework handle to supply a shared DRAManager, and updating tests and installer manifests to enable and verify the feature.

Add builders for DRA objects (ResourceClaim, ResourceSlice, DeviceClass, DeviceRequest) and a helper to create pods with resource claims.
Integrate SharedDRAManager into scheduler cache, framework handle, and predicates plugin lifecycle (PreFilter, Filter, Reserve, Unreserve, PreBind, PreBindRollBack).
Update installer manifests, Helm charts, Makefile, and end-to-end (E2E) tests to enable the DynamicResourceAllocation feature gate and required RBAC.

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/scheduler/util/test_utils.go	Added DRA builders and `BuildPodWithResourceClaim`
pkg/scheduler/uthelper/helper.go	Populated cache with DeviceClasses, ResourceClaims, ResourceSlices
pkg/scheduler/plugins/util/k8s/framework.go	Added `WithSharedDRAManager` option to pass DRAManager into plugins
pkg/scheduler/plugins/predicates/predicates.go	Wired DRA plugin into PreFilter, Filter, Reserve, Unreserve, PreBind
pkg/scheduler/cache/cache.go	Initialized and exposed `sharedDRAManager` in the scheduler cache
pkg/scheduler/actions/allocate/allocate.go	Left TODO for PostFilter extension point for DRA

Comments suppressed due to low confidence (1)

pkg/scheduler/util/test_utils.go:101

Fix typo in comment: change 'builts' to 'builds'.

// BuildPodWithPVC builts Pod object with pvc volume

pkg/scheduler/actions/allocate/allocate.go

pkg/scheduler/util/test_utils.go

pkg/scheduler/plugins/predicates/predicates.go

test/e2e/dra/builder.go

test/e2e/dra/main_test.go

hack/e2e-kind-config.yaml

docs/user-guide/how_to_enable_dra.md

pkg/scheduler/plugins/predicates/predicates.go

Signed-off-by: JesseStutler <chenzicong4@huawei.com>

pkg/scheduler/plugins/predicates/predicates.go

Monokaix · 2025-05-27T06:20:58Z

/approve

volcano-sh-bot · 2025-05-27T06:21:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Monokaix

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Monokaix]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wangyang0616 · 2025-05-27T06:34:42Z

/lgtm

### What changes were proposed in this pull request? This PR aims to upgrade `Volcano` to 1.12.1 in K8s integration test document and GA job. ### Why are the changes needed? To bring the latest improvements and security fixes. - https://github.com/volcano-sh/volcano/releases/tag/v1.12.1 - volcano-sh/volcano#4336 - https://github.com/volcano-sh/volcano/releases/tag/v1.12.0 - volcano-sh/volcano#4099 - volcano-sh/volcano#3799 - volcano-sh/volcano#4207 - GHSA-hg79-fw4p-25p8 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51343 from dongjoon-hyun/SPARK-52639. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to upgrade `Volcano` to 1.12.1 in K8s integration test document and GA job. ### Why are the changes needed? To bring the latest improvements and security fixes. - https://github.com/volcano-sh/volcano/releases/tag/v1.12.1 - volcano-sh/volcano#4336 - https://github.com/volcano-sh/volcano/releases/tag/v1.12.0 - volcano-sh/volcano#4099 - volcano-sh/volcano#3799 - volcano-sh/volcano#4207 - GHSA-hg79-fw4p-25p8 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#51343 from dongjoon-hyun/SPARK-52639. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

volcano-sh-bot requested review from archlitchi, merryzhou and Thor-wl October 31, 2024 02:37

volcano-sh-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 31, 2024

This was referenced Oct 31, 2024

Add dynamic resource allocation design details. #3487

Closed

[WIP]Add separate plugin for DRA(Dynamic Resource Allocation). #3577

Closed

volcano-sh-bot requested a review from googs1025 October 31, 2024 02:54

googs1025 reviewed Oct 31, 2024

View reviewed changes

JesseStutler force-pushed the dra_dev branch from 7870c64 to b577237 Compare October 31, 2024 13:42

volcano-sh-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 31, 2024

JesseStutler force-pushed the dra_dev branch 2 times, most recently from de77070 to b120b42 Compare November 1, 2024 09:37

JesseStutler force-pushed the dra_dev branch from b120b42 to 72d27ea Compare November 13, 2024 03:30

JesseStutler mentioned this pull request Nov 15, 2024

Request to be a member of Volcano community volcano-sh/community#62

Closed

volcano-sh-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 16, 2024

volcano-sh-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 11, 2024

JesseStutler force-pushed the dra_dev branch from 72d27ea to d57bdeb Compare December 26, 2024 06:32

volcano-sh-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 26, 2024

volcano-sh-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 9, 2025

JesseStutler mentioned this pull request Feb 11, 2025

Request to be an Reviewer of Volcano community volcano-sh/community#66

Closed

stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 25, 2025

stale bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 5, 2025

JesseStutler force-pushed the dra_dev branch from d57bdeb to 52b50a3 Compare May 15, 2025 09:06

volcano-sh-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 15, 2025

JesseStutler changed the title ~~feature: Add dynamic resource allocation(DRA) plugin~~ [WIP]feature: Add dynamic resource allocation(DRA) plugin May 15, 2025

JesseStutler force-pushed the dra_dev branch 2 times, most recently from 4462d05 to c3d892c Compare May 23, 2025 08:51

JesseStutler requested review from googs1025 May 23, 2025 09:26

Monokaix requested a review from Copilot May 23, 2025 09:31

Copilot AI reviewed May 23, 2025

View reviewed changes

pkg/scheduler/actions/allocate/allocate.go Show resolved Hide resolved

pkg/scheduler/util/test_utils.go Show resolved Hide resolved

googs1025 reviewed May 26, 2025

View reviewed changes

pkg/scheduler/plugins/predicates/predicates.go Show resolved Hide resolved

googs1025 reviewed May 26, 2025

View reviewed changes

test/e2e/dra/builder.go Show resolved Hide resolved

googs1025 reviewed May 26, 2025

View reviewed changes

test/e2e/dra/main_test.go Outdated Show resolved Hide resolved

Monokaix reviewed May 26, 2025

View reviewed changes

hack/e2e-kind-config.yaml Show resolved Hide resolved

Monokaix reviewed May 26, 2025

View reviewed changes

docs/user-guide/how_to_enable_dra.md Show resolved Hide resolved

JesseStutler force-pushed the dra_dev branch from c3d892c to 9e38587 Compare May 26, 2025 07:30

Monokaix reviewed May 26, 2025

View reviewed changes

pkg/scheduler/plugins/predicates/predicates.go Show resolved Hide resolved

JesseStutler force-pushed the dra_dev branch from 9e38587 to 68bd5ad Compare May 26, 2025 11:27

feature: Add DRA Implementation

90d9251

Signed-off-by: JesseStutler <chenzicong4@huawei.com>

JesseStutler force-pushed the dra_dev branch from 68bd5ad to 90d9251 Compare May 26, 2025 12:27

wangyang0616 reviewed May 27, 2025

View reviewed changes

pkg/scheduler/plugins/predicates/predicates.go Show resolved Hide resolved

volcano-sh-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 27, 2025

volcano-sh-bot assigned wangyang0616 May 27, 2025

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label May 27, 2025

volcano-sh-bot merged commit c3cba84 into volcano-sh:master May 27, 2025
18 checks passed

JesseStutler mentioned this pull request Jun 24, 2025

Support resource claim in volcano queue #4400

Open

dongjoon-hyun mentioned this pull request Jul 1, 2025

[SPARK-52639][K8S][INFRA][DOCS] Upgrade Volcano to 1.12.1 apache/spark#51343

Closed

feature: Add dynamic resource allocation(DRA) plugin #3799

feature: Add dynamic resource allocation(DRA) plugin #3799

Uh oh!

Conversation

JesseStutler commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Background

Verfication

Uh oh!

googs1025 commented Oct 31, 2024

Uh oh!

googs1025 Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

JesseStutler Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

JesseStutler Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

googs1025 Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

JesseStutler Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

googs1025 Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

JesseStutler Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

stale bot commented Apr 25, 2025

Uh oh!

hwdef commented May 5, 2025

Uh oh!

Monokaix commented May 23, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Monokaix commented May 27, 2025

Uh oh!

volcano-sh-bot commented May 27, 2025

Uh oh!

wangyang0616 commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

JesseStutler commented Oct 31, 2024 •

edited

Loading