Issue in LessEqual affecting enqueueability decision of proportion plugin (in v1.4.0), requesting hotfix to v1.4.0

[kye308]

What happened:

When there are no jobs running in a queue, and the queue capability specifies a ScalarResource (i.e. gpu) attribute other than cpu/memory, the proportion plugin will always chose to reject enqueueing new jobs.

Problematic comparison code in proportion plugin:

volcano/pkg/scheduler/plugins/proportion/proportion.go

Line 295 in 8423cb5

inqueue := minReq.Add(attr.allocated).Add(attr.inqueue).LessEqual(api.NewResource(queue.Queue.Spec.Capability), api.Infinity)

Bug (?) in LessEqual:

volcano/pkg/scheduler/api/resource_info.go

Line 349 in 8423cb5

if leftValue == -1 || !lessEqualFunc(leftValue, rightValue, minResource) {

What you expected to happen:

For a queue with no jobs, the allocated and inqueue values were both 0 CPU, 0 Gi Mem.

Our queue specifies a capability of the form

    cpu: X
    ephemeral-storage: Y Gi
    memory: Z Gi
    nvidia.com/gpu: N

nvidia.com/gpu is identified as a ScalarResource and appears to trigger the if condition here:

volcano/pkg/scheduler/api/resource_info.go

Line 349 in 8423cb5

if leftValue == -1 || !lessEqualFunc(leftValue, rightValue, minResource) {

Thus any submitted jobs will be stuck in Pending state.

How to reproduce it (as minimally and precisely as possible):

Try to schedule any job with the scheduler config below and a queue configured as specified above on a cluster with no running jobs.

Anything else we need to know?:

scheduler config

actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
  - name: conformance
- plugins:
  - name: proportion
  - name: predicates
    arguments:
      predicate.GPUSharingEnable: false
      predicate.CacheEnable: false
      predicate.ProportionalEnable: false

Environment:

• Volcano Version: v1.4.0
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[kye308]

The observed issue was fixed when we added this patch: #1769. Can we add this to v1.4.0 as well.

[Thor-wl]

Can we add this to v1.4.0 as well.

Yes, glad to see that.

[Thor-wl]

/cc @hwdef Can you help for that?

[hwdef]

@Thor-wl
ok, I'll check this.It may have to wait until Monday.

[hwdef]

@kye308 @Thor-wl I think this issue can be closed.

[kye308]

thank you!