Refactor reclaim action to process all tasks in a job

[GautamBytes]

What type of PR is this?

/kind improvement /kind bug

What this PR does / why we need it:

This PR fixes a critical bug in the scheduler's reclaim action where an entire queue could be incorrectly skipped during preemption.

The previous logic would evaluate only the first task of a starving job. If that single task failed a preliminary check (e.g., its preemptionPolicy was set to Never), the scheduler would discard the entire queue for that cycle. This prevented other valid, preemptable tasks in the same job, or other jobs in the same queue, from ever being considered for reclamation.

To fix this, the reclaim action's main loop has been refactored to mirror the robust nested queue -> job -> task structure found in the allocate action. This ensures:

• All candidate tasks within a job are correctly evaluated.
• The scheduler's preemption logic is more fair and correct.
• Performance is improved by removing an inefficient pop/push cycle.
• Code consistency across core scheduler actions is increased.

Which issue(s) this PR fixes:

Fixes #3738

Special notes for your reviewer:

The core of this change is refactoring the reclaim action's main loop to align with the existing, proven pattern in the allocate action. This not only fixes the bug but also improves code consistency and performance. A new unit test has been added to specifically cover this failure scenario.

Does this PR introduce a user-facing change?

None

[volcano-sh-bot]

@GautamBytes: The label(s) kind/improvement, kind//kind cannot be applied. These labels are supported: ``

Details

In response to this:

What type of PR is this?

/kind improvement /kind bug

What this PR does / why we need it:

This PR fixes a critical bug in the scheduler's reclaim action where an entire queue could be incorrectly skipped during preemption.

The previous logic would evaluate only the first task of a starving job. If that single task failed a preliminary check (e.g., its preemptionPolicy was set to Never), the scheduler would discard the entire queue for that cycle. This prevented other valid, preemptable tasks in the same job, or other jobs in the same queue, from ever being considered for reclamation.

To fix this, the reclaim action's main loop has been refactored to mirror the robust nested queue -> job -> task structure found in the allocate action. This ensures:

• All candidate tasks within a job are correctly evaluated.
• The scheduler's preemption logic is more fair and correct.
• Performance is improved by removing an inefficient pop/push cycle.
• Code consistency across core scheduler actions is increased.

Which issue(s) this PR fixes:

Fixes #3738

Special notes for your reviewer:

The core of this change is refactoring the reclaim action's main loop to align with the existing, proven pattern in the allocate action. This not only fixes the bug but also improves code consistency and performance. A new unit test has been added to specifically cover this failure scenario.

Does this PR introduce a user-facing change?

None

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign shinytang6
You can assign the PR to them by writing /assign @shinytang6 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[GautamBytes]

/assign @JesseStutler

@JesseStutler , can you help me figure out why this code_verify workflow is failing , i am having hard time fixing it . Everytime i change something to fix it , it shows completely new error. It is not happening with my other prs.

[lowang-bh]

please split pr into different prs or commits, each pr/commits implement only one function, such as refact, bugfix.

[JesseStutler]

/cc
/priority high

[kev1N916]

@JesseStutler I've opened a pull request to solve this issue at #4634

[volcano-sh-bot]

@GautamBytes: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JesseStutler]

@GautamBytes Hi, are you still working on this? You need to rebase the latest code and fix all the CIs

[GautamBytes]

@JesseStutler sorry , i got an internship and currently busy there . Would be great if anyone inherit my pr or asks for commit access and gets it merged!