optimizate jobflow controller to reduce invalid reconcile #3441

calvin0327 · 2024-04-26T08:59:18Z

I found a some bit err message when using jobflow feature, I create a jobflow resource ref:
https://github.com/volcano-sh/volcano/blob/master/example/jobflow/JobFlow.yaml
https://github.com/volcano-sh/volcano/blob/master/example/jobflow/JobTemplate.yaml

here's controller manager logs:

[root@master01 ~]# kubectl logs -n volcano-system volcano-controllers-744bc4796d-jbncj | grep ^E
E0425 10:34:49.690189       1 jobflow_controller_action.go:69] Failed to update status of JobFlow default/test: Operation cannot be fulfilled on jobflows.flow.volcano.sh "test": the object has been modified; please apply your changes to the latest version and try again
E0425 10:34:49.707411       1 jobflow_controller_action.go:69] Failed to update status of JobFlow default/test: Operation cannot be fulfilled on jobflows.flow.volcano.sh "test": the object has been modified; please apply your changes to the latest version and try again
E0425 10:34:50.321009       1 queue_controller_action.go:85] Failed to update status of Queue default: Operation cannot be fulfilled on queues.scheduling.volcano.sh "default": the object has been modified; please apply your changes to the latest version and try again.
E0425 10:34:51.395417       1 queue_controller_action.go:85] Failed to update status of Queue default: Operation cannot be fulfilled on queues.scheduling.volcano.sh "default": the object has been modified; please apply your changes to the latest version and try again.
E0425 10:35:04.721574       1 jobflow_controller_action.go:69] Failed to update status of JobFlow default/test: Operation cannot be fulfilled on jobflows.flow.volcano.sh "test": the object has been modified; please apply your changes to the latest version and try again
E0425 10:35:04.736015       1 jobflow_controller_action.go:69] Failed to update status of JobFlow default/test: Operation cannot be fulfilled on jobflows.flow.volcano.sh "test": the object has been modified; please apply your changes to the latest version and try again
E0425 10:35:05.568771       1 jobflow_controller_action.go:69] Failed to update status of JobFlow default/test: Operation cannot be fulfilled on jobflows.flow.volcano.sh "test": the object has been modified; please apply your changes to the latest version and try again
E0425 10:35:05.581852       1 jobflow_controller_action.go:69] Failed to update status of JobFlow default/test: Operation cannot be fulfilled on jobflows.flow.volcano.sh "test": the object has been modified; please apply your changes to the latest version and try again
E0425 10:35:20.711708       1 queue_controller_action.go:85] Failed to update status of Queue default: Operation cannot be fulfilled on queues.scheduling.volcano.sh "default": the object has been modified; please apply your changes to the latest version and try again.
E0425 10:35:21.731150       1 queue_controller_action.go:85] Failed to update status of Queue default: Operation cannot be fulfilled on queues.scheduling.volcano.sh "default": the object has been modified; please apply your changes to the latest version and try again.
E0425 10:35:34.692296       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-b, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-b> is not ready
E0425 10:35:34.695945       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-b, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-b> is not ready
E0425 10:35:34.698687       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-c, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-c> is not ready
E0425 10:35:34.701790       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-c, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-c> is not ready
E0425 10:35:34.707817       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-d, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-d> is not ready
E0425 10:35:34.712693       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-d, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-d> is not ready
E0425 10:35:34.714371       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-e, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-e> is not ready
E0425 10:35:34.715187       1 jobflow_controller_action.go:300] Failed to delete job of JobFlow default/test: jobs.batch.volcano.sh "test-a" not found
E0425 10:35:34.715210       1 jobflow_controller_action.go:46] Failed to delete jobs of JobFlow default/test: jobs.batch.volcano.sh "test-a" not found
E0425 10:35:34.717377       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-e, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-e> is not ready
E0425 10:35:34.723456       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-a, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: job <default/test-a> is not ready
E0425 10:35:34.728548       1 job_controller.go:334] Failed to get job by <Queue: , Job: default/test-a, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0> from cache: failed to find job <default/test-a>

The pr focuses only on jobflow_controllers.go errors.

volcano-sh-bot · 2024-04-26T08:59:25Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign shinytang6
You can assign the PR to them by writing /assign @shinytang6 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/controllers/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: calvin <wen.chen@daocloud.io>

calvin0327 · 2024-04-26T09:03:39Z

pkg/controllers/jobflow/jobflow_controller_action.go

@@ -63,7 +65,26 @@ func (jf *jobflowcontroller) syncJobFlow(jobFlow *v1alpha1flow.JobFlow, updateSt
 	}
 	jobFlow.Status = *jobFlowStatus
 	updateStateFn(&jobFlow.Status, len(jobFlow.Spec.Flows))
-	_, err = jf.vcClient.FlowV1alpha1().JobFlows(jobFlow.Namespace).UpdateStatus(context.Background(), jobFlow, metav1.UpdateOptions{})
+
+	err = retry.RetryOnConflict(retry.DefaultRetry, func() error {


Added the retry mechanism after resource version conflicts to avoid the next reconcile.

It's truly a problem, but this way seems will be more time consuming.

calvin0327 · 2024-04-26T09:07:03Z

pkg/controllers/jobflow/jobflow_controller_action.go

@@ -281,6 +302,12 @@ func (jf *jobflowcontroller) deleteAllJobsCreatedByJobFlow(jobFlow *v1alpha1flow
 	for _, job := range jobList {
 		err := jf.vcClient.BatchV1alpha1().Jobs(jobFlow.Namespace).Delete(context.Background(), job.Name, metav1.DeleteOptions{})
 		if err != nil {
+			if apierrors.IsNotFound(err) {


ingnore this error return if the job no longer exist.

calvin0327 · 2024-04-26T09:07:15Z

/auto-cc

calvin0327 · 2024-04-28T01:42:16Z

@lowang-bh @hwdef PTAL

volcano-sh-bot · 2024-10-24T06:08:19Z

@calvin0327: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stale · 2025-02-01T01:07:46Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

hwdef · 2025-02-10T09:10:00Z

I'll check this later.
Thanks for your contribution.

hwdef · 2025-02-10T09:10:49Z

@calvin0327
Are you still doing this? Please rebase the master branch.

stale · 2025-04-25T23:44:45Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

volcano-sh-bot requested review from hwdef and hzxuzhonghu April 26, 2024 08:59

volcano-sh-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 26, 2024

optimizate jobflow controller to reduce invalid reconcile

a6fad98

Signed-off-by: calvin <wen.chen@daocloud.io>

calvin0327 force-pushed the optimizate-workflow-controller branch from fb66ac0 to a6fad98 Compare April 26, 2024 09:01

calvin0327 commented Apr 26, 2024

View reviewed changes

volcano-sh-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 24, 2024

stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 1, 2025

stale bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 10, 2025

stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 25, 2025

stale bot closed this May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

optimizate jobflow controller to reduce invalid reconcile #3441

optimizate jobflow controller to reduce invalid reconcile #3441

Uh oh!

calvin0327 commented Apr 26, 2024 •

edited

Loading

Uh oh!

volcano-sh-bot commented Apr 26, 2024

Uh oh!

calvin0327 Apr 26, 2024

Uh oh!

Monokaix May 23, 2024

Uh oh!

calvin0327 Apr 26, 2024 •

edited

Loading

Uh oh!

calvin0327 commented Apr 26, 2024

Uh oh!

calvin0327 commented Apr 28, 2024

Uh oh!

volcano-sh-bot commented Oct 24, 2024

Uh oh!

stale bot commented Feb 1, 2025

Uh oh!

hwdef commented Feb 10, 2025

Uh oh!

hwdef commented Feb 10, 2025

Uh oh!

stale bot commented Apr 25, 2025

Uh oh!

Uh oh!

optimizate jobflow controller to reduce invalid reconcile #3441

optimizate jobflow controller to reduce invalid reconcile #3441

Uh oh!

Conversation

calvin0327 commented Apr 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

volcano-sh-bot commented Apr 26, 2024

Uh oh!

calvin0327 Apr 26, 2024

Choose a reason for hiding this comment

Uh oh!

Monokaix May 23, 2024

Choose a reason for hiding this comment

Uh oh!

calvin0327 Apr 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

calvin0327 commented Apr 26, 2024

Uh oh!

calvin0327 commented Apr 28, 2024

Uh oh!

volcano-sh-bot commented Oct 24, 2024

Uh oh!

stale bot commented Feb 1, 2025

Uh oh!

hwdef commented Feb 10, 2025

Uh oh!

hwdef commented Feb 10, 2025

Uh oh!

stale bot commented Apr 25, 2025

Uh oh!

Uh oh!

calvin0327 commented Apr 26, 2024 •

edited

Loading

calvin0327 Apr 26, 2024 •

edited

Loading