KEDA deleting Job objects unnecessarily during startup #6233
Comments
|
It's an interesting point and I'm not 100% sure if this has been solved in recent versions. Which KEDA version are you using? |
|
Used keda version $ git diff
diff --git a/controllers/keda/scaledjob_controller.go b/controllers/keda/scaledjob_controller.go
index 98c1ce87..1c09f232 100755
--- a/controllers/keda/scaledjob_controller.go
+++ b/controllers/keda/scaledjob_controller.go
@@ -279,11 +279,22 @@ func (r *ScaledJobReconciler) deletePreviousVersionScaleJobs(ctx context.Context
return "Cannot get list of Jobs owned by this scaledJob", err
}
+ generation := strconv.FormatInt(scaledJob.GetGeneration(), 10)
if len(jobs.Items) > 0 {
logger.Info("RolloutStrategy: immediate, Deleting jobs owned by the previous version of the scaledJob", "numJobsToDelete", len(jobs.Items))
}
for _, job := range jobs.Items {
job := job
+ jobGenerationStr, label_present := job.GetLabels()["scaledjob-generation"]
+ if label_present {
+ logger.Info("scaledjob-generation label present", "generation", generation,
+ "jobGeneration", jobGenerationStr, "job.Generation", job.Generation)
+ jobGeneration, _ := strconv.ParseInt(jobGenerationStr, 10, 64)
+ if jobGeneration >= scaledJob.GetGeneration() {
+ logger.Info("Not deleting current generation jobs")
+ continue
+ }
+ }
propagationPolicy := metav1.DeletePropagationBackground
if scaledJob.Spec.Rollout.PropagationPolicy == "foreground" {
diff --git a/pkg/scaling/executor/scale_jobs.go b/pkg/scaling/executor/scale_jobs.go
index 52f7ea37..1fcce707 100644
--- a/pkg/scaling/executor/scale_jobs.go
+++ b/pkg/scaling/executor/scale_jobs.go
@@ -126,12 +126,14 @@ func (e *scaleExecutor) generateJobs(logger logr.Logger, scaledJob *kedav1alpha1
}
scaledJob.Spec.JobTargetRef.Template.Labels["scaledjob.keda.sh/name"] = scaledJob.GetName()
+ scaledJobGeneration := strconv.FormatInt(scaledJob.GetGeneration(), 10)
labels := map[string]string{
"app.kubernetes.io/name": scaledJob.GetName(),
"app.kubernetes.io/version": version.Version,
"app.kubernetes.io/part-of": scaledJob.GetName(),
"app.kubernetes.io/managed-by": "keda-operator",
"scaledjob.keda.sh/name": scaledJob.GetName(),
+ "scaledjob-generation": scaledJobGeneration,
}
for key, value := range scaledJob.ObjectMeta.Labels {
labels[key] = valueAnalysis with sample producer consumer with rabbitmq:Upon updating the manifest of ScaledJob Deletion of older job occurHowever, when keda-operator is restarted, Older jobs were not deleted Not deleting current generation jobs ; Because
|
|
However, when I forked it, and tried to implement above logic into the current code
|
|
so, I guess that we can close this as it's solved and the fix will be shipped during next release, right? |
|
Yes @JorTurFer, this issue can be closed. Thank you! for your time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
KEDA is deleting
Jobscreated byScaledJobswith rollout strategydefaulton startup. Rollout strategydefaultis supposed to terminate existing Jobs whenever a ScaledJob is being updated , but KEDA appears to incorrectly interpret reconciliation related to initial controller watch establishment as updates, causingJobswith current configuration to be terminated prematurely.This is causing artificial job delays for users, especially if KEDA restarts frequently while Jobs are in progress.
A possible solution is to modify KEDA to distinguish between ScaledJob modifications and controller startup.
The text was updated successfully, but these errors were encountered: