Put tasks that get resource quota error in pending #56
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michaelawilkins
changed the title
akhurana001
reviewed
May 5, 2021
| @@ -455,20 +455,19 @@ func shouldRetry(app *v1beta2.SparkApplication) bool { | |||
| } | |||
| case v1beta2.PendingSubmissionState: | |||
| var interval int64 = 257 | |||
| if app.Spec.Mode == v1beta2.ClientMode && hasRetryIntervalPassed(&interval, app.Status.SubmissionAttempts, app.CreationTimestamp) && app.Status.SubmissionAttempts < 14 { | |||
| if app.Spec.Mode != v1beta2.ClusterMode && hasRetryIntervalPassed(&interval, app.Status.SubmissionAttempts, app.CreationTimestamp) && app.Status.SubmissionAttempts <= 14 { | |||
There was a problem hiding this comment.
let's not hardcode 14 and use the field set as part of CRD Spec. Defaulting if not set in CRD is fine.
There was a problem hiding this comment.
okay i can set to submissionRetries but if its not in crd it will break the operator
There was a problem hiding this comment.
You can default in https://github.com/lyft/spark-on-k8s-operator/blob/master/pkg/apis/sparkoperator.k8s.io/v1beta2/defaults.go to be safe ?
There was a problem hiding this comment.
going to test in flyte staging before merging this
There was a problem hiding this comment.
rohit-menon
approved these changes
May 5, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this change there is an expoential retry for failures regarding resource quota, they are failed and then the spark application is resubmitted so the flyte ui just shows task as failed. It is requested that the task not fail and retry all 14 submissions using the same entry. This now done by keeping the spark application in pending submission until all 14 retries are done so on the flyte ui it is seen as still running.
https://jira.lyft.net/browse/DATAHELP-4360
https://jira.lyft.net/browse/DATAHELP-4388
Before and After on flyte UI using functional test
before: https://flyte-staging.lyft.net/console/projects/flytefunctionaltests/domains/development/executions/vejslkh4vp
after: https://flyte-staging.lyft.net/console/projects/flytefunctionaltests/domains/development/executions/t7gaw9zu5i
Screenshots from running on sonata-staging--
Run #13
After 14 Runs