Backoff Limit Per Index For Indexed Jobs

[jensentanlo]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add New Indexed Job backoff limit mode that is counted per index rather than per job

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3850-backoff-limits-per-index-for-indexed-jobs

• Discussion Link: An Indexed Job mode that allows every index to execute kubernetes#109712

• Primary contact (assignee): mimowo

• Responsible SIGs: apps

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.29
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s): Backoff limit per Job Index #3967
  • Code (k/k) update PR(s):
    • Support BackoffLimitPerIndex in Jobs kubernetes#118009
    • Extend the Job API for BackoffLimitPerIndex kubernetes#119294
  • Docs (k/website) update PR(s): Docs update for Job's backoff limit per index (alpha in 1.28) website#41921

• Beta

  • KEP (k/enhancements) update PR(s):
    • Update KEP-3850 "Backoff Limit Per Index" for Beta #4228
    • Update KEP3850 "BackoffLimitPerIndex for Indexed Jobs" #4321
  • Code (k/k) update PR(s):
    • Graduate BackoffLimitPerIndex to Beta kubernetes#121356
    • Introduce the job_finished_indexes_total metric kubernetes#121292
    • Indicate Job Backoff Limit Per Index reason consts are beta kubernetes#121471
    • Backoff limit per index e2e test kubernetes#121368
    • Add remaining e2e tests for Job BackoffLimitPerIndex based on KEP kubernetes#121633
    • Benchmark job with backoff limit per index kubernetes#121393
  • Docs (k/website) update(s): Docs update for Job Backoff Limit Per Index in Beta website#43388

[jensentanlo]

/sig apps
/wg batch

[alculquicondor]

/assign @mimowo

[alculquicondor]

In addition to configuring the backoff per index, we should probably have FailIndex as one of the actions for pod failure policies.

[soltysh]

/milestone v1.28
/stage alpha
/label lead-opted-in

[aramase]

Hello @mimowo 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC Friday, 16th June 2023.

This enhancement is targeting for stage alpha for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[mimowo]

@aramase I think the first point is addressed as the KEP has been merged: #3967.

[mimowo]

@aramase is there anything missing to make it tracked?

[Atharva-Shinde]

Hey @mimowo
With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you :)

[Rishit-dagli]

Hello @mimowo 👋, 1.28 Docs Lead here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[aramase]

Hey again @mimowo 👋

Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

Here’s the enhancement’s state for the upcoming code freeze:

• All the PRs that are related to your enhancement are linked in the above issue description (for tracking purposes). This includes code, tests, and documentation related PR/s.
• All code related PR/s are merged or are in merge-ready state ( i.e they have approved and lgtm labels applied) by the code freeze deadline. This includes any tests related PR/s too.

I see kubernetes/kubernetes#118009 PR in the issue description. If there are any other k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[Atharva-Shinde]

Hey @mimowo 👋 Enhancements Lead here,
With kubernetes/kubernetes#118009 and
kubernetes/kubernetes#119294 merged as per the issue description, this enhancement is now tracked for v1.28 Code Freeze!

[jensentanlo]

I performed some manual testing on this feature and saw everything working as expected, a short summary of the details are below if you're interested.

---

I ran on a local kind cluster (1.28) with alpha feature gate enabled, indexed jobs with completions = 1000,
mainly checking whether:

1. All pods ran to completion or failure
2. All failed indices are correctly recorded on the job object

Related to indexed jobs in general but not this specific feature, I was also interested in the delete behavior, because I've had trouble with bulk deletions of non-indexed jobs in the past, but it looks like everything was correctly cleaned up relatively quickly, even though I was churning through a couple indexed jobs (so thousands of pods) on my local machine.

[salehsedghpour]

/remove-label lead-opted-in

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[kannon92]

@mimowo if I am not mistaken, this feature should have a stage of beta.

It turns out I can update the label!

[kannon92]

/stage beta

[salehsedghpour]

Hi @soltysh, @mimowo, and @kannon92 , Enhancements Team here! Just wondering, if you are aiming to have this Enhancement in 1.30. If yes, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

[alculquicondor]

No plans to graduate in this release.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[alculquicondor]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mimowo]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mimowo]

/remove-lifecycle stale

[mbobrovskyi]

Hi folks! We are using this feature on kjobctl here. The use case requires moving to the next index, even if there's an issue with the previous one. So that we don't stop running the job when one index fails. It would be great move forward to stable version.