Graceful node shutdown

[bobbypage]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubelet can detect and gracefully terminate pods during a node shutdown. Ref: Kubelet should handle node shutdown  kubernetes#91472
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown
• Discussion Link:
• Primary contact (assignee): @bobbypage
• Responsible SIGs: sig/node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (1.20)
  • Beta release target (1.21)
  • Stable release target (TBD)

• Alpha (related PRs, can be assigned to a specific track later)
  • KEP (k/enhancements) update PR(s):
    • Add node shutdown KEP #2001
    • Mark graceful node shutdown KEP as implementable #2063
    • Update Graceful Node Shutdown KEP 1.20 Alpha #2295
  • Issues and Code (k/k) update PR(s): Implement Graceful Node Shutdown in Kubelet kubernetes#96129
  • Docs (k/website) update(s): Graceful node shutdown (alpha 1.20 feature) docs [KEP: 2000] website#24918
• Beta (related PRs, can be assigned to a specific track later)
  • KEP (k/enhancements) update PR(s):
    • KEP 2000: Update graceful shutdown KEP for beta #2472
    • KEP-2000: Update graceful node shutdown KEP regarding pod termination  #3131
  • Issues and Code (k/k) update PR(s):
    • Add GracefulNodeShutdown e2e test kubernetes#98658
    • Promote kubelet graceful node shutdown to beta kubernetes#99735
  • Docs (k/website) update(s): add 1.21 graceful node shutdown blog post website#27335

• GA
  • KEP (k/enhancements) update PR(s):
    • KEP-2000: Add note in KEP Proposal for shutdown -h now systemd version compatibility #4640
  • Issues and Code (k/k) update PR(s): some test and bugfix after beta
    • 1.22 Improve terminated pod message when node is shutting down kubernetes#102840
    • Fix dbus config path for GracefulNodeShutdown e2e kubernetes#103566
    • 1.32 Ensure volumes are unmounted during graceful node shutdown kubernetes#125070
    • Blockers
      • DaemonSet controller and Graceful Node Shutdown manager disagree when making workloads placement decision kubernetes#122912
      • Graceful node drain results in never-cleaned-up pods in various Error/Completed phases kubernetes#122122
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[bobbypage]

/sig node

[kikisdeliveryservice]

@bobbypage Do you want this in 1.20 or 1.21?

If so I added a comment here: #2001 (review)

Since you will need to: Update the related Issue to get it in the milestone, add graduation criteria (alpha, beta, etc) and mark this as implementable. by Enhancements Freeze October 6th

[bobbypage]

Hi @kikisdeliveryservice

I'd like to target 1.20 if possible, I've updated the KEP and issue as such, please see the corresponding comment there: #2001 (comment)

Please let me know if anything else is missing.

Thanks for your help!

[kikisdeliveryservice]

Hi @bobbypage !

KEP will need graduation criteria for alpha and in the kep.yaml update to show 1.20 as being alpha.

After that you're good to go =)

[bobbypage]

Thanks @kikisdeliveryservice -- I updated the KEP with that info

[kikisdeliveryservice]

Hi @bobbypage

The KEP seems to be missing graduation criteria for alpha:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown#graduation-criteria

Please update and merge by October 6th

[kikisdeliveryservice]

That was fast! The KEP is done & good to go for 1.20 👍

[kikisdeliveryservice]

Hi @bobbypage 😄

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

[annajung]

Hello @bobbypage 👋 , 1.20 Docs lead here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[annajung]

Hi @bobbypage

The docs placeholder deadline is this Friday. Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline

Also, please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for Review

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[bobbypage]

Hi @annajung I've created a placeholder PR here: kubernetes/website#24918

The PR for this is still in flight, so I'm not completely sure if it'll make the timeline for 1.20 cut, but will try :). Otherwise, we'll have to target this for 1.21.

[annajung]

Thanks for the update! No worries, you still have a few more days before the code freeze.
The release team will keep track of this until the last day possible, hope you can get it in for 1.20

[kikisdeliveryservice]

Hey @bobbypage

I see that kubernetes/kubernetes#96129 is still open and in active review. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

[kikisdeliveryservice]

Looks like the PR merged! 🥳

[SergeyKanzhelev]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[SergeyKanzhelev]

/remove-lifecycle stale

[SergeyKanzhelev]

What is preventing this from GA-ing? It is entering the perma-beta state, and we try not to do it.

[sftim]

Should we provide Windows support for handling node shutdown gracefully?

[tuibeovince]

@SergeyKanzhelev @mrunalp will this still be a target for GA-ing in v1.30? Thank you

[atiratree]

There are still outstanding issues that need to be resolved before GA-ing, like kubernetes/kubernetes#122912

[tuibeovince]

I really want to have a clearer picture of the current status of this feature.

As to my knowledge:

the major issues that are currently blocking the GA are

• DaemonSet controller and Graceful Node Shutdown manager disagree when making workloads placement decision kubernetes#122912
• Graceful node drain results in never-cleaned-up pods in various Error/Completed phases kubernetes#122122

and we have an upcoming Declarative Node Maintenance that will have some overlap with GracefulNodeShutdown
#4212

Is this correct?
Thank you!

[atiratree]

There are a few more relevant issues, please see the Motivation section of the Declarative Node Maintenance #4213, but I am not sure about the severity.

[kannon92]

@smarterclayton Are you planning on working on the endpoint issue with graceful node shutdown in 1.32?

[haircommander]

/milestone v1.32
/label lead-opted-in

[kwilczynski]

/assign kwilczynski

[kwilczynski]

/unassign mvortizr

[tjons]

Hello @kwilczynski @bobbypage 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage stable for v1.32 (correct me, if otherwise).

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday, October 3rd, 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP has a production readiness review that has been completed and merged into k/enhancements.
• Update the KEP to use the latest template. There is a new PRR question added to the template: Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?. Please ensure that all required sections (not marked Optional in https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/README.md`) are present.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[kwilczynski]

@tjons, thank you for a friendly poke. We will do our best to meet the deadline.

[kwilczynski]

@haircommander, after talking with @mrunalp recently, I believe we will keep this feature as Beta in 1.32, too.

Additionally, @atiratree has also expressed concerns about a potential promotion of Graceful Node Shutdown in 1.32, especially from the SIG Apps point of view.

@tjons, we probably won't be pursuing 1.32 further here, but let's wait for some consensus about it.

[tjons]

/milestone clear
/remove-label lead-opted-in