Node memory swap support

[ehashman]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes nodes support swap memory.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap
• Discussion Link: https://docs.google.com/document/d/1CZtRtC8W8FwW_VWQLKP9DW_2H-hcLBcH9KBbukie67M/edit# Kubelet/Kubernetes should work with Swap Enabled kubernetes#53533
• Primary contact (assignee): @iholder101 @kannon92 @pacoxu
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.22
  • Beta1 release target (x.y): 1.28
  • Beta2 release target (x.y): 1.29-1.31
  • Stable release target (x.y): 1.32
• Alpha
  • KEP (k/enhancements) update PR(s): Node system swap support #2602
  • Code (k/k) update PR(s):
    • CI/test infra changes: Setup new node image with swap enabled test-infra#22602 Add swap periodics for node test-infra#22809
    • CRI and kubelet changes Alpha node swap support kubernetes#102823
  • Docs (k/website) update PR(s): KEP-2400: Alpha node support for swap memory website#28838
• Beta1
  • KEP (k/enhancements) update PR(s):
    • KEP-2400: Update swap KEP for 1.23 beta #2858
    • KEP-2400: Graduate swap to Beta 1 #3957
  • Issues and Code (k/k) update PR(s):
    • Support cgroups v2 with auto-calculated LimitedSwap for Burstable QoS pods
      • [KEP-2400] Enable Swap Support for Cgroup v2 Only kubernetes#119427
      • [KEP-2400] Enable auto-calculated LimitedSwap for Burstable QoS pods kubernetes#119428
      • Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
    • Add Swap stats to /stats/summary and /metrics/resource
      • [KEP-2400] Add swap memory to the Kubelet stats API kubernetes#119425
      • Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
    • Determine a set of metrics for measuring node performance with swap enabled
      • [KEP-2400] Determine a set of metrics for measuring node performance with swap enabled kubernetes#119424
      • Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
      • Add a metric for machine swap capacity google/cadvisor#3339
    • Improve test coverage and make sure tests are green
      • [KEP-2400] Swap: improve test coverage and make sure tests are green kubernetes#119430
      • Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
  • Docs (k/website) update(s):
    • Swap graduates to Beta1: add a blog-post website#42009
    • Swap graduates to Beta1: add documentation and update feature gates website#42167
  • Misc
    • Bugfix: MachineInfo Clone() to also clone "SwapCapacity" google/cadvisor#3293
    • Bump cadvisor version to v0.47.3 kubernetes#119225
  • Archive / Outdated
• Beta2
  • KEP (k/enhancements) update PR(s):
    • 1.29 Graduate swap to beta 2 in 1.29 #4275
    • 1.30 [KEP-2400] - Update KEP with findings from beta2 #4401
    • 1.30 [KEP-2400] update latest milestone for swap KEP #4484
  • Issues and Code (k/k) update PR(s):
    • Add swap conformance tests kubernetes#122175
    • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#122241
    • 1.30
      • [KEP-2400] add no swap as the default option for swap kubernetes#122745
      • [KEP-2400]: Fail limited swap cgroup v1 kubernetes#123738
    • 1.31
      • [Swap][Tests][KEP2400] Add swap serial stress tests, improve NodeConformance tests and adapt NoSwap behavior kubernetes#123557
      • [KEP2400] Avoid logging that swap cgroup controller is missing for every container kubernetes#123749
      • 1.31 [KEP-2400] Mount tmpfs memory-backed volumes with a noswap option if supported kubernetes#124060
      • 1.31 [KEP-2400]: Restrict access to swap for containers in high priority Pods kubernetes#125277
      • 1.31 [KEP-2400]: Swap e2e tests: skip swap stress tests if swap is not provisioned kubernetes#125402
  • Docs (k/website) update(s):
    • 1.30 [KEP-2400] Update swap with beta2 deprecations website#45128
• GA (related PRs, can be assigned to a specific track later)
  • KEP (k/enhancements) update PR(s):
    • [KEP-2400] Node swap updates, GA criterias and clarifications #4701
  • Issues and Code (k/k) update PR(s):
    • [KEP-2400] [Swap] [Discussion]: Add swap as a ResourceName for pod spec resources API kubernetes#120801
    • [KEP-2400] [Swap]: Verify memory pressure behavior with swap enabled kubernetes#120800
    • [KEP-2400] [Swap]: SwapConformance e2e testing kubernetes#120798.
    • [KEP-2400] Address swap impact on memory-backed volumes kubernetes#105978
    • Add a test for OOMKilled for LimitedSwap kubernetes#121820
    • Add the ability to set a system-reserved quantity of swap on a node
      • [KEP-2400] [Swap] Add the ability to set a system-reserved quantity of swap on a node kubernetes#120802
      • [KEP-2400] Add the ability to set a system-reserved quantity of swap on a node kubernetes#105019 dup by 120802
      • add --system-reserved support for swap(cgroup v2 only): only for burstable pods kubernetes#105271
    • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#119327
    • [Swap Feature] Investigate what happens if a cluster has some swap enabled nodes and others that are not enabled. kubernetes#122279
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig node

[karan]

@cookieisaac

[ehashman]

/stage alpha
/milestone v1.22

[jrsapi]

Greetings @ehashman!
Enhancement shadow checking in with a few reminders. 1.22 Enhancements Freeze starts at 23:59:59 pst on Thursday, May 13. A few items needing review for this KEP:

• Enhancement README

• PRR approved

• Node memory swap support #2400 updated with current target and alpha information.

Thanks!

[ehashman]

All of this is covered in #2602

[jrsapi]

Greetings @ehashman!
Thanks for the follow-up. After reviewing the KEP and PRR everything looks on target. The enhancement is marked at risk but once merged we can move its status to tracked. A reminder that tomorrow 5/13 the Enhancement freeze starts at 23:59:59 PST.

Thanks!

[ehashman]

Work breakdown for 1.22

• Documentation for enabling swap (TBD - @ehashman?)
• CI environment/test updates (@ike-ma)
  • Build images with swap for 2 Linux distros
  • Add jobs to test-infra that use the images and enable the swap feature flag/kubelet option for node e2e suite
  • See also Test Plan
• CRI and kubelet changes (@ehashman)
  • Complete API changes per KEP
  • Add e2e tests as appropriate

External to k8s but still need to happen:

• Containerd update to use new CRI
• CRI-O update to use new CRI

Once the above CRI updates happen, ensure that CI environment is using latest container runtimes with updated CRI.

[jrsapi]

Greetings @ehashman ,
Enhancement shadow checking with a reminder that we are 2 weeks away from code freeze (July 8, 2021). Can you like the k/k PR/s that are needed to implement this enhancement for the 1.22 milestone?

[jrsapi]

Greetings @ehashman,
A friendly reminder that code freeze is this Thursday, July 8th and we're tracking the following k/k PR:

• Alpha node swap support kubernetes#102823
  The PR has to be merged by code freeze and currently, this KEP is marked "At Risk".
  Also, Docs PR placeholder deadline is Friday, July 9th. Here are instructions on opening a PR in dev-1.22 branch.

Thanks!

[ehashman]

PR just merged. Docs placeholder is kubernetes/website#28838

[jrsapi]

Awesome! Thanks for the update. Moving this to "Tracked".

[ehashman]

We are good to go for 1.22! Docs complete.

[ehashman]

/milestone v1.23

[ehashman]

/stage beta

[mickeyboxell]

Hi @SergeyKanzhelev @kannon92 👋 v1.31 Enhancements team here. Could you help me understand what the plans are for this KEP in v1.31?

[kannon92]

@iholder101 can you help answer this question? AFAIK we are still keeping this in beta and there should be a KEP update soon.

[iholder101]

Hi @SergeyKanzhelev @kannon92 👋 v1.31 Enhancements team here. Could you help me understand what the plans are for this KEP in v1.31?

Hey!

The KEP is intended to GA at 1.32. Yesterday we've moved this issue to the 1.32 milestone to reflect that.
I'm now working on a PR to update the KEP.

[mickeyboxell]

@iholder101 Will there be any updates to the KEP during the v1.31 release cycle? If not, I'll remove it from our tracker.

[iholder101]

@iholder101 Will there be any updates to the KEP during the v1.31 release cycle? If not, I'll remove it from our tracker.

Hey @mickeyboxell!
There is an ongoing KEP update (#4701), but we aim to GA at 1.32, so I'm not sure it's important to track for the 1.31 release.

[sreeram-venkitesh]

/label tracked/no

[SergeyKanzhelev]

/unassign

[tjons]

Hello 👋 1.32 Enhancements Lead here,

I'm closing milestone 1.31 now,
If you have more work on this enhancement to complete in v1.32, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.32. Thanks!

/milestone clear

[iholder101]

/milestone v1.32

[k8s-ci-robot]

@iholder101: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.32

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[haircommander]

/milestone v1.32
/label lead-opted-in

[hoerup]

Shouldn't this ticket be labeled with stage/stable ?

[pacoxu]

/stage stable
/remove-stage beta

[jenshu]

Hello @iholder101 @kannon92 @pacoxu 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage stable for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• In the KEP readme, can you please ensure that the Release Signoff Checklist section contains all the checkboxes from the latest template?
• In the kep.yaml, the latest-milestone needs to be updated to v1.32
• The PRR needs a line with the stage stable and an approver.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[jenshu]

Hello 👋, 1.32 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in 1.32, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

/milestone clear

[pacoxu]

https://groups.google.com/d/msgid/kubernetes-sig-node/CA%2BoyvWYSqK_0T%2BUQTYNWTX9rNRWE9iOBjVpXffMW8pAu4ESnXg%40mail.gmail.com

• An exception is open for this KEP.

[kannon92]

We will keep this feature in beta and propose beta 3 for this with a focus on evictions.

kubernetes/kubernetes#128137

We also updated the KEP to reflect this new state so putting this here for clarity.

[tjons]

@kannon92 are there code changes for this KEP planned as part of beta 3?