Quotas for Ephemeral Storage

[RobertKrawitz]

Enhancement Description

• Use filesystem quotas to monitor local ephemeral storage utilization
• Kubernetes Enhancement Proposal: 1029-ephemeral-storage-quotas
• Primary contact (assignee): @RobertKrawitz @pacoxu
• Responsible SIGs: SIG-Node
• Reviewer(s) - @dashpole @derekwaynecarr
• Approver: #derekwaynecarr @dchen1107
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.15
  • Beta release target (x.y): 1.31 (User Namespace Support User Namespaces in pods #127 is beta in v1.30)
    • promoted in v1.25.0 and then reverted in v1.25.1
  • Stable release target (x.y): v1.33 or later
• Alpha
  • KEP (k/enhancements) update PR(s): Updates from SIG-Node meeting 20181204 #646
  • Code (k/k) update PR(s): Ephemeral storage monitoring via filesystem quotas kubernetes#66928
  • Docs (k/website) update(s): Add support for quotas for ephemeral storage monitoring. website#14268
• Reverted Beta (first-try: promoted in v1.25.0 and then reverted in v1.25.1)
  • KEP (k/enhancements) update PR(s): promote ephemeral-storage-quotas to beta in 1.25 #2697
  • Code (k/k) update PR(s):
    • promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#107329
    • add VolumeStatCalDuration metrics for fsquato monitoring benchmark kubernetes#107201
    • assignQuota checks if the underlying medium supports quotas and if so setting it kubernetes#107302
    • add warning log if volume calculation took too long than 1 second kubernetes#107490
  • Docs(k/website) updates: Revert: promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta website#36401
  • Revert PR: Revert "promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta" kubernetes#112076, Changelog-1.25: add 1.25.0 known issues section kubernetes#112106, LocalStorageCapacityIsolationFSQuotaMonitoring revert to alpha for regression #3485
• Beta (second-try) 1.31
  • KEP (k/enhancements) update PR(s): [v1.29] ephemeral-storage-quotas: repromote to beta #3821
    • sig-node/1029-ephemeral-storage-quotas/*: #4699
  • Code (k/k) update PR(s):
    • Add CI for quota monitoring test-infra#28614
    • 1.27 fsquota: only generate pod uuid is nil kubernetes#112624
    • 1.27 kubelet: Fix fs quota monitoring on volumes kubernetes#115314
    • [pending UserNamespace beta]promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#112626
    • [pending UserNamespace beta]promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#112626 (comment) kernel @haircommander pointed out that it's in a user namespace it can't change the projid and if not, project ID could be changed. So we want to wait for User Namespace beta before promoting this to beta.
    • 1.31 pkg/volume/*: Enable quotas in user namespace kubernetes#126014
  • Docs (k/website) update(s):
    • [WIP] Update LocalStorageCapacityIsolationFSQuotaMonitoring to beta website#43455
    • Promote ephemeral storage quota to beta(1.31) website#46967

[RobertKrawitz]

/sig node

[derekwaynecarr]

per sig-node discussion, this is good.

the kep will merge here: #646

/milestone v1.15

[kacole2]

@RobertKrawitz This is OK to track for 1.15. However, please update the KEP with established graduation criteria for Beta and Alpha when possible.

/stage alpha

[makoscafee]

Hey, @RobertKrawitz @derekwaynecarr 👋 I'm the v1.15 docs Lead.
Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[RobertKrawitz]

Yes, this will require new documentation.

I will open the website PR today.

[kacole2]

Hi @RobertKrawitz @derekwaynecarr . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[makoscafee]

Hey, @RobertKrawitz @derekwaynecarr .Just a friendly reminder we're looking for at least a draft/placeholder PR against k/website (branch dev-1.15) due by Thursday, May 30th 2019 @ EOD PST.

[RobertKrawitz]

@makoscafee The k/website PR is kubernetes/website#14268

[kacole2]

Hi @RobertKrawitz @derekwaynecarr , today is code freeze. I do not see a reply for any k/k PRs to track for this merge. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. If there is no response, or you respond with PRs to track and they are not merged by EOD PST, this will be dropped from the 1.15 Milestone. After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[RobertKrawitz]

The PR for this is kubernetes/kubernetes#66928

[RobertKrawitz]

/hold cancel

[kacole2]

Hi @RobertKrawitz , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.6 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[kcmartin]

Hello @RobertKrawitz -- 1.17 Enhancement Shadow here! 🙂

I wanted to reach out to see if this enhancement will be graduating to alpha/beta/stable in 1.17?


Please let me know so that this enhancement can be added to 1.17 tracking sheet.

Thank you!

🔔Friendly Reminder

The current release schedule is

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[PannagaRao]

I will be looking into this

[haircommander]

PR for beta (third try): #4699

[mrunalp]

/milestone v1.31
/label lead-opted-in

[dipesh-rawat]

Hello @RobertKrawitz @pacoxu @PannagaRao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting for stage beta for 1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• Please update kep.yaml to reflect beta for v1.31

It looks like #4699 will address most of these issues.

• Get sig-node/1029-ephemeral-storage-quotas/*: #4699 merged before the enhancements freeze.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hello @RobertKrawitz @pacoxu @PannagaRao 👋, Enhancements team here.

Now that PR #4699 has been merged, all the KEP requirements are in place and merged into k/enhancements.

Before the enhancement freeze, it would be appreciated if following nits could be addressed:

• Update issue description to mention the following:
  • KEP (k/enhancements) update PR(s): sig-node/1029-ephemeral-storage-quotas/*: #4699

Aside from the minor nit mentioned above, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is now marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[hacktivist123]

Hello @RobertKrawitz 👋, 1.31 Docs Shadow here.

Does this enhancement work planned for 1.31 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.31 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday June 27, 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarised with the docs requirement for the release.

Thank you!

[a-mccarthy]

Hi @RobertKrawitz,

👋 from the v1.31 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!
Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[Princesso]

Hi @RobertKrawitz @pacoxu @PannagaRao ,
1.31 Doc lead here. I noticed that this Enhancement work already has a draft PR here, however, the PR is closed. Can you re-open the PR against dev-1.31 branch in the k/website repo. This deadline for having a draft PR up is Thursday June 27, 2024 18:00 PDT.

[Princesso]

Hi @RobertKrawitz @pacoxu @PannagaRao, this is a gentle reminder that today is the deadline for having the doc draft PR opened against dev-1.31.

[haircommander]

here's the PR kubernetes/website#46967

[a-mccarthy]

hi @haircommander, friendly reminder about the upcoming blog opt-in and placeholder deadline on July 3rd. Please open a blog placeholder PR if you are interested in contributing a blog.

[dipesh-rawat]

Hey again @RobertKrawitz @pacoxu @PannagaRao 👋, 1.31 Enhancements team here,

Just checking in as we approach code freeze at at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• pkg/volume/*: Enable quotas in user namespace kubernetes#126014

Additionally, please let me know if there are any other PRs in k/k not listed in the descrption that we should track for this KEP, so that we can maintain accurate status.

For this KEP, we would need to do the following:

• Ensure all PRs are prepared for merging (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

If you anticipate missing code freeze, you can file an exception request in advance.

The status of this enhancement is marked as at risk for code freeze.

Edited: To include the PR related to the KEP

[dipesh-rawat]

Hey again @RobertKrawitz @pacoxu @PannagaRao 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach code freeze in around two weeks time, at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

The current status of this enhancement is marked as at risk for code freeze. There are a few requirements mentioned in the comment #1029 (comment) that still need to be completed.

If you anticipate missing code freeze, you can file an exception request in advance.

[dipesh-rawat]

Hey again @RobertKrawitz @pacoxu @PannagaRao 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach code freeze next week, at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

The current status of this enhancement is marked as at risk for code freeze. There are a few requirements mentioned in the comment #1029 (comment) that still need to be completed.

If you anticipate missing code freeze, you can file an exception request in advance.

[dipesh-rawat]

Hey again @RobertKrawitz @pacoxu @PannagaRao 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach code freeze in less than 2 days, at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

The current status of this enhancement is marked as at risk for code freeze. There are a few requirements mentioned in the comment #1029 (comment) that still need to be completed.

If you anticipate missing code freeze, you can file an exception request in advance.

[dipesh-rawat]

Hey again @RobertKrawitz @pacoxu @PannagaRao 👋, 1.31 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

With all the implementation(code related) PRs merged as per the issue description:

• pkg/volume/*: Enable quotas in user namespace kubernetes#126014

Additionally, please let me know if there are any other PRs in k/k not listed in the description that we should track for this KEP, so that we can maintain accurate status.

This enhancement is now marked as tracked for code freeze for the 1.31 Code Freeze!

[kannon92]

Hi, what are the plans for 1.32?

[haircommander]

this only just went to beta so I presume nothing until 1.33

[tjons]

Inadvertently added this to the 1.32 tracking board

/remove-label lead-opted-in

[pacoxu]

this only just went to beta so I presume nothing until 1.33

As this depends on the User Namespace feature and is beta but by default false/disabled, I assume that the next step is beta 2, default true which needs to be done after UserNamespacesSupport is beta and by default true.