Add support for a drop-in kubelet configuration directory

[haircommander]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support for a kubelet drop-in configuration directory
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3983-drop-in-configuration
• Discussion Link: SIG-Node April 25th meeting SIG-Arch May 4 meeting
• Primary contact (assignee): @haircommander @sohankunkerkar
• Responsible SIGs: SIG-Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.30
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s): KEP-3983: Add support for a drop-in kubelet configuration directory #4031, KEP 3983: target to beta for 1.29 #4242
  • Code (k/k) update PR(s): cmd/kubelet: implement drop-in configuration directory for kubelet kubernetes#119390, Retarget drop-in kubelet configuration dir feature to Alpha kubernetes#121193
  • Docs (k/website) update PR(s): Update documentation to reflect kubelet.conf.d changes for k8s 1.28 website#42013, docs: prepare KubeletConfigDropinDir to Beta website#43542
• Beta
  • KEP (k/enhancements) update PR(s):
    • 1.30 KEP-3983: target to beta for 1.30 #4419
    • 1.31 KEP-3983: update the beta requirements #4588
    • 1.32 kep-3983: update Beta requirements before moving to GA #4889
  • Code (k/k) update PR(s):
    • 1.30 [KEP-4419]: promote KubeletConfigDropInDir feature to beta kubernetes#122907
  • Docs (k/website) update(s):
    • [dev-1.30] docs: Update KubeletConfigDropinDir doc information website#45665
    • KEP 3983: placeholder for beta retarget website#48482
  • Other related PRs
    • Merge kubelet configuration files using versioned serialized data kubernetes#127421
    • kubelet: add flag for upconverting a component config kubernetes#125344
• Stable
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[haircommander]

/sig node

[SergeyKanzhelev]

/milestone v1.28

Adding to the milestone for tracking based on discussions at sig node meeting.

[ffromani]

+1!!! (non binding)
@haircommander I'd love to help here to get this in kubelet

[SergeyKanzhelev]

/stage alpha

[SergeyKanzhelev]

/label lead-opted-in

[ruheenaansari34]

Hello @haircommander 👋, 1.28 Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting stage alpha for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to take care of:

• Merging KEP update PR
• The KEP number seems to be incorrect ie., should this be 3983 instead of 3893?

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[yuqi-zhang]

Hi @ruheenaansari34 , I've updated the ID, thanks for pointing that out!

Is there anything else needed at the moment other than getting consensus to merge the enhancement?

[ruheenaansari34]

@yuqi-zhang Please take care of the following before the Enhancement freeze to meet the requirements:

• The status in the kep.yaml should be implementable here -

  enhancements/keps/sig-node/3983-drop-in-configuration/kep.yaml

  Line 7 in 8e75faa

  status: provisional

• Merging KEP update PR

The status of this enhancement is still marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[yuqi-zhang]

Thanks! I've marked it as implementable after discussion with @haircommander . We will work to get it merged

[Atharva-Shinde]

Thanks @yuqi-zhang :)
With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze 🚀.
The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well.

[Rishit-dagli]

Hello @haircommander 👋, 1.28 Docs Lead here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Rishit-dagli]

Hey @haircommander , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[haircommander]

yes I will take care of it, thanks @Rishit-dagli !

[haircommander]

done! kubernetes/website#42013

[ruheenaansari34]

Hey again @haircommander 👋
Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[Atharva-Shinde]

Hey @haircommander 👋 Enhancements Lead here,
With kubernetes/kubernetes#119390 merged as per the issue description I updated above, this enhancement is now tracked for v1.28 Code Freeze. Thanks!

[Rishit-dagli]

Hello @haircommander wave: please take a look at Documenting for a release - PR Ready for Review to get your docs PR ready for review before Tuesday 25th July 2023. Thank you!

Ref: kubernetes/website#42013

[haircommander]

thank you! I have moved it out of draft

[katcosgrove]

Hey y'all! I'm swinging by for the v1.28 Docs team. Today is the docs deadline. Is there anything we can help you with to get this merged?

[haircommander]

sorry I am updating it now, I will get this done soon.

[npolshakova]

/remove-label lead-opted-in

[SergeyKanzhelev]

/milestone v1.29
/stage beta
/label lead-opted-in

[liggitt]

We need to address todos as a criteria to move forward with this KEP.

That's up to you. Those TODOs have to be resolved before kubelet revs its configuration format to v1beta2 or v1, certainly.

[SergeyKanzhelev]

We need to address todos as a criteria to move forward with this KEP.

That's up to you. Those TODOs have to be resolved before kubelet revs its configuration format to v1beta2 or v1, certainly.

Yep. There is little that will help us catch this when we have v1. So I think it's best fixed as part of this KEP.

[shecodesmagic]

Hello @haircommander 👋, v1.32 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage stable for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on
  Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[ffromani]

Hello! I'm among the KEP wranglers that sig-node set up to help sig-node KEPs make progress smoothly during the 1.32 timeframe. Hello @haircommander, per #3983 (comment) my understanding is we want to make progress in the 1.32 cycle but the GA proper is deferred to 1.33, is my understanding correct? IIUC the next steps are mostly address feedback and address open issue, is my understanding correct? thanks!

[sohankunkerkar]

Hello @haircommander, per #3983 (comment) my understanding is we want to make progress in the 1.32 cycle but the GA proper is deferred to 1.33, is my understanding correct? IIUC the next steps are mostly address feedback and address open issue, is my understanding correct? thanks!

@ffromani I think we need to update the beta requirements as you rightly pointed out and address TODOs mentioned by Jordan in the recent fix. As I understand it, we will proceed to GA once we have sufficient feedback from users regarding the real-time use cases of this feature.

[ffromani]

thanks @sohankunkerkar . The PRR deadline is Oct 3rd, but given your last updates I think we can keep the current status and stay in beta. If you think otherwise, please submit an update to make sure you opt in the PRR process.

[johnbelamaric]

Ok, sounds like we are staying in beta and no need for a PRR - please let me know if I misunderstood, I am taking this off my board.

[johnbelamaric]

I also marked it as beta on the enhancements board, instead of stable.

[sohankunkerkar]

Ok, sounds like we are staying in beta and no need for a PRR

That's right!

[hegerdes]

Don't want to clutter this issue but I had some small problems with the drop-in config dir. One of it got resolved but I don't know if the over has been fixed yet. It is described in kubernetes/kubernetes#124796

TL;DR:
Same config with --config flag and drop-in dir specified at the same time crashed the kubelet with a stacktrace

[shecodesmagic]

Since you have decided to stay in Beta, and with all the requirements fulfilled for Beta this enhancement is now marked as tracked for the upcoming enhancements freeze 🚀

[pacoxu]

As this KEP does not have a Feature gate, it used an env KUBELET_CONFIG_DROPIN_DIR_ALPHA to guard the feature.

• the alpha env was removed in v1.30: [KEP-4419]: promote KubeletConfigDropInDir feature to beta kubernetes#122907

enhancements/keps/sig-node/3983-drop-in-configuration/kep.yaml

Lines 26 to 27 in 25777c1

	alpha: "v1.28"
	beta: "v1.31"

updated the beta version to 1.31; but Jordan's PR kubernetes/kubernetes#127421 was merged in 1.32. Is this a typo?

[kannon92]

@pacoxu no, that is not a typo. Jordan merged a bug fix but this was promoted to beta in 1.31.

[pacoxu]

@pacoxu no, that is not a typo. Jordan merged a bug fix but this was promoted to beta in 1.31.

• env KUBELET_CONFIG_DROPIN_DIR_ALPHA was removed in v1.30 ([KEP-4419]: promote KubeletConfigDropInDir feature to beta kubernetes#122907)
• kep status of beta was updated to v1.31 (KEP update #4889)?? I mean this update(of the beta version) may be wrong.
• a bug fix was merged recently in v1.32 (Jordan's fix)

I mean that it should be already beta in v1.30 as the alpha ENV was removed in v1.30.

[SergeyKanzhelev]

yes, it stays in beta for one more release with the idea to pre-enable the drop-in directory and ensure that future version bump of config api will not break this

[mbianchidev]

Hey @haircommander,👋 from the v1.32 Communications Team!

We'd love for you to consider writing a feature blog about your enhancement.
Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and it is graduating.

To opt-in, let us know by opening a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we finalize the blog schedule.

[hacktivist123]

Hello @haircommander 👋 1.32 Docs Shadow here.

Does this enhancement work planned for 1.32 require any new docs or modifications to existing docs?
If so, please follow the steps here to open a PR against the dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, October 24th 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[hacktivist123]

Hello @haircommander 👋 1.32 Docs Shadow here.

This is just a reminder to open a placeholder PR against dev-1.32 branch in the k/website repo for this (steps available here) for this KEP if it requires new or modifications to existing docs:

The deadline for this is Thursday, Oct 24 at 18:00 PDT.
Thanks! 🚀

[haircommander]

we're retargeting beta here so I don't think there should be a docs update. is that correct @sohankunkerkar ?

[SergeyKanzhelev]

we're retargeting beta here so I don't think there should be a docs update. is that correct @sohankunkerkar ?

My understanding was that we will be doing this: "Set the default value of --config-dir to /etc/kubernetes/kubelet.conf.d. Users can disable this field by setting it to an empty string."

If so, some docs update will be needed!

[haircommander]

good point, done here kubernetes/website#48482

[liggitt]

My understanding was that we will be doing this: "Set the default value of --config-dir to /etc/kubernetes/kubelet.conf.d. Users can disable this field by setting it to an empty string."

Does that error if /etc/kubernetes/kubelet.conf.d does not exist? If so, won't that break ~all existing installations that haven't created that directory? That doesn't seem reasonable.

[haircommander]

I would expect the kubelet attempt to create the directory in that case

[liggitt]

I would expect the kubelet attempt to create the directory in that case

hmm... I wouldn't... I'd expect config locations on disk to be read-only from the kubelet's perspective

[ffromani]

The following may be silly, but still: golang's os.ReadDir return a clear specific error in case the directory being open doesn't exist. Maybe we can catch it with os.IsNotExist and carry along only on this path?
(disclosure: don't remember what the KEP prescribes in this case)

[SergeyKanzhelev]

@sohankunkerkar it will be great if you can summarize the behavior of misconfiguration/misconfiguration.

Thinking about it - maybe the typical installation will be that both config file and config dir will be specified as an arguments for the systemd unit so they will be (mostly) readonly and not expected to be changed. In this case, making the default to be a predefined folder doesn't make too much sense.

Also if default will be configured - make sure to update e2e_node framework to change the folder for every test run. So there is no leftovers from the previous test making the next test fail.

[brandond]

Maybe the typical installation will be that both config file and config dir will be specified as an arguments for the systemd unit so they will be (mostly) readonly and not expected to be changed

Are we talking defaults for kubeadm-based distros now? Or just the default behavior for the kubelet binary?

[SergeyKanzhelev]

@haircommander @sohankunkerkar what do you say we remove this defaulting? It feels that the common usage pattern will be to specify this folder in systemd unit config anyways. So having a default folder is more troubles and surprises for end users than a benefit.