:bug: `join` does not allow user-defined (aka custom) static pod manifests unless the user ignores a preflight check

[dlipovetsky]

What keywords did you search in kubeadm issues before filing this one?

DirAvailable

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): v1.22.0

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened?

I want to run a user-defined static pod on my machines, so I place it in /etc/kubernetes/manifests/foo.yaml on each machine.

[Update: To be clear, I mean a static pod manifest written to the file system by the user, not a static pod manifest written to the filesystem by kubeadm.]

On the first machine, I run kubeadm init, which succeeds. On other machines, I run kubeadm join, which fails a preflight check:

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty

What you expected to happen?

I expect kubeadm to allow user-defined (aka custom) static pod manifests to be present in the manifests directory.

It is reasonable for kubeadm to have a preflight check for the presence of specific files in the manifests directory. For example, kubeadm derives the contents of, and writes the static pods for kube-apiserver, kube-controller-manager, kube-scheduler, and, unless external etcd is used, for etcd as well. If these static pods are already present, it is reasonable for a preflight check to fail.

How to reproduce it (as minimally and precisely as possible)?

1. Create a cluster (in order to join a machine to it)
2. On the machine, place a file in /etc/kubernetes/manifests
3. On the machine, run kubeadm join

Anything else we need to know?

This affects the Cluster API Kubeadm Bootstrap Provider (CABPK), because it writes files to the machine (including /etc/kubernetes/manifests) before running kubeadm.

If I want to place a user-defined static pod manifest with CABPK, I always have to ignore this preflight check.

CABPK itself ignores the preflight check in one specific case.

---

I started a brief discussion prior to filing the issue: https://kubernetes.slack.com/archives/C8TSNPY4T/p1631126245196000

/cc @fabriziopandini @neolit123

[neolit123]

What you expected to happen?
...

makes sense to me and this is what we discussed on slack.
i have added the help wanted label, so if anyone wishes to take this as a fix for 1.23, please go ahead.

as previously noted, we cannot backport the change to <1.23.

[SataQiu]

/assign

[fabriziopandini]

I have a couple of questions here before green lightning the change.

• Our preferred ways to support static pod mutations is via config files or via patches for the advanced use cases, and this relies on the idea that kubeadm does the heavy lifting, ensures everything is consisntent, and the user applies small changes. Allowing the user to provide static pod manifests out of band kind of defeats this idea. What is the underlying use case we are trying to solve here with custom etcd manifests?
• I don't remember the history of preflight checks, and unfortunately I did not find comment in the code, but I assume that the idea was to prevent the users to run init or join twice. Are we sure that by removing this check we are not exposing the user to those kind of problems?

[randomvariable]

Our preferred ways to support static pod mutations is via config files or via patches for the advanced use cases, and this relies on the idea that kubeadm does the heavy lifting.

I don't think that's the use case here. It's more to allow placing of additional static manifests for things where a daemonset isn't useful. So it'll be fine to error if we see static pod manifests that match the ones kubeadm is about to lay down, but ignore other ones.

Additional static pods are mostly used on control plane machines, for example adding an etcd encryption provider for the API server, which was the original use case kubernetes-sigs/cluster-api#3053 was addressing.

[dlipovetsky]

Thanks for the input, @fabriziopandini and @randomvariable .

As @randomvariable says, the use case is additional static pod manifests. I will amend the description to say this explicitly.

So it'll be fine to error if we see static pod manifests that match the ones kubeadm is about to lay down, but ignore other ones.

+1

[dlipovetsky]

I don't remember the history of preflight checks, and unfortunately I did not find comment in the code, but I assume that the idea was to prevent the users to run init or join twice. Are we sure that by removing this check we are not exposing the user to those kind of problems?

I agree that user safety is a priority. If we limit the check to the static pod manifests written by kubeadm, we continue to keep the user safe from running init/join twice, but we allow the user to manage their own static pod manifests.

[dlipovetsky]

Whatever we choose to do with this preflight check, let's please apply it consistently to both init and join. Today it only applies to join.