Install the kube-proxy main container without privileged mode

[uablrek]

A new flag --init-only was added to kube-proxy that executes all configuration steps that required privileged mode and then exits. It is intended to be used in an initContainer to allow the main container to run with NET_ADMIN rights only.

Please read the (not yet published) blog post for more info (temporary direct-link).

Kubeadm should support this installation. It will increase the startup time for kube-proxy, so IMO it should be optional, but perhaps be the default.

/kind feature

Versions

kubeadm version (use kubeadm version): $\ge$ v1.29

Environment:

• Kubernetes version (use kubectl version): $\ge$ v1.29
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Container runtime (CRI) (e.g. containerd, cri-o): N/A
• Container networking plugin (CNI) (e.g. Calico, Cilium): N/A
• Others:

Anything else we need to know?

Regression tests are OK (of course), but I suppose the real e2e test will be with this feature enabled, which mean with an updated kubeadm if I understand correctly. I have however executed e2e tests in KinD with:

FOCUS='\[sig-network\].*ervice'
SKIP='Disruptive|Serial|ESIPP|DNS|GCE|finalizer'

which actually tests more of kube-proxy than the conformance suite.

Upgrade is not explicitly tested, but as a part of testing in KinD I have updated the kube-proxy manifest.

[uablrek]

PR kubernetes/kubernetes#120864

[uablrek]

/cc @danwinship @aojea @sftim

[neolit123]

thanks for logging this issue and sending that PR.
a de-privilidge solution for kube-proxy was required for a long time.

A new flag --init-only was added to kube-proxy that executes all configuration steps that required privileged mode and then exits. It is intended to be used in an initContainer to allow the main container to run with NET_ADMIN rights only.

i see a Microsoft reviewer on the PR, so i'm going to assume it also works on Windows.
is that correct?

Kubeadm should support this installation. It will increase the startup time for kube-proxy, so IMO it should be optional, but perhaps be the default.

do you have estimates for that? - e.g. 20% increase on a desktop PC. and what window are we measuring - e.g. kube-proxy DS
creation, up until main container is running on a single node?

some consumers of kubeadm are quite sensitive to that, inc minikube, kind, others. my preference would be to make this the default, non-configurable path.

in the incoming releases, we are about to release a new API v1beta4, and we are just about to add a new sub-struct for "proxy" configuration there. so now it would be a good time for a knob for this kube-proxy "startup-mode".
if we do add this with a knob though, that would mean the change will be available when v1beta4 is out.

if we use something common like a feature-gate we can add it in 1.29.

...some decision making is required here.

[neolit123]

Regression tests are OK (of course), but I suppose the real e2e test will be with this feature enabled, which mean with an updated kubeadm if I understand correctly.

we have a variety of tests jobs that can exercise this functionality (inc upgrade). if we add a knob it would be slightly more complex, since we have to add new targeted jobs.

FOCUS='[sig-network].*ervice'

we only run conformance tests in kubeadm e2e jobs.
what specific tests outside of the conf suite would be required to ensure this feature works properly?

to my understanding it's all about the startup sequence and setup. if it fails...it fails. from there on the conformance suite should suffice when testing the runtime aspect?

[uablrek]

i see a Microsoft reviewer on the PR, so i'm going to assume it also works on Windows.
is that correct?

No, the function will not work on Windows. The --init-only should do nothing, and exit with an error code, and the kube-proxy must still be privileged, please see kubernetes/kubernetes#120864 (comment)

[uablrek]

the conformance suite should suffice when testing the runtime aspect?

I don't expect more than the conformance tests to test this feature. But there are some e2e tests that tests aspects of "services" (i.e. kube-proxy) that are not in the conformance suite. I have explicitly executed those I know of.

[neolit123]

i see a Microsoft reviewer on the PR, so i'm going to assume it also works on Windows.
is that correct?

No, the function will not work on Windows. The --init-only should do nothing, and exit with an error code, and the kube-proxy must still be privileged, please see kubernetes/kubernetes#120864 (comment)

that's a problem, because the kube-proxy manifest has no OS branching in kubeadm, currently.
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/addons/proxy/manifests.go#L54
why wasn't the flag made no-op instead of throwing an error on Windows?

if we add this change there has to be a _windows.go specific gobuild tag based file that injects the init container part for !Windows.

[neolit123]

the conformance suite should suffice when testing the runtime aspect?

I don't expect more than the conformance tests to test this feature. But there are some e2e tests that tests aspects of "services" (i.e. kube-proxy) that are not in the conformance suite. I have explicitly executed those I know of.

ok, i think we can go without extending what we test in our jobs for this feature.

[neolit123]

i see a Microsoft reviewer on the PR, so i'm going to assume it also works on Windows.
is that correct?

No, the function will not work on Windows. The --init-only should do nothing, and exit with an error code, and the kube-proxy must still be privileged, please see kubernetes/kubernetes#120864 (comment)

that's a problem, because the kube-proxy manifest has no OS branching in kubeadm, currently. https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/addons/proxy/manifests.go#L54 why wasn't the flag made no-op instead of throwing an error on Windows?

if we add this change there has to be a _windows.go specific gobuild tag based file that injects the init container part for !Windows.

commented on the PR about using a warning instead of error:
kubernetes/kubernetes#120864 (comment)

[uablrek]

do you have estimates for that? - e.g. 20% increase on a desktop PC. and what window are we measuring - e.g. kube-proxy DS
creation, up until main container is running on a single node?

I actually expect a very small increase. I believe that the bulk of the startup time is to load the image, and it is only done once. Then there is an overhead to create/start the init-container of course. The actual init code should run fast, but there is an unnecessary connection setup to the API-server that may be removed in the future.

It is hard to measure since it is highly environment dependent. But I can try in KinD on my PC with local image registry (which is kind of unfair).