kubeadm: add notes about network setup to the "create cluster" doc #43872

neolit123 · 2023-11-10T14:48:25Z

This is a top 3 question on support forums.
"How to pass custom IP to kubeadm?"

At one point there was a blog post on how to do this step-by-step, but we rejected that blog due to too many mistakes and not being a recommended practices, generally.

In this PR, we add some detail on how to do it and tag the non-default-route-IP way as not-recommended with a warning.

It is not considered as a foot gun for users, per se, but the setup is just awkward, prone to mistakes and not easy to maintain. If k8s supported something like a global /etc/kubernetes/DEFAULT_IP config of sorts, it would have seemed much more manageable.

fixes #43862
^ NOTE: marked as "fixes", because "create a cluster with kubeadm" is the main landing page for kubeadm users. from there they can choose where to go to - "init", "join", "HA", but they must read how to setup a host from the "create a cluster" page. otherwise they can hit a number of issues like - swap, OOM, etc.

NOTE: this is correctly targeting the "main" branch as documenting existing functionality.

neolit123 · 2023-11-10T14:48:48Z

/sig cluster-lifecycle
/cc @pacoxu @SataQiu
/assign @sftim

netlify · 2023-11-10T14:50:54Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`34f93dd`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/6555faadd111330008fb0e87
😎 Deploy Preview	https://deploy-preview-43872--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

sftim

Overall I like this.

I do suggest some tweaks.

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

sftim · 2023-11-10T17:07:19Z

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

+Kubernetes components will try to use the one that has a global unicast IP address
+(non-loopback, non-link local, non-point2point).


We might want to document what happens when there are two of these (less common, but actually again this could apply to my laptop: I have wired ethernet and wireless and both interfaces have global unicast IPv6 addresses).

the logic is quite complicated and a bit confusing - i.e. there are two methods.
IIRC, the second method surfaced when IPv6 support was added.

1

if /proc/net/route does not exist it will enumerate interfaces using Go lang's net.Interfaces() which is OS portable. on Linux the enumeration is done with a set of POSIX syscalls, order should be the same as ip route. but i don't want to document such a thing as implementation can vary between kernels, CLI tools, etc. there is no guarantee of order determinism.

it would then do this:

// chooseIPFromHostInterfaces looks at all system interfaces, trying to find one that is up that // has a global unicast address (non-loopback, non-link local, non-point2point), and returns the IP. // addressFamilies determines whether it prefers IPv4 or IPv6

https://github.com/kubernetes/apimachinery/blob/12dc3f82eb47fc972b5be0b222f4422cb1b51612/pkg/util/net/interface.go#L309C1-L312

here we can say that the first IP on an interface that satisfies the above criteria is used.

2

if /proc/net/route exists both /proc/net/route and /proc/net/route_ipv6 are parsed with only default routes.
if the preferred IP family is v6, the v6 routes take precedence.

this is more deterministic. routes are appended in that same order.

then the IP is determined slightly differently, as there is a fallback to global IP on loopback:

// chooseHostInterfaceFromRoute cycles through each default route provided, looking for a // global IP address from the interface for the route. If there are routes but no global // address is obtained from the interfaces, it checks if the loopback interface has a global address. // addressFamilies determines whether it prefers IPv4 or IPv6

https://github.com/kubernetes/apimachinery/blob/12dc3f82eb47fc972b5be0b222f4422cb1b51612/pkg/util/net/interface.go#L426-L430

@aojea can surely provide more context on this, but IMO what we can document is the following, trying to find a balance between somewhat descriptive and vague (to not enter rabbit holes):

If two or more default gateways are present on the host, a Kubernetes component will try to use the first one it encounters that has a suitable global unicast IP address. While making this choice, the exact ordering of gateways might vary between different operating systems and kernel versions.

SataQiu

/lgtm

k8s-ci-robot · 2023-11-13T02:49:25Z

LGTM label has been added.

Git tree hash: 4a7e4099c44ec364bc67c03552f6d7f47dffc33d

SataQiu · 2023-11-13T03:16:57Z

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

+without passing a custom IP address to a Kubernetes component, the component
+will exit with an error. If two or more default gateways are present on the host,


the component will exit with an error

Sometimes, this may be not true.
For kubelet, the detection order is:

// 1) Use nodeIP if set (and not "0.0.0.0"/"::") // 2) If the user has specified an IP to HostnameOverride, use it // 3) Lookup the IP from node name by DNS // 4) Try to get the IP from the network interface used as default gateway // // For steps 3 and 4, IPv4 addresses are preferred to IPv6 addresses // unless nodeIP is "::", in which case it is reversed.

Although there is no default route, it does not exit with -1 when DNS resolution is available.
But overall, we recommend setting a default route for hosts.

i'm a bit confused reading the kubelet code. in NodeAddress there is this comment that you mentioned and i remember it from before, but is it actually up to date?

i see a fallback to ResolveBindAddress from the apimachinery interface/IP detection code, which is what kube-proxy, apiserver, kubeadm, etc are doing:
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodestatus/setters.go#L197-L225

which ends up as:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/net/interface.go#L426-L430
(as mentioned in the above comments)
#43872 (comment)

Although there is no default route, it does not exit with -1 when DNS resolution is available.
But overall, we recommend setting a default route for hosts.

i can see this making sense in a cloud provider env where the cloud is responsible for assigning the kubelet IPs.
looks like there is this:
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodestatus/setters.go#L125-L142

but isn't this the error that is returned if no IP can be detected?
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodestatus/setters.go#L227-L230

If no default route on the node, IIRC, the kubeadm command will exit if not set in the configuration. (Not tested.)

neolit123 · 2023-11-16T09:25:08Z

looking for more reviews / LGTM / approve on this one.

pacoxu · 2023-11-16T10:33:34Z

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

+
+For kubelets on all nodes, the `--node-ip` option can be passed in
+`.nodeRegistration.kubeletExtraArgs` inside a kubeadm configuration file
+(`InitConfiguration` or `JoinConfiguration`).


A dual-stack example is provided in https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/dual-stack-support/. Could we add the dual stack example link here?

added link to the page under the kubelet --node-ip example.

This is a top 3 question on support forums. "How to pass custom IP to kubeadm?" At one point there was a blog post on how to do this step-by-step, but we rejected that blog due to too many mistakes and not being a recommended practices, generally. In this PR, we add some detail on how to do it and tag the non-default-route-IP way as not-recommended with a warning. It is not considered as a foot gun for users, per se, but the setup is just awkward, prone to mistakes and not easy to maintain. If k8s supported something like a global /etc/kubernetes/DEFAULT_IP config of sorts, it would have seemed much more manageable.

pacoxu

/lgtm

k8s-ci-robot · 2023-11-16T11:20:17Z

LGTM label has been added.

Git tree hash: 3eea437188dd0054eb6c309e22855780c8d85650

neolit123 · 2023-11-22T16:21:22Z

@sftim this has received 2 LGTM's from SIG CL tech reviewers.

sftim

This change leaves the website better than without making it.

/approve

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

sftim · 2023-11-22T16:43:09Z

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

+[Dual-stack support with kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support).
+
+{{< note >}}
+IP addresses become part of certificates SAN fields. Changing these IP addresses would require


Suggested change

IP addresses become part of certificates SAN fields. Changing these IP addresses would require

The IP addresses that you assign to control plane components become part of their X.509 certificates'

subject alternative name fields. Changing these IP addresses would require

sftim · 2023-11-22T16:43:23Z

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

+IP addresses become part of certificates SAN fields. Changing these IP addresses would require
+signing new certificates and restarting the affected components, so that the change in
+certificate files is reflected. See
+[Manual certificate renewal](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal)


OK, but this is actually replacement rather than renewal.

since it's not possible to change a certificate in-place, the process is always re-place. although, it's documented as renewal.

A renewal typically means that the new CSR and old CSR only differ by their validity period, and the motivation for renewal is expiry (ideally “imminent” rather than “recent”).
Different issuers / trust anchors have their own specific policies around whether it's a renewal or not.

I mention this because readers might think that renewal isn't the process they want.

sftim · 2023-11-22T16:44:07Z

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md

+To find out what this IP is on a Linux host you can use:
+
+```shell
+ip route show # Look for a line starting with "default via"


What if there's more than one default route? Readers might not be sure what to do.

k8s-ci-robot · 2023-11-22T16:44:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pacoxu, sftim

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [sftim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

neolit123 · 2023-11-22T17:07:06Z

@sftim looks like you left actionable comments, but this merged.
i will send a follow up where you can LGTM / approve.

sftim · 2023-11-22T18:08:47Z

I was happy to see this change land as-is; further, iterative improvement is welcome.

neolit123 · 2023-11-22T18:09:39Z

I was happy to see this change land as-is; further, iterative improvement is welcome.

follow up is here:
#44038

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Nov 10, 2023

k8s-ci-robot assigned sftim Nov 10, 2023

k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Nov 10, 2023

k8s-ci-robot requested review from pacoxu, SataQiu, natalisucks and reylejano November 10, 2023 14:48

sftim reviewed Nov 10, 2023

View reviewed changes

neolit123 force-pushed the 1.29-add-notes-about-network-setup-to-create-doc branch 2 times, most recently from 5d99808 to 1e2aeca Compare November 10, 2023 16:29

sftim reviewed Nov 10, 2023

View reviewed changes

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md Outdated Show resolved Hide resolved

content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md Outdated Show resolved Hide resolved

neolit123 force-pushed the 1.29-add-notes-about-network-setup-to-create-doc branch from 1e2aeca to f43da4e Compare November 10, 2023 16:48

sftim reviewed Nov 10, 2023

View reviewed changes

neolit123 force-pushed the 1.29-add-notes-about-network-setup-to-create-doc branch from f43da4e to 0f84cd0 Compare November 10, 2023 18:21

neolit123 mentioned this pull request Nov 11, 2023

kubeadm: support updating certificate organization during 'kubeadm certs renew' kubernetes/kubernetes#121841

Merged

SataQiu reviewed Nov 13, 2023

View reviewed changes

k8s-ci-robot assigned SataQiu Nov 13, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 13, 2023

SataQiu reviewed Nov 13, 2023

View reviewed changes

neolit123 mentioned this pull request Nov 16, 2023

kubeadm unable to select an IP from default routes. kubernetes/kubeadm#2604

Closed

pacoxu reviewed Nov 16, 2023

View reviewed changes

neolit123 force-pushed the 1.29-add-notes-about-network-setup-to-create-doc branch from 0f84cd0 to 34f93dd Compare November 16, 2023 11:19

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2023

k8s-ci-robot requested review from SataQiu and sftim November 16, 2023 11:19

pacoxu approved these changes Nov 16, 2023

View reviewed changes

k8s-ci-robot assigned pacoxu Nov 16, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2023

sftim reviewed Nov 22, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 22, 2023

k8s-ci-robot merged commit 91dbcc5 into kubernetes:main Nov 22, 2023
4 checks passed

neolit123 mentioned this pull request Nov 22, 2023

kubeadm: revisit the "Network setup" section post-feedback #44038

Merged

windsonsea mentioned this pull request Nov 23, 2023

[zh] Add translation to create-cluster-kubeadm.md #44044

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: add notes about network setup to the "create cluster" doc #43872

kubeadm: add notes about network setup to the "create cluster" doc #43872

neolit123 commented Nov 10, 2023 •

edited

Loading

neolit123 commented Nov 10, 2023

netlify bot commented Nov 10, 2023 •

edited

Loading

sftim left a comment

sftim Nov 10, 2023

neolit123 Nov 10, 2023 •

edited

Loading

SataQiu left a comment

k8s-ci-robot commented Nov 13, 2023

SataQiu Nov 13, 2023

neolit123 Nov 13, 2023 •

edited

Loading

pacoxu Nov 16, 2023

neolit123 commented Nov 16, 2023

pacoxu Nov 16, 2023

neolit123 Nov 16, 2023

pacoxu left a comment

k8s-ci-robot commented Nov 16, 2023

neolit123 commented Nov 22, 2023

sftim left a comment

sftim Nov 22, 2023

sftim Nov 22, 2023

neolit123 Nov 22, 2023

sftim Nov 22, 2023

sftim Nov 22, 2023

k8s-ci-robot commented Nov 22, 2023

neolit123 commented Nov 22, 2023

sftim commented Nov 22, 2023

neolit123 commented Nov 22, 2023

		Kubernetes components will try to use the one that has a global unicast IP address
		(non-loopback, non-link local, non-point2point).

		without passing a custom IP address to a Kubernetes component, the component
		will exit with an error. If two or more default gateways are present on the host,

	IP addresses become part of certificates SAN fields. Changing these IP addresses would require
	The IP addresses that you assign to control plane components become part of their X.509 certificates'
	subject alternative name fields. Changing these IP addresses would require

kubeadm: add notes about network setup to the "create cluster" doc #43872

kubeadm: add notes about network setup to the "create cluster" doc #43872

Conversation

neolit123 commented Nov 10, 2023 • edited Loading

neolit123 commented Nov 10, 2023

netlify bot commented Nov 10, 2023 • edited Loading

✅ Pull request preview available for checking

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 Nov 10, 2023 • edited Loading

Choose a reason for hiding this comment

1

2

SataQiu left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 13, 2023

Choose a reason for hiding this comment

neolit123 Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 commented Nov 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pacoxu left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 16, 2023

neolit123 commented Nov 22, 2023

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 22, 2023

neolit123 commented Nov 22, 2023

sftim commented Nov 22, 2023

neolit123 commented Nov 22, 2023

neolit123 commented Nov 10, 2023 •

edited

Loading

netlify bot commented Nov 10, 2023 •

edited

Loading

neolit123 Nov 10, 2023 •

edited

Loading

neolit123 Nov 13, 2023 •

edited

Loading