From 34f93dd62f0255585cce95824f94f99459cd99f8 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Fri, 10 Nov 2023 16:45:49 +0200 Subject: [PATCH] kubeadm: add notes about network setup to the "create cluster" doc This is a top 3 question on support forums. "How to pass custom IP to kubeadm?" At one point there was a blog post on how to do this step-by-step, but we rejected that blog due to too many mistakes and not being a recommended practices, generally. In this PR, we add some detail on how to do it and tag the non-default-route-IP way as not-recommended with a warning. It is not considered as a foot gun for users, per se, but the setup is just awkward, prone to mistakes and not easy to maintain. If k8s supported something like a global /etc/kubernetes/DEFAULT_IP config of sorts, it would have seemed much more manageable. --- .../tools/kubeadm/create-cluster-kubeadm.md | 62 +++++++++++++++++-- 1 file changed, 57 insertions(+), 5 deletions(-) diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md index 5db35089784f0..613776e1e9c55 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md @@ -72,6 +72,8 @@ Any commands under `kubeadm alpha` are, by definition, supported on an alpha lev ### Preparing the hosts +#### Component installation + Install a {{< glossary_tooltip term_id="container-runtime" text="container runtime" >}} and kubeadm on all the hosts. For detailed instructions and other prerequisites, see [Installing kubeadm](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/). @@ -84,6 +86,61 @@ kubeadm to tell it what to do. This crashloop is expected and normal. After you initialize your control-plane, the kubelet runs normally. {{< /note >}} +#### Network setup + +kubeadm similarly to other Kubernetes components tries to find a usable IP on +the network interface associated with the default gateway on a host. Such +an IP is then used for the advertising and/or listening performed by a component. + +To find out what this IP is on a Linux host you can use: + +```shell +ip route show # Look for a line starting with "default via" +``` + +Kubernetes components do not accept custom network interface as an option, +therefore a custom IP address must be passed as a flag to all components instances +that need such a custom configuration. + +To configure the API server advertise address for control plane nodes created with both +`init` and `join`, the flag `--apiserver-advertise-address` can be used. +Preferably, this option can be set in the [kubeadm API](/docs/reference/config-api/kubeadm-config.v1beta3) +as `InitConfiguration.localAPIEndpoint` and `JoinConfiguration.controlPlane.localAPIEndpoint`. + +For kubelets on all nodes, the `--node-ip` option can be passed in +`.nodeRegistration.kubeletExtraArgs` inside a kubeadm configuration file +(`InitConfiguration` or `JoinConfiguration`). + +For dual-stack see +[Dual-stack support with kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support). + +{{< note >}} +IP addresses become part of certificates SAN fields. Changing these IP addresses would require +signing new certificates and restarting the affected components, so that the change in +certificate files is reflected. See +[Manual certificate renewal](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal) +for more details on this topic. +{{}} + +{{< warning >}} +The Kubernetes project recommends against this approach (configuring all component instances +with custom IP addresses). Instead, the Kubernetes maintainers recommend to setup the host network, +so that the default gateway IP is the one that Kubernetes components auto-detect and use. +On Linux nodes, you can use commands such as `ip route` to configure networking; your operating +system might also provide higher level network management tools. If your node's default gateway +is a public IP address, you should configure packet filtering or other security measures that +protect the nodes and your cluster. +{{< /warning >}} + +{{< note >}} +If the host does not have a default gateway, it is recommended to setup one. Otherwise, +without passing a custom IP address to a Kubernetes component, the component +will exit with an error. If two or more default gateways are present on the host, +a Kubernetes component will try to use the first one it encounters that has a suitable +global unicast IP address. While making this choice, the exact ordering of gateways +might vary between different operating systems and kernel versions. +{{< /note >}} + ### Preparing the required container images This step is optional and only applies in case you wish `kubeadm init` and `kubeadm join` @@ -117,11 +174,6 @@ a provider-specific value. See [Installing a Pod network add-on](#pod-network). known endpoints. To use different container runtime or if there are more than one installed on the provisioned node, specify the `--cri-socket` argument to `kubeadm`. See [Installing a runtime](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime). -1. (Optional) Unless otherwise specified, `kubeadm` uses the network interface associated -with the default gateway to set the advertise address for this particular control-plane node's API server. -To use a different network interface, specify the `--apiserver-advertise-address=` argument -to `kubeadm init`. To deploy an IPv6 Kubernetes cluster using IPv6 addressing, you -must specify an IPv6 address, for example `--apiserver-advertise-address=2001:db8::101` To initialize the control-plane node run: