Simplify support for configurable Kubernetes service IP address range

[mmerkes]

Overview

What would you like to be added:

On 10/05/2020, EKS released support for configurable Kubernetes service IP address range. To support this feature, kubelet needs to be configured with a custom cluster DNS IP.

To make that happen, customers need to set the --dns-cluster-ip argument when running bootstrap.sh in their userdata or avoid setting --b64-cluster-ca and --apiserver-endpoint, which will result in calling the EKS DescribeCluster API to get the service IPv4 CIDR dynamically. Otherwise, kubelet will be configured with default values.

NOTE: This is not an issue for customers using EKS Managed node groups because --b64-cluster-ca, --apiserver-endpoint and --dns-cluster-ip are all set when calling the bootstrap script in the userdata of the LaunchTemplate.

Why is this needed:

If customers are not aware of the details for the bootstrap script, the cluster DNS IP will be incorrectly set for all of their nodes, and they will need to spend time to debug the issue. While there are workarounds for this issue, customers need to set a correct combination of arguments in the bootstrap script to do the right thing. Ideally, this works out of the box for customers, regardless of their cluster configuration.

Implementation Options

Call EKS DescribeCluster API if DNS_CLUSTER_IP is unset

We could check if DNS_CLUSTER_IP is unset when calling the bootstrap script, along with B64_CLUSTER_CA and APISERVER_ENDPOINT, and call the EKS DescribeCluster API to make sure that we have the correct configuration.

However, this is exactly what was done previously, but it had to be reverted because customers with private VPCs could not access the EKS APIs without networking workarounds. This would not be a viable option unless VPC endpoint support for EKS API became an EKS feature.

On top of that, customers set the --b64-cluster-ca and --apiserver-endpoint to avoid calling the EKS DescribeCluster API for various reasons, a big one being to avoid throttling on EKS APIs. If customers have a large number of nodes spinning up at one time, it can cause throttling and nodes to fail to spin up. Those customers will need to know that they will need to start setting --dns-cluster-ip, if they don't already. Or, another workaround needs to be provided.

Add boolean argument to bootstrap script to always call EKS DescribeCluster

Customers could set something like always-describe-cluster to simplify the logic of the branching to get the kubelet configuration from EKS APIs. This would be a simple check here to always get the latest configuration.

Do nothing

This is not the ideal option for customers using configurable Kubernetes service IP address ranges, but it would be the least impactful to other customers. It would leave customers with the existing options today:

1. Do not set --dns-cluster-ip, --b64-cluster-ca or --apiserver-endpoint
2. Do set --dns-cluster-ip, --b64-cluster-ca and --apiserver-endpoint
3. Use EKS managed node groups

[r3kzi]

Thanks for opening a feature request @mmerkes !

I was not aware of that previous change.

We went with option 2 and put --dns-cluster-ip beside --b64-cluster-ca and --apiserver-endpoint as an extra bootstrap arg.

Since we are using an Terraform module to create an EKS cluster and associated worker instances we weren't able to use Option 1. Both --b64-cluster-ca and --apiserver-endpoint are set by the module, see here.

I put a review of EKS managed node groups on our backlog.

[cartermckinnon]

Closing re: #636