Have kubelet use systemd cgroup driver on al2 and ubuntu

[reegnz]

Description

Kubernetes documentation indicates that for stability reasons
one should run kubernetes with the systemd cgroup driver if the
init system itself is systemd.

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-drivers

The same change is also be made in awslabs/amazon-eks-ami:

awslabs/amazon-eks-ami#521

Fixes #2960

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the userdocs directory)
• Manually tested
• Added labels for change area (e.g. area/nodegroup), target version (e.g. version/0.12.0) and kind (e.g. kind/improvement)
• Make sure the title of the PR is a good description that can go into the release notes

[reegnz]

Verified manally for Amazon Linux 2:

• eksctl create cluster with the new built binary
• entered the node with kubectl node-shell (https://github.com/kvaps/kubectl-node-shell)
• checking /etc/docker/daemon.json -> ok, contains proper systemd cgroup config
• checking /etc/systemd/system/kubelet.service.d/10-eksclt.al2.conf -> uses /etc/eksctl/kubelet.yaml
• checking /etc/eksctl/kubelet.yaml -> ok, contains proper systemd cgroup config
• systemctl -t slice -> shows kubepods.slice, kubepods-burstable.slice and kubepods-besteffort.slice exist
• systemd-cgls -> shows kubepods.slice below system.slice in cgroup hierarchy
• systemctl status containerd.service -> you can see runc processes spawned by it are using the -systemd-cgroup flag
• systemd-cgtop -> output contains slices belonging to kubepods.slice

Verified manually for Ubuntu 2004:

• eksctl create cluster --node-ami-family 'Ubuntu2004' with the new built binary
• entered the node with kubectl node-shell (https://github.com/kvaps/kubectl-node-shell)
• checking /etc/docker/daemon.json -> ok, contains proper systemd cgroup config
• snap get kubelet-eks config -> ok, returns /etc/eksctl/kubelet.yaml
• checking /etc/eksctl/kubelet.yaml -> ok, contains proper systemd cgroup config
• systemctl -t slice -> shows kubepods.slice, kubepods-burstable.slice and kubepods-besteffort.slice exist
• systemd-cgls -> shows kubepods.slice below system.slice in cgroup hierarchy
• systemctl status containerd.service -> you can see runc processes spawned are using the -systemd-cgroup flag
• systemd-cgtop -> output contains slices belonging to kubepods.slice

[Callisto13]

Thanks @reegnz the team will get to this asap 👍

[Callisto13]

LGTM, thanks for catching this 👍

[romeotheriault]

This change causes problems for us using ubuntu 16 (also doesn't work well on ubuntu 18) as our node ami with the included versions of systemd. It seems there are some bugs in systemd's cgroup driver in ubuntu 16 and 18 that cause docker to hang on creating containers, etc... and in turn it causes kubelet to hang and the nodes to become "NotReady". Downgrading eksctl to eksctl version 0.34.0 (the last version before this change) fixes the issue.

It would be great if eksctl could provide a flag to allow the users to change (or choose which of) these types of options to use instead of eksctl changing a default underneath it's users. Especially for older OS releases.

[Callisto13]

@romeotheriault could you open an issue for this please?