Kubernetes on ARM

Implement

• Run on AWS
• No Shell Scripts
• Use ARM Nodes to save money
• Use ASG/spot instances for Control Plane, including etcd, persist data across restart
• Route 53 with DNSSEC signed Zones
• Use IMDSv2 only
  • flatcar/Flatcar#220 (comment) - Waiting for Afterbrun update in Flatcar
• Use SSM Fleet Manager
  • TODO: Configure cloud-watch Agent for system logs
• IPv6 only network, IPv6 Dual Stack
  • https://blog.devopstom.com/ipv6-only-ec2/ - Github, DockerHub, SSM has no IPv6 endpoints
  • Amazon EC2 Loadbalancer requires IPv4, no native IPv6 supported
  • Flatcar Provision Tools has no wide IPv6 support
    • coreos/ignition#1340
    • coreos/afterburn#726
• Flatcar Container Linux
• aws-encryption-provider
  • See: https://aws.amazon.com/de/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/

TODO Cluster

• rootless control plane
• ✅ CoreDNS
• ✅ Run CIS Benchmark
• IAM Roles for Service Accounts (IRSA)
  • ✅ Using Service Account Token Volume Projection
  • ✅ amazon-eks-pod-identity-webhook
• Kubernetes CNI
  • kube-proxy in IPVS mode
  • ✅ amazon-vpc-cni-k8s for Networking
  • calico for NetworkPolicy
• Cloud Provider
  • ✅ cloud-provider-aws
  • ✅ kubelet-csr-approver to automatically approve kubelet server crts
• Storage
  • ✅ aws-ebs-csi-driver
• Node Scaling
  • ✅ karpenter
  • ✅ node-termination-handler
• Ingress Controller
  • ✅ aws-load-balancer-controller
  • ✅ external-dns
• Authentication
  • OIDC/Login with Github
  • aws-iam-authenticator
• Backup
  • Velero
• Cluster Policies
  • https://kyverno.io/
  • https://kubevious.io/
• Node local DNS cache ✅ Node Problem Detector
• gVisor
• https://github.com/flatcar-linux/flatcar-linux-update-operator/

TODO Application Side

• ArgoCD
• Reloader
• SealedSecrets ✅ cert-manager

TODO Future

• https://goteleport.com/pricing/
• https://litmuschaos.io/
• https://falco.org/
• crossplane
• AWS Controllers for Kubernetes (ACK)

Issues & Comments created for this homelab

• etcd-io/etcd#13847
• coreos/ignition#1340
  • coreos/ignition#1341
• coreos/afterburn#726
• kubernetes/cloud-provider-aws#327
  • kubernetes/cloud-provider-aws#328
  • kubernetes/cloud-provider-aws#333
• kubernetes/cloud-provider-aws#335
• hashicorp/terraform-provider-aws#24009
• hashicorp/terraform-provider-tls#181
• aws/eks-charts#721 / aws/amazon-vpc-cni-k8s#1949
• flatcar/Flatcar#707
• flatcar-archive/coreos-overlay#1800
• aws/karpenter-provider-aws#1652
  • aws/karpenter-provider-aws#1653
• aws/amazon-ssm-agent#442
  • aws/amazon-ssm-agent#439
• kubernetes-sigs/aws-ebs-csi-driver#1204