Multi-node Kubernetes clusters with Lima VM

[louhisuo]

If someone wants to run multi-node Kubernetes clusters on Lima VM, I have created a Github repo which gives guidance how to build a setup. Please check Kubernetes on macOS.

Big thanks for Lima VM team making this possible :)

[afbjorklund]

Continued from:

• Adding worker node to K8s cluster created using Lima #1181

---

We talked briefly about having such an example, but for a simpler setup (such as a two-node cluster using flannel)

• Multi-node kubernetes clusters #1184

Previously it was mostly waiting for the (multi-host) networking to be available, but I think user-v2 should be ok now.

So it should be possible to allow the k3s.yaml and k8s.yaml to be used for multi-node, with some tweaks to the yaml.

[louhisuo]

With this discussion I just merely wanted to share how I ended up building multi-node Kubernetes clusters using Lima. Discussions in #1184 really helped me forward.

Few comments regarding design choices.

Two node clusters (one CP and one Worker node) are probably sufficient for the most of people. I have need for more complex setups, therefore I was exploring one CP and 3x Worker nodes, as well as HA control plane setups.

Flannel is good lightweight CNI and unless network policies or more advanced features are required it is good choice. My driver to use Cilium was eBPF, Network Policies, as well as integrated L2 LB, Ingress Controller (L7 LB) and Gateway API (L4/L7 LB) support. Probably the most of people do not need these in development environments.

user-v2 vs shared networks. I explored both and ended up for shared network because then I am able to have L2 LB IP pool configured in upper range of 192.168.105.0/24 network and access type LoadBalancer services from a macOS host.

What I found perhaps the most challenging (from security perspective is how to distribute certificateKey and bootstrapTokens across nodes during cluster creation. I ended up to configure them insecurely into kubeadm config files. For me this was acceptable as my setups are short lived sandboxes.

[tewarivarun]

Thank you, I was about to try this out, will use your setup as a starting point

[louhisuo]

Tiny update to my related to my Kubernetes on macOS (Apple silicon) project. With the latest Lima-VM 0.23.2 and socket_vmnet 1.1.5 updates I feel that multi-node kubernetes cluster deployments have become very stable. Networking issues which caused a cluster become unstable have now been resolved and it is breeze to work with Kubernetes on Lima-VM. Thanks a lot.