Upgrade VMC testbed to a more recent version of K8s

[antoninbas]

The Jenkins jobs (e2e, conformance, networkpolicy) are run on testbeds with K8s v1.20.1. This version is getting a bit old, and some notable networking changes happened since that version, e.g. the EndpointSlice feature went GA. At the moment, it is is not possible to run recent upstream K8s e2e tests, as a lot of tests will fail when using a conformance image version >= v1.21:

E0406 22:19:57.419864      25 reflector.go:138] k8s.io/kubernetes/test/e2e/framework/service/jig.go:441: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: the server could not find the requested resource

The tests expect the v1.EndpointSlice resource to exist, which is not the case in a v1.20 cluster.

We should look into updating testbeds to a more recent K8s version (maybe v1.22 or v1.23), so we can use a more recent conformance image version.

I don't see a more recent OVA available on Github: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere#kubernetes-versions-with-published-ovas, but maybe we can build our own or maybe there are OVAs available for VMC.

[antoninbas]

@XinShuYang do you think this is something that could be done?

[XinShuYang]

@antoninbas Yes, the latest OVA version is 1.20. As I know @edwardbadboy is working on supporting 1.24 testbed on AWS. I am also working on private lab testbed upgrade.

[XinShuYang]

I noticed containerd is not the default runtime in vagrant cluster. Do we have the plan to switch to containerd as the default runtime for kubelet? @antoninbas

[antoninbas]

@XinShuYang containerd is the default runtime in the Vagrant-based testbed, what makes you say it is not?

#2583

[XinShuYang]

@antoninbas I was wrong to say Vagrant testbed not use containerd. Actually I reused vagrant playbook to setup environment in private CI testbed. The problem is I also installed docker by another script at the same time. In this case kubelet will use docker as default runtime because Vagrant playbook doesn't add "--container-runtime-endpoint" parameter for kubelet. Do you think it's a wrong idea to install docker and containerd on the same node? I am afraid we still need docker to build images. Or maybe we can create one more "image" node which only installs docker for image building and delivery?

[antoninbas]

I am afraid we still need docker to build images.

Do you need to build images on the K8s Nodes?

If you want to keep Docker on the Nodes, I suggest that we start setting nodeRegistration.criSocker in the Kubeadm configurations (for init and for join). You can do this change in the vagrant playbook if you want.

I imagine that starting with K8s v1.24 (with the removal of Dockershim), K8s will stop defaulting to Docker?

[XinShuYang]

I imagine that starting with K8s v1.24 (with the removal of Dockershim), K8s will stop defaulting to Docker?

Yes, they announced Dockershim will no longer be included in Kubernetes.
I wonder if we have tried any CI test in Vagrant cluster? I tried to run e2e test on my containerd testbed and found some errors. If tests can pass on Vagrant cluster it may caused by my environment issue, otherwise we need to update CI test code. @antoninbas

[antoninbas]

I wonder if we have tried any CI test in Vagrant cluster

I run individual e2e tests from time to time when I am troubleshooting something. I don't think I have run the full test suite in quite some time.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days