Enable CI tests in K8s 1.25

[torredil]

What is this PR about? / Why do we need it?

• Upgrade k8s.io/kubernetes to 1.25.
• Upgrade github.com/onsi/ginkgo to v2.
• Use kubetest2 for e2e-kubernetes and use the upstream binary for this test.
• Split K8S_VERSION in tests between kOps and eksctl so that we can test them independently.
• Removed migration tests due to in-tree driver removal.
• Remove CSIDriverRegistry, CSINodeInfo, and CSIBlockVolume from hack/kops-patch.yaml as the feature gates have been GA since K8s v1.18.

Signed-off-by: Eddie Torres torredil@amazon.com

[ConnorJC3]

Needs eksctl version bump to https://github.com/weaveworks/eksctl/releases/tag/v0.109.0-rc.0 for k8s 1.23 support.

[olemarkus]

@torredil I wonder if it is worth implementing a feature in kops where you tell kops the addon is enabled, but it's selfmanaged. That way kops will behave as if it installed the addon, but won't deploy the manifest. We do this e.g for cert-manager, which a lot of people prefer to self-manage.

This will make it a lot easier to migrate to 1.24, where CSI driver is mandatory and cannot be disabled.

[ConnorJC3]

kops patch needs merge with this to disable security feature of preventing nodes from accessing IMDSv2:

spec:
  instanceMetadata:
    httpPutResponseHopLimit: 3

[torredil]

@olemarkus Can you clarify this point: "CSI driver is mandatory and cannot be disabled". My understanding was that even in kOps v1.24, users would still be able to disable the CSI driver as currently done in kops-patch.yaml.

cloudConfig:
    awsEBSCSIDriver:
      enabled: false

If this is not the case, then I think the solution you are proposing would be worth implementing.

[olemarkus]

@olemarkus Can you clarify this point: "CSI driver is mandatory and cannot be disabled". My understanding was that even in kOps v1.24, users would still be able to disable the CSI driver as currently done in kops-patch.yaml.

cloudConfig:
    awsEBSCSIDriver:
      enabled: false

If this is not the case, then I think the solution you are proposing would be worth implementing.

To be precise, kOps 1.24 with K8s 1.24 will enable external cloud-controller-manager, which in turn requires CSI. So the above snippet will fail with "external cloud controller manager requires CSI driver to be enabled" or something to that effect.

[ConnorJC3]

@olemarkus thoughts on just disabling the external CCM in CI as a temporary fix? I can't think of any reason it would be needed in CI.

Also, I think I might have seen someone mention this elsewhere on another issue, but I wonder if as a long term fix it would be better to add a kops option to override the EBS CSI Driver container repo/tag. Then, the CI could just deploy the driver via kops which would be easier and probably more accurately represent real-world customer installations.

[torredil]

Adding this here for context: As of kOps 1.22, new clusters running Kubernetes 1.22+ on AWS will restrict Pod access to the instance metadata service. This means that Pods will also be prevented from directly assuming instance roles. See https://kops.sigs.k8s.io/iam_roles/#adding-additional-policies

[olemarkus]

@olemarkus thoughts on just disabling the external CCM in CI as a temporary fix? I can't think of any reason it would be needed in CI.

kOps is moving away from in-tree CCM. From kOps perspective it doesn't make much sense to add any option to disable it.

Also, I think I might have seen someone mention this elsewhere on another issue, but I wonder if as a long term fix it would be better to add a kops option to override the EBS CSI Driver container repo/tag. Then, the CI could just deploy the driver via kops which would be easier and probably more accurately represent real-world customer installations.

Yes. That is also an option. There are quite a few images involved though, so a question of how much configurability you need. E.g do you want to test all the sidecars?

FWIW, we run the CSI driver e2e test against all our configurations now. So e.g the 1.23 results can be seen here: https://testgrid.k8s.io/kops-k8s-1.23#kops-aws-k8s-1-23 (tests containing the string "External Storage")

[ghost]

@torredil anything we can do to get this merged anytime soon? We currently are running our own tests manually on every driver release because upstream doesn't bother running them for k8s 1.25

[olemarkus]

@MurphyPuppy See e.g https://testgrid.k8s.io/kops-k8s-1.23#kops-aws-k8s-1-23 mentioned above. The k8s community already tests this driver on multiple versions and configurations.

[torredil]

/retest

[olemarkus]

Some great work here!

[ConnorJC3]

/hold for more reviews
/lgtm other than needing an irebase to cleanup the history

[ConnorJC3]

/lgtm

[wongma7]

/lgtm
thx

[torredil]

Good to go
thanks team 👍

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gtxu, torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gtxu,torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gtxu]

/lgtm
/unhold