Improve validation during upgrade e2e tests

[sbueringer]

We recently found an issue in KCP upgrades. This only lead to failures in self-hosted tests, but in general KCP was upgrading through the entire control plane and only static pods came up (kube-proxy, CNI did not). Also the Nodes weren't ready during the upgrade.

I think we should improve our e2e test coverage to detect if the control plane is in a state like this during upgrades.

This should be either additional validation while the upgrade is running or potentially we can e.g. deploy a StatefulSet with PDBs that runs on CP nodes. In this case the test would have failed with the issue from #10947

Bonus: Potentially the Statefulset with PDBs we deploy could also use volumes for some bonus test coverage for "wait for volume detach" but it's unclear if we have a CSI implementation for CAPD that would work for that (maybe the provisioner that comes with kind out-of-the-box)

[sbueringer]

/triage accepted

[sbueringer]

/help

[k8s-ci-robot]

@sbueringer:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[chrischdi]

/assign