Add integration tests for auto machine preservation

[aaronfern]

What this PR does / why we need it:
This PR adds integration tests for auto machine preservation.

The following cases have been covered:

• Ensure failed machine is preserved and rejoins the cluster on recovery
• Failed machines that cross the max threshold are not preserved
• machinePreserveTimeout is honoured for failing machines
• Machine should not be preserved when the preserve=false annotation is present
• Preserved machine should stop being preserved when preserve=false annotation is added
• When AutoPreserveFailedMachineMax is reduced, failed machines that exceed the new AutoPreserveFailedMachineMax count are moved to terminating

Which issue(s) this PR fixes:
Fixes partially #1123

Special notes for your reviewer:
To trigger machine failures, this PR takes the approach of applying a validating admission policy to block all kubelet updates. This causes kubelet not to be able to update it's node lease or it's node status and results in the node going into the Unknown state.
This is a k8s native way of achieving node failures and has no dependency on the provider or any other node/worker-pool setting

This might not work with the virtual provider of the virtual provider does not use kubelets, but imo this can be considered a special case and can be tackled in a separate PR

Release note:

Added integration tests for auto machine preservation

[thiyyakat]

/assign

[gagan16k]

Thanks for the PR!

[gagan16k]

I have some points/questions regarding this function

• Should we use defer instead of gomega.Expect(c.TargetCluster.DeleteVAPToRestartKubeletUpdates(ctx)).To(gomega.BeNil()) for cleanup steps like these? In case of any test assertion failure above where this is called, the VAP and VAPB might not get cleared if Ginkgo aborts the test immediately.
• I think we should also add this function in cleanTestResources(). If any matching resources get left behind, this way they get cleared after(and before) test runs.
• In case the IT process crashes (with no chance to execute the cleanup), the resources can get left behind in the cluster. Currently(pre-PR) even the test machine deployment, and the machines themselves can persist if this happens. Would it be worthwhile to add a make target to clean this up?

[aaronfern]

I've added DeleteVAPToRestartKubeletUpdates to DeferCleanups within tests and have added vap/vapb cleanup to cleanTestResources() in fbfc7b5
PTAL

[thiyyakat]

Thanks for adding these much-needed tests! 👍

[thiyyakat]

@aaronfern , should we have some IT to test reduction of AutoPreserveFailedMachineMax? If the number of preserved failed machines is equal to AutoPreserveFailedMachineMax already, and the value is reduced by 1, a machine should move to Terminating.

[gardener-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from thiyyakat. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aaronfern]

@aaronfern , should we have some IT to test reduction of AutoPreserveFailedMachineMax? If the number of preserved failed machines is equal to AutoPreserveFailedMachineMax already, and the value is reduced by 1, a machine should move to Terminating.

@thiyyakat Okay, sure, makes sense. I will add this test

[thiyyakat]

Thanks for adding the test and addressing the comments. Just a few nits.

[thiyyakat]

There are a few if-else if blocks that can be converted to if-else blocks. PTAL.

[gagan16k]

Some more comments, PTAL