Add out of service taint remediation

[k-keiichi-rh]

This PR is adding a new remediation strategy based on kubernetes/enhancements#1116
In k8s 1.26, kubernetes/enhancements#1116 is a Beta feature. This feature will be graduated to GA in k8s 1.27.

This PR is implemented based on the discussion in #17.
The following is the new remediation strategy for the out-of-service taint:

1. When detecting node failure by NHC, SNR sets NoExecute taint on the failed Node
2. The failed node is rebooted by watchdog
3. SNR waits for a fixed time to complete rebooting the failed node
4. SNR sets the out-of-service taint
   => This taint expects that the node is in shutdown or power off state (not in the middle of restarting) according to the enhancement
   However, SNR works well with this taint as we discussed in [RFC] Support for out-of-service taint #17 (comment).
5. If the failed node is rebooted and there is no terminating pods and volumeattachements, SNR deletes the out-of-service taint.
   => SNR waits for deleting workloads on the failed node with a timer. If the timer is expired, the reconcile method is triggered.
6. The fencing complete is set
7. After setting the fencing complete, it starts cleaning up and then the NoExecute taint is deleted and the failed node becomes schedulable again
8. The remediation is completed.

ECOPROJECT-1228

[openshift-ci]

Hi @k-keiichi-rh. Thanks for your PR.

I'm waiting for a medik8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[slintes]

I'm having a hard time to follow the remdiation flow. I'd suggest to reorder methods, and maybe rename some of them in order to get rid of too many methods with similar names (remediate/remove/deleteResources)?

e.g.:

Reconcile

remediateWithResourceDeletion
removeNodeResourcesByResourceDeletionWrapper -> deleteResourcesWrapper (not nice that we need this wrapper, but I have no quick better idea)
removeNodeResourcesByResourceDeletion -> deleteResources (much easier and better to understand IMHO)

remediateByAddingOutOfServiceTaint -> remediateWithOutOfServiceTaint (align name with remediateWithResourceDeletion)
removeNodeResourcesByAddingOutOfServiceTaint -> useOutOfServiceTaint (much easier and better to understand IMHO)

remediateWithResourceRemoval

[slintes]

thanks a lot for renaming / reordering, much better readable now IMHO 🙂

/ok-to-test

[k-keiichi-rh]

/retest-required

[k-keiichi-rh]

/retest-required

[k-keiichi-rh]

/test 4.11-openshift-e2e

[slintes]

One nit inside, otherwise lgtm. Needs rebase though.

[slintes]

Looks good to me, nice work!
Leaving the final review to @mshitrit

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: k-keiichi-rh, slintes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [slintes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[slintes]

couldn't get rid of the "change requested" state in another way 🤷🏼‍♂️

/lgtm cancel

[k-keiichi-rh]

/test 4.13-openshift-e2e

[mshitrit]

/lgtm