Add tolerations for taints on NVIDIA specific node groups

[Skarlso]

Description

Closes #5277

I believe this will do the trick, but I have to manually test it first. Is this right, @cPu1? I'm a bit unfamiliar with this part of the code. :)

TODO:

• Manual run

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the userdocs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes
• (Core team) Added labels for change area (e.g. area/nodegroup) and kind (e.g. kind/improvement)

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[Skarlso]

With the following config file:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: gb-test-cluster-1
  region: us-west-2
  version: '1.22'

nodeGroups:
  - name: ng-1
    minSize: 1
    maxSize: 2
    desiredCapacity: 1
    instanceType: p2.xlarge
    taints:
      feaster: "true:NoSchedule" 

Achieved this on the nvidia pod:

Tolerations:                 CriticalAddonsOnly op=Exists
                             feaster=true
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             nvidia.com/gpu:NoSchedule op=Exists

And the pod is successfully scheduled on the node:

kube-system   nvidia-device-plugin-daemonset-c4ln5   1/1     Running   0          11m

[cPu1]

Is this right, @cPu1?

Yes, the approach does look right to me 🙂

[Himangini]

Extend unit tests

I can't see this in the files diff, where is this added? 🤔

[Skarlso]

Right, that didn't happen because there aren't any. :D

[Himangini]

👍🏻
At some point, we should add some tickets to improve testing around this and taints in general 💡