[Feature] nvidia device plugin not scheduled on GPU nodes if they have additional taints

[yuvipanda]

What were you trying to accomplish?

I'm trying to setup an unmanaged GPU nodegroup with some additional taints, and have eksctl automatically install the nvidia device plugin daemonset correctly.

What happened?

Because I've additional taints on my nodegroup and the nvidia-device-plugin daemonset doesn't have tolerations for those, it's never scheduled on the GPU nodegroup.

How to reproduce it?

Here's the appropriate nodegroup definition:

      {
         "availabilityZones": [
            "us-west-2b"
         ],
         "desiredCapacity": 0,
         "iam": {
            "withAddonPolicies": {
               "autoScaler": true
            }
         },
         "instanceType": "p2.xlarge",
         "labels": {
            "hub.jupyter.org/node-purpose": "user",
            "k8s.dask.org/node-purpose": "scheduler",
            "node.kubernetes.io/instance-type": "p2.xlarge"
         },
         "maxSize": 500,
         "minSize": 0,
         "name": "nb-p2-xlarge",
         "ssh": {
            "publicKeyPath": "ssh-keys/uwhackweeks.key.pub"
         },
         "tags": {
            "k8s.io/cluster-autoscaler/node-template/label/hub.jupyter.org/node-purpose": "user",
            "k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose": "scheduler",
            "k8s.io/cluster-autoscaler/node-template/label/node.kubernetes.io/instance-type": "p2.xlarge",
            "k8s.io/cluster-autoscaler/node-template/taint/hub.jupyter.org/dedicated": "user:NoSchedule",
            "k8s.io/cluster-autoscaler/node-template/taint/hub.jupyter.org_dedicated": "user:NoSchedule"
         },
         "taints": {
            "hub.jupyter.org/dedicated": "user:NoSchedule",
            "hub.jupyter.org_dedicated": "user:NoSchedule"
         },
         "volumeSize": 80
      },

The created nvidia-device-plugin daemonset only has the following tolerations:

      - key: CriticalAddonsOnly
        operator: Exists
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

So it never gets scheduled on my GPU nodes.

eksctl should recognize additional taints on the nodegroup and automatically add tolerations for that to the nvidia-device-plugin it autoinstalls.

Logs

Anything else we need to know?

eksctl is awesome and the automatig GPU driver installation is a killer idea.

Versions

$ eksctl info
➜ eksctl info
eksctl version: 0.97.0
kubectl version: v1.23.5
OS: darwin

[Skarlso]

Hi @yuvipanda.

Thank you for you issue.

We could try and patch the created addon daemonset after creation, but that wouldn't hold after an upgrade for example.

The aws-sdk-go-v2 doesn't expose this ability. We can patch the daemonset and apply some tolerations ( although which tolerations to apply will not be autodetected for fear of applying a tolerations which should have been ( maybe it could be configured through the addon configuration section )). We already path aws-node anyways. But the best would a native support from the SDK. I suggest opening an issue there as well.

[cPu1]

@yuvipanda, that's a valid request, I can see the utility of this feature. The device plugin is installed as part of create cluster and create nodegroup when an EKS-Optimized Accelerated AMI with a GPU-enabled instance type is used. If your ClusterConfig contains a single nodegroup that matches this criterion, then eksctl can apply the taints from that nodegroup as tolerations to the device plugin, and if the ClusterConfig contains multiple such nodegroups that do not all have the same taints config, eksctl can combine the set of taints config from all such nodegroups and apply them as tolerations.

I have removed the bug label as this is more of a missing feature than a bug.

[Skarlso]

Ah I thought for some reason that the taints are dynamic.

[cPu1]

Ah I thought for some reason that the taints are dynamic.

Taints can be changed via kubectl taint nodes or by using aws eks update-nodegroup-config but this is outside of the purview of eksctl and would not be supported.

[Skarlso]

Taints can be changed via kubectl taint nodes or by using AWS eks update-nodegroup-config but this is outside of the purview of eksctl and would not be supported.

I meant that they can't be determined during a run or have arbitrary values which could cause problems with other things being scheduled. But that's not the case here. :) So never mind :)

[yuvipanda]

Thanks @cPu1! The behavior you describe is exactly what I want. I agree that taints made outside of eksctl should be out of scope here.

[Skarlso]

@yuvipanda hey, can you please share a config file for my convenience? :) Thanks

[yuvipanda]

@Skarlso does this work? https://github.com/2i2c-org/infrastructure/blob/81b4b506c700646870da079248ee39de72b1fd2f/eksctl/uwhackweeks.jsonnet#L23 :D

[Skarlso]

It's fine, I wrote one I can use for testing. :D The linked PR should fix your problem. :)