No resource specifications for neuron-device-plugin ds

[RobertLucian]

The current daemonset for k8s-neuron-device-plugin.yaml doesn't have any resource requests/limits. I take that setting requests/limits is a good strategy to prevent the container from hogging too many resources.

I'm thinking of something resembling this:

resources:
  requests:
    cpu: 200m
    memory: 500Mi
  limits:
    memory: 500Mi

What I'm asking is: do the resource requirements change depending on the number of Inferentia chips available? Or depending on other unknown factors?

[mrnikwaws]

Hi Robert,

Thanks for asking this question. This container is purely for handling kubelet requests and so uses little CPU or memory. If you have 16 jobs on 16 neuron cores there will be some scaling, but the resource requirements are minimal to start with.

For now the team aren't see value in these kinds of limits for this container, but please let us know if you are seeing problems with container resources and we'll look at this again.

If I don't hear back from you I will plan to close this issue.

[RobertLucian]

Gotcha. Thanks for detailing the situation. I'll let you know if I see any problems with the container resources. In the meantime, I think it's okay to close this issue. I'll leave it up to you.