Device logical IDs must be contiguous

[RobertLucian]

Image this: I have a few pods running on an inf1.6xlarge instance. Each pod has a neuron runtime running as a sidecar. When each pod asks for 2 inf1 devices, there's a very high chance that one of them will get 2 non-contiguous logical IDs. Here's an example of what I get:

bash-4.2# neuron-rtd -g unix:/sock/neuron.sock -x
nrtd[17]: [NRTD:nrtd_main] nrtd build using:1.0.6222.0
nrtd[17]: [NRTD:InitTongas] MLA logical Ids must be contiguous
nrtd[17]: [NRTD:nrtd_main] Failed to initialize devices: 0000:00:1c.0 0000:00:1f.0 , attempt: 1
neuron-rtd[17]: [TDRV:tdrv_destory] TDRV not initialized

This causes the pod to enter into a crash loop and that's undesirable.

The only solution I see at the moment is to only have pods asking 1 or exactly how many inf chips are on the given instance (in this case it would be 4).

@mrnikwaws any idea what could be done here?

[RobertLucian]

Actually, this seems to be a problem when requesting all inf devices on an instance as well. Here's what I get:

nrtd[1]: [NRTD:nrtd_main] nrtd build using:1.0.6222.0
nrtd[1]: [NRTD:InitTongas] MLA logical Ids must be contiguous
nrtd[1]: [NRTD:nrtd_main] Failed to initialize devices: 0000:00:1d.0 0000:00:1e.0 0000:00:1f.0 0000:00:1c.0 , attempt: 1
neuron-rtd[1]: [TDRV:tdrv_destory] TDRV not initialized

Deleting the pod and letting it recreate itself worked the 2nd time:

nrtd[1]: [NRTD:nrtd_main] nrtd build using:1.0.6222.0
nrtd[1]: [TDRV:reset_mla] Resetting 0000:00:1c.0
nrtd[1]: [TDRV:reset_mla] Resetting 0000:00:1d.0
nrtd[1]: [TDRV:reset_mla] Resetting 0000:00:1e.0
nrtd[1]: [TDRV:reset_mla] Resetting 0000:00:1f.0
nrtd[1]: [TDRV:tdrv_init_one_mla_phase2] Initialized Inferentia: 0000:00:1f.0
nrtd[1]: [TDRV:tdrv_init_one_mla_phase2] Initialized Inferentia: 0000:00:1d.0
nrtd[1]: [TDRV:tdrv_init_one_mla_phase2] Initialized Inferentia: 0000:00:1e.0
nrtd[1]: [TDRV:tdrv_init_one_mla_phase2] Initialized Inferentia: 0000:00:1c.0
D0508 22:57:08.389372935       1 env_linux.cc:71]            Warning: insecure environment read function 'getenv' used
nrtd[1]: [NRTD:RunServer] Server listening on unix:/sock/neuron.sock

But this is consistent and it's repeating itself. I suppose this is a bug at this point. What do you think?

[mrnikwaws]

Hi Robert,

Thanks for reporting this problem. We're investigating and will post back here when we have more information

[aws-taylor]

Hello Robert,

Just a follow-up, we have completed the code and testing for this fix and are in the process of getting the change incorporated into our release. If you are interested in participating in a preview release and discussing your use-case please let us know.

Regards,
Taylor

[RobertLucian]

Thanks a lot for coming back with a reply @aws-taylor. Just to keep you updated, we still need this functionality for sure, but we can wait until its official release. Any idea on its ETA until it lands in the official version?

[micwade-aws]

Plan is to deliver a k8 scheduler extension that will deal with the need for contiguous device ids by the next release (or sooner). Given that we're dealing with this in the scheduler and not the driver, there's going to be new documentation outlining what specifically needs to be adjusted for k8 clusters.

[micwade-aws]

@RobertLucian - the latest release included the k8 scheduler: https://github.com/aws/aws-neuron-sdk/releases/tag/v1.7.0

Please let us know if this misses the mark in terms of what you need to take advantage of inf1.6xlarge and inf1.24xlarge sizes.

[RobertLucian]

@micwade-aws thanks for making this a reality! I'll keep you updated in case something doesn't work or is unclear. Thanks a lot again!