embedded hwloc 2.7.1 is too old for Intel processors with Ponte Vecchio accelerators

[hppritcha]

The hwloc 2.7.1 embedded in main and 5.0.x is too old for Intel processors with Ponte Vecchio accelerators. The lstopo built using this version or older of hwloc segfaults when run on such processors, and prterun experiences a similar segfault when run using the embedded hwloc 2.7.1.

Solution is to use a 2.8 or newer version of hwloc.

[jsquyres]

This should also be noted in the docs and/or in the configury test for the minimum hwloc version.

[rhc54]

Isn't that going to be a significant issue for the distros? I believe their defaults are quite a bit older, aren't they?

[hppritcha]

This might be hard to do a configury check at the moment. Frontend nodes will likely be vanilla Intel XE, only backends will likely have the Ponte Vecchio accelerators that seem to cause the issue.

Where in the docs would you recommend writing a blurb about this?

[rhc54]

Ah, so you want to do a runtime check of the version? I guess we can do that simply enough - would have to be in PMIx so we can cover both mpirun and direct launch modes. Would you mind opening an issue over there so we don't forget?

@jsquyres Is this going to be an issue re default hwloc versions on distros (I'm thinking of Amazon here, so @bwbarrett )? I don't know of any other solution, frankly, though I wonder if it wouldn't segfault if we asked it to not include those devices. We can do that if we pass the appropriate flags - @hppritcha has that been tried?

[hppritcha]

I guess a runtime check in pmix would be nice, although the segfault was happening inside prte i think. anyway easy workaround at ANL is to use the hwloc they installed in a spack build out.

[jsquyres]

@hppritcha Ah, you edited the description of this PR -- I think I understand better now: it's a run-time error with older hwloc on Intel with PV accelerators. In this case, is it easy to add a run-time check to see a) if we're on a machine with Intel PV accelerators, and b) the version of hwloc?

If we can detect both of these things at run time, then we should probably show_help() a warning that advises the user of the issue and that they might need to re-build Open MPI with an hwloc >= v2.8.

[rhc54]

Pretty sure there is a "get_version" function in hwloc - will fiddle with it in pmix as that is the base layer that provides the topology.

[jsquyres]

Pretty sure there is a "get_version" function in hwloc - will fiddle with it in pmix as that is the base layer that provides the topology.

K. @hppritcha is there an easy way to tell that we're on a platform with Intel PV accelerators? Perhaps the presence of some file in /sys or something?

[rhc54]

Good point - we don't want to blanket block hwloc versions less than 2.8 for all platforms

[hppritcha]

@rhc54 says he knows a way - see openpmix/openpmix#2893
nothing obvious from lspci or kernel modules that jumps out to me.

[rhc54]

Let's ask @bgoglin - is there a way for us to detect that Intel PV accelerators are present on a system prior to using HWLOC? Please see above discussion as to why that is an important question. Any guidance would be appreciated!

[bgoglin]

I don't have access to a PV server, I am going to ask. I'd assume we'd need to look for specific PCI vendor:device IDs.

[bgoglin]

PCI vendor:device is indeed a good way to identify PV devices, and the list at https://pci-ids.ucw.cz/read/PC/8086 looks correct when it reports device ID = 0x0db[05-9ab] for Ponte Vecchio.

However the hwloc bug is related to device with multiple "levelzero subdevices", not to PV specifically, and there are some non-PV devices with multiple subdevices. But those are likely rare for now (discrete GPUs), at least when used in HPC. One solution would be to query level-zero subdevices instead of looking for PCI id. I am not aware of any way to query subdevices from sysfs or anything outside the L0 library.

By the way, the hwloc fix (762fdc4bfb8fc33b304511149a532a122ade395f) is basically the only change in 2.7.x after 2.7.1. I can release a 2.7.2 if needed.

[rhc54]

Hmmm...that sounds like we need to simply block everything below 2.7.2? It's an ugly "fix", but I cannot think of any other way to protect against segfault - can you?

Adding an L0 dependency is possible, I suppose, but given the history we've had with that library, not something I'm wild about doing. If someone wants to contribute it (it would go in the PMIx pgpu/intel component), I'm willing to look at it. Otherwise, the HWLOC cutoff is the only solution I can see.