ucp_mem_map Failure with AMD 7900XTX GPU

[chris-barnes-at-etherform-com]

Hello,
I am trying to get peer-to-peer working between a Mellanox/NVidia ConnectX-6 and AMD 7900xtx GPU using UCX. I believe I have everything correctly working up until the point where I am registering the memory region to get the memory handle using ucp_mem_map().

when I call ucp_mem_map, I see the following trace:

[1743444013.720397] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffe8d600000..0x7ffe8fffffff
[1743444013.720400] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffe94000000..0x7ffe955fffff
[1743444013.720402] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffe8c000000..0x7ffe8fffffff
[1743444013.720403] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7fffeb97e000..0x7fffeb9befff
[1743444013.720405] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffe80000000..0x7ffe83ffffff
[1743444013.720407] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffeea000000..0x7ffeea3efb1f
[1743444013.720408] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffeea000000..0x7ffee9ffffff
[1743444013.720410] [eng1:13991:0] rcache.c:368 UCX TRACE ucp_rcache: find regions in 0x7ffeea3efb20..0x7ffeea3efb1f
[1743444013.721948] [eng1:13991:0] ib_md.c:287 UCX ERROR ibv_reg_mr(address=0x7ffeea000000, length=4127520, access=0xf) failed: Invalid argument
[1743444013.721955] [eng1:13991:0] ucp_mm.c:76 UCX ERROR failed to register address 0x7ffeea000000 (rocm) length 4127520 on md[0]=rxe_dev: Input/output error (md supports: host|rocm)
[1743444013.721958] [eng1:13991:0] rcache.c:1036 UCX DEBUG failed to register region 0x1314bf0 [0x7ffeea000000..0x7ffeea3efb20]: Input/output error

At the moment I am running on my development machine with two processes trying to communicate with each other. My simple test is to move memory from GPU memory to host memory through UCX. The GPU memory is allocated in HIP, then loaded with test data (using normal HIP calls) then I register the memory using the device memory address. At this point I see the trace posted above.

UCX is compiled with ROCM support, IOMMU is enabled. I am currently using the soft-RDMA driver (could this be the issue?). I see that the GPU is in an IOMMU group all by itself (group 14).

In the trace it appears to be that UCX recognizes that the memory is "rocm", and then says that the network device supports "host|rocm" memory types. The specific error in "ibv_reg_mem" seems to indicate that there is an error registering that memory with remote read/write access (note that it incorrectly shows "Invalid argument" as the error, the documents indicate that the error message actually means that the "access" parameter is wrong).

Does anyone have any suggestions on how to resolve this issue?

One other issue that I am seeing is that when I run "rocminfo" I see "IOMMU Support:: None" here, despite the fact that the device is in an iommu group. Is this important? Is there a way to resolve this?

Thank you!

Environment variables:
UCX_TLS: rc,ud,dc,rocm_copy,rocm_ipc
HSA_ENABLE_SDMA: 1
UCX_RNDV_SCHEME: put_zcopy
HSA_FORCE_FINE_GRAIN_PCIE: 1
UCX_LOG_LEVEL: TRACE

I am using UCX v1.9 (whatever is on the main branch as of about two weeks ago.)

[edgargabriel]

@chris-barnes-at-etherform-com Thank you for filing the issue. This is most likely not a UCX issue itself, but a system setup question:

• can you please post the OS distro and kernel version that you are using?
• what MOFED or DOCA OFED version did you install?
• did you disable PCIe ACS on your system?

Unfortunately, the precise combination of linux kernel and OFED versions that one needs to have at the moment is a bit tricky, because one needs to either have versions that will still allow the ib_peer_mem memory registration mechanism to work, or a version of the kernel that supports the dmabuf based registration (the ROCm runtime has a few additional requirements that are met in newer kernels and newer distros, but not necessarily in some older ones)

[edgargabriel]

I did notice (after I posted) that I was able to allocate host memory and then map it into device memory space and then the ibv_reg_mr works (when using the device address). I am not sure what's going on there or if that would actually be a peer-to-peer transaction...

This is not the same as device memory. This is still host memory which has also been mapped into the GPU page tables, it does not require peer-mem support

[chris-barnes-at-etherform-com]

Alrighty, here's the update.

1. I was originally on the soft-RDMA Linux driver, not MLNX or DOCA (etc). I installed DOCA OFED and now "rdma" doesn't work on my development machine. Confession: I don't have the ConnectX-6 on this machine :) I was hoping for a solution for development.

2. I moved to the machine with the ConnectX-6 on it and now completing the ibv_reg_mr now successfully. This has OFED-24.10-1.1.4 (for completeness).

3. I am still having an error with packing the rkey, but I haven't spent any time debugging that yet so I may be back.

4. Is there a solution that doesn't involve having a physical ConnectX-6+ installed in my machine so that I can do development locally (instead of via remote desktop on a production machine)?.

Thank you!

[edgargabriel]

4. Is there a solution that doesn't involve having a physical ConnectX-6+ installed in my machine so that I can do development locally (instead of via remote desktop on a production machine)?.

It depends on your goal. You can do local, intra-node development with UCX and ROCm without an IB NIC and the OFED stack, that is something that I am doing on my local workstation all the time (with the same GPU that you are using). If your development goals involve something with inter-node communication, its a bit trickier.

[chris-barnes-at-etherform-com]

Alright, I am back at it again. Bottom line: I was able to get past the ibv_reg_mr part yesterday but I am back to the ibv_reg_mr error.

Previously I installed the lastest DOCA drivers but I was not able to get a successful RDMA transfer to work. As I mentioned, I got past the ibv_reg_mr error, but then had some issues with the rkey. Under this circumstance I was using "ucp_memh_pack()" to prepare the rkey which was failing; note that the documents point people to using this method currently but it does not seem to work (I can provide more details once I can get past the ibv_reg_mr() problem again).

I switched to using ucp_rkey_pack() and this was working for sending the rkey, memory address and various other parameters to the remote side using Active Messages. The rkey was unpacked on the remote side and then the call to ucp_get_nbx() was failing. I think the most recent error that I got was "-48" which didn't map to a UCS error message. Note that at this point I was using the "tcp" transport because it was inexplicably not detecting the RDMA transport. This is about where I ended the day yesterday.

I realized this morning that I was still using the software-RDMA tool to enable the device (as shown by "ibv_devices"); I had mistakenly thought that "rdma" was a standard OFED tool and that it would have been replaced by a Mellanox tool. I noticed that I was not seeing an Mellanox device under /dev/mst anymore, even though the Ethernet device was still working. So, I decided to go back to the Mellanox OFED, so I am running "MLNX_OFED_LINUX-24.10-2.1.8.0 (OFED-24.10-2.1.8)" at the moment and I see the /dev/mst device as well as "mlx5_0" when running "ibv_devices". It looks like MLNX_OFED replaced all of the UCX libraries, so I rebuilt and reinstalled them.

I now see that my application forms a link ("endpoint") using "rc_mlx5" and "ud_mlx5" transports and I have "rocm_copy" and "rocm_ipc" in there as well. However, I am back to (approximately) where I started which is the following error:

"ib_md.c:287 UCX ERROR ibv_reg_mr(address=0x76e0eb200000, length=4194304, access=0x10000f) failed: Bad address
ucp_mm.c:76 UCX ERROR failed to register address 0x76e0eb200000 (rocm) length 4194304 on md[0]=mlx5_0: Input/output error (md supports: host|rocm)"

I verified that I have the GPU in an iommu group (by itself) as well as the NIC. The NIC is showing up correctly with "ibv_devices" so that all seems to be good to go now. I will look at disabling ACS now. I still see "IOMMU Support:: None" when I do "rocminfo", even though IOMMU with PT is enabled - do you know what it takes for this to show something different?

I will post an update when I get further with ACS.

Thanks

[chris-barnes-at-etherform-com]

Alright, I think PCI ACS is disabled, I am not 100% sure. Here is the output of lspci for the GPU

c3:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX/7900 GRE/7900M] (rev c8) (prog-if 00 [VGA controller])
Subsystem: XFX Limited RX-79XMERCB9 [SPEEDSTER MERC 310 RX 7900 XTX]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 315
NUMA node: 0
IOMMU group: 30
Region 0: Memory at 10b000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at 10b800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at e000 [size=256]
Region 5: Memory at d8000000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at d8100000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [2d0 v1] Process Address Space ID (PASID)
PASIDCap: Exec+ Priv+, Max PASID Width: 10
PASIDCtl: Enable- Exec- Priv-
Capabilities: [320 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Capabilities: [410 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [450 v1] Lane Margining at the Receiver
Kernel driver in use: amdgpu
Kernel modules: amdgpu

I tried adding "pcie_acs_override=downstream,multifunction" to the grub parameters on boot up and that did not produce any output in the boot log ("dmesg") indicating that ACS was overridden so I assume that method of disabling ACS is no longer functional...?

However, from what I understand, since "ACSCap" has all minus signs next to it, there are no controls in place. Is that correct?

Regardless, I am still getting the same error mentioned in my previous post ("Bad Address").

[edgargabriel]

To your other question, iommu has to be set to pt (passthrough) and amd_iommu=on

[chris-barnes-at-etherform-com]

Hello,

I believe I have the latest OFED (OFED-internal-25.01-0.6.0), a recent kernel (6.11.0-21-generic #21~24.04.1-Ubuntu), PCIe ACS is disabled through the BIOS and OS (using a number of methods, all with the same result). "amd_iommu=on iommu=pt" are set in the grub kernel command line. I see the GPU in IOMMU group 21 and the NIC in IOMMU group 25. rocminfo reports "DMAbuf Support: YES".

Result of ucx_info -v:

Library version: 1.19.0

Library path: /lib/libucs.so.0

API headers version: 1.19.0

Git branch '', revision 4c43ec2

Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --enable-mt --prefix=/usr --with-rdmacm --with-verbs --with-mlx5 --enable-cma --enable-examples --without-java --without-go --with-rocm=/opt/rocm --without-xpmem

(I copied the OFED "configure" parameters and swapped out CUDA with ROCM)

I have been able to successfully complete an RDMA from system ("host") memory but not from GPU memory with the error:
"ib_md.c:287 UCX ERROR ibv_reg_mr(address=0x76e0eb200000, length=4194304, access=0x10000f) failed: Bad address
ucp_mm.c:76 UCX ERROR failed to register address 0x76e0eb200000 (rocm) length 4194304 on md[0]=mlx5_0: Input/output error (md supports: host|rocm)"

Could you post what combinations of ROCm/OFED/Kernel are known to work?

Is there anything required to allow the NIC to access GPU memoryspace or use DMABufs?

Is there anything required to force HIP to use DMABufs, or does it do that by default (e.g., via hipMalloc() )?

Is there anything else to look into? I have completely exhausted everything that I have found to make this work.

Thanks

[edgargabriel]

@chris-barnes-at-etherform-com can you ping me individually? I think we can sort it out in an email chain or chat easier, and then just post a solution here

[chris-barnes-at-etherform-com]

@edgargabriel Thanks for getting back to me. I am not sure how to contact you directly as there is no apparent method of doing so through GitHub.

[chris-barnes-at-etherform-com]

For anyone who comes here in the future: Setting the environment variable UCX_ROCM_COPY_DMABUF=y fixed the issue. :) Thank you @edgargabriel

Kernel must be capable of supporting DMABuf. I noticed that "rocminfo" will tell you if it does (towards the top).