[SYCL][CUDA][HIP] Add support for sycl::aspect::ext_intel_pci_address

[al42and]

Using (cu|hip)DeviceGetPCIBusId API call, present since at least CUDA 8.0 / ROCm 2.8.

Unlike L0, we do not require setting SYCL_ENABLE_PCI=1 environment variable.

[al42and]

$ cat test_pci.cpp 
#include <iostream>
#include <sycl/sycl.hpp>

int main() {
  auto devices = sycl::device::get_devices(sycl::info::device_type::gpu);

  for (const auto &device : devices) {
    try {
      std::cout << "Device Name: "
                << device.get_info<sycl::info::device::name>() << std::endl;
      std::cout << "Device has ext_intel_pci_address: "
                << device.has(sycl::aspect::ext_intel_pci_address) << std::endl;
      std::cout
          << "PCI Bus ID: "
          << device.get_info<sycl::ext::intel::info::device::pci_address>()
          << std::endl;
    } catch (sycl::exception &e) {
      std::cerr << "SYCL exception caught: " << e.what() << std::endl;
    }
    std::cout << std::endl;
  }

  return 0;
}

$ clang++ -fsycl test_pci.cpp -o test_pci && SYCL_ENABLE_PCI=1  ./test_pci
Device Name: Intel(R) Arc(TM) A770 Graphics
Device has ext_intel_pci_address: 0
PCI Bus ID: SYCL exception caught: Native API failed. Native API returns: -30 (PI_ERROR_INVALID_VALUE) -30 (PI_ERROR_INVALID_VALUE)

Device Name: Intel(R) UHD Graphics 770
Device has ext_intel_pci_address: 0
PCI Bus ID: SYCL exception caught: Native API failed. Native API returns: -30 (PI_ERROR_INVALID_VALUE) -30 (PI_ERROR_INVALID_VALUE)

Device Name: Intel(R) Arc(TM) A770 Graphics
Device has ext_intel_pci_address: 1
PCI Bus ID: 0000:0b:00.0

Device Name: Intel(R) UHD Graphics 770
Device has ext_intel_pci_address: 1
PCI Bus ID: 0000:00:02.0

Device Name: NVIDIA GeForce RTX 3060
Device has ext_intel_pci_address: 1
PCI Bus ID: 0000:01:00.0

Device Name: AMD Radeon RX 6400
Device has ext_intel_pci_address: 1
PCI Bus ID: 0000:07:00.0

[jinz2014]

Could you please explain the values "0" and "1" for the same device ? Thanks.

Device Name: Intel(R) Arc(TM) A770 Graphics
Device has ext_intel_pci_address: 0
PCI Bus ID: SYCL exception caught: Native API failed. Native API returns: -30 (PI_ERROR_INVALID_VALUE) -30 (PI_ERROR_INVALID_VALUE)

Device Name: Intel(R) Arc(TM) A770 Graphics
Device has ext_intel_pci_address: 1
PCI Bus ID: 0000:0b:00.0

[al42and]

Could you please explain the values "0" and "1" for the same device ? Thanks.

The first ones (without the aspect) are from the OpenCL backend; the second ones (with the aspect) are from LevelZero. That's a known limitation of the OpenCL backend (and, until this PR, of CUDA and HIP too).

The Intel devices are not directly related to this MR, but they demonstrate that the output format is consistent.

[jchlanda]

Would you be able to convert your test_pci.cpp sample into an e2e test?

[al42and]

Would you be able to convert your test_pci.cpp sample into an e2e test?

There is https://github.com/intel/llvm/blob/sycl/sycl/test-e2e/Basic/intel-ext-device.cpp. I'm not sure how it's output is verified in CI, but it works ok for me on NVIDIA and AMD devices:

Platform #7:
Device #1: NVIDIA GeForce RTX 3060:
Backend: CUDA
PCI address = 0000:01:00.0
Device UUID = 115971881914139240342461923712712616593242
Device ID = 0

Platform #8:
Device #1: AMD Radeon RX 6400:
Backend: Unknown
PCI address = 0000:07:00.0
Device UUID = 888800000000000000
Device ID = 0

(The UUID for the HIP device is suspicious, should I open another issue? And it fails the "Expect total_memory >= free_memory" assertion for Arc A770, since total memory is 0, but free memory is non-zero. But the PCIE addresses are reported fine).

[jchlanda]

Would you be able to convert your test_pci.cpp sample into an e2e test?

There is https://github.com/intel/llvm/blob/sycl/sycl/test-e2e/Basic/intel-ext-device.cpp. I'm not sure how it's output is verified in CI, but it works ok for me on NVIDIA and AMD devices:

Platform #7:
Device #1: NVIDIA GeForce RTX 3060:
Backend: CUDA
PCI address = 0000:01:00.0
Device UUID = 115971881914139240342461923712712616593242
Device ID = 0

Platform #8:
Device #1: AMD Radeon RX 6400:
Backend: Unknown
PCI address = 0000:07:00.0
Device UUID = 888800000000000000
Device ID = 0

(The UUID for the HIP device is suspicious, should I open another issue? And it fails the "Expect total_memory >= free_memory" assertion for Arc A770, since total memory is 0, but free memory is non-zero. But the PCIE addresses are reported fine).

Sorry, wasn't aware of that test.

Opening a ticket wouldn't hurt, thanks.

[steffenlarsen]

LGTM! Thank you!