[AMD] Investigate test failures for different hardware

[veselypeta]

Some Program/Kernel/Enqueue Conformance tests fail differently depending on the hardware being used. This might be an issue with the confromance tests or the plugin implemenations. In #970 we marked these tests optional with the {{OPT}} keyword in the match file to allow the CI to pass, but we should investigate why these tests are failing on some hardware.

[JackAKirk]

Are you running on officially supported hardware/os/rocm version combinations? : https://docs.amd.com/en/latest/release/gpu_os_support.html#linux-supported-gpus

[veselypeta]

We have an RX6800XT which I suppose is not officially supported.

[JackAKirk]

We have an RX6800XT which I suppose is not officially supported.

Whilst unsupported devices/os/rocm version combinations may work (at least for some workloads), we should not be expecting testing to pass on such devices. Test failures on such non-officially supported setups may point to real problems on supported devices, but it can be very hard for us to know this a priori, if we only see the failures on an unsupported device. Although sometimes if you scour https://github.com/RadeonOpenCompute/ROCm/issues you can be pointed to related issues with rocm, and sometimes users ask about failures on officially unsupported devices: usually they are simply referred to the fact that their device is not supported and no support can be given. Unfortunately the intel/llvm hip backend CI is currently using an unsupported device with a rocm version incompatible with the CI os, which is coinciding with failures that can't be reproduced outside the CI (see intel/llvm#7634 (comment)). Hopefully the CI will be reverted to a supported setup soon.

[lukaszstolarczuk]

@veselypeta, can you please point me out to logs (or give me an example output) for different HW? We have 2 runners for AMD in CI, they both use the same HW (Radeon RX 6800 XT) but they use various rocm version (5.6.0 vs 5.7.0) and perhaps some packages are missing on one of OSes. I could try and check both machines if I knew where to look.

Of course, I'm assuming we're talking about the HW available in our CI 😄

[lukaszstolarczuk]

I just realized the second machine has now the updated version of rocm (5.7.0). Could you please repeat the testing, perhaps now it works as expected....?

[veselypeta]

@lukaszstolarczuk I'm not exactly sure what the issue actually is and it might just be user error. But I see different failures based on the hardware I use. In CI the gfx1030(RX6800XT) segfaults on some enqueubuffer tests, while this passes for me locally on my RX6650XT. Since my card is not officially supported however, I suppose we should just take the results from the offically supported cards as expected behaviour. However, it would be nice to still be able to have passing tests locally even with an unsupported card as it works (most of the time 🤞 )

[veselypeta]

Perhaps this is related, but I'm seeing another test fail differently across the two runners:

• pc_amd_23 - Passed
• pc_amd_34 - Failed

This is a timing test so perhaps it's not reliable

[JackAKirk]

Perhaps this is related, but I'm seeing another test fail differently across the two runners:

* [pc_amd_23](https://github.com/oneapi-src/unified-runtime/actions/runs/6653316695/job/18079038446?pr=1001) - Passed

* [pc_amd_34](https://github.com/oneapi-src/unified-runtime/actions/runs/6653316695/job/18079038200?pr=1001) - Failed

This is a timing test so perhaps it's not reliable

It would be good if the pipeline ran

sycl-ls

after build
or something else that tells you the device. As far as I can see this info is not provided in the output. Also it would be good to see the OS and rocm versions of the two runners.

[lukaszstolarczuk]

Perhaps this is related, but I'm seeing another test fail differently across the two runners:

* [pc_amd_23](https://github.com/oneapi-src/unified-runtime/actions/runs/6653316695/job/18079038446?pr=1001) - Passed

* [pc_amd_34](https://github.com/oneapi-src/unified-runtime/actions/runs/6653316695/job/18079038200?pr=1001) - Failed

This is a timing test so perhaps it's not reliable

It would be good if the pipeline ran

sycl-ls

after build or something else that tells you the device. As far as I can see this info is not provided in the output. Also it would be good to see the OS and rocm versions of the two runners.

I just had a similar though - I filled an issue for that and I'll try to do it soon (#1002)

As for the moment, these 2 runners have the same OS (Ubuntu 22.04), the same AMD card (Radeon RX 6800 XT), and the same rocm version (5.7.0), but they have different CPUs - "amd_23" has i7-4790 CPU @ 3.60GHz and "amd_34" has i7-6700K CPU @ 4.00GHz. Could this impact the mentioned test?

They might also differ a little in packages installed (since they were prepared independently).

[JackAKirk]

As for the moment, these 2 runners have the same OS (Ubuntu 22.04), the same AMD card (Radeon RX 6800 XT), and the same rocm version (5.7.0), but they have different CPUs - "amd_23" has i7-4790 CPU @ 3.60GHz and "amd_34" has i7-6700K CPU @ 4.00GHz. Could this impact the mentioned test?

They might also differ a little in packages installed (since they were prepared independently).

Lord knows.

It looks like it is this test:

unified-runtime/test/conformance/device/urDeviceGetInfo.cpp

Line 16 in a2d23dc

{UR_DEVICE_INFO_SINGLE_FP_CONFIG, sizeof(ur_device_fp_capability_flags_t)},

But also from what I can tell the output is not clear on what exactly is failing.

Also the cards are the same one as used in intel/llvm CI which are not officially supported.

This message on the ROCm github suggest that RX6800 XT is likely to mostly work with ROCm:

ROCm/ROCm#1714 (comment)

But the same message emphasizes that it is not officially supported. Also there are other messages suggesting recent issues with the driver using this card:

ROCm/ROCm#1590 (comment)

There is also the point that ROCm device support is constrained by ROCm version and OS combination (see the issue 1590 above). The good thing is that for unified-runtime CI the ROCm version and OS combination appears to be officially supported at least.

You can probably tell by now that the ROCm driver support is a bit troublesome.. Which is why I think it would be sensible to stay on the safe side of things.