When will llama.cpp's vulkan provide support for Intel Arc's matrix core?

[JamasChuang94]

When will llama.cpp's vulkan provide support for Intel Arc's matrix core?

[0cc4m]

When someone figures out a way to implement that. The current VK_KHR_cooperative_matrix extension works well on AMD and on Nvidia, but on Intel it just reduces performance significantly. There are some driver issues and unclear implementation details at play, it's not straightforward to fix.

[h9j6k]

There are some driver issues and unclear implementation details at play, it's not straightforward to fix.

0cc4m, I saw you guys' exchange last month on mesa website, have you guys discussed any progress or way forward since then? On my DG1, coopmat does not work at all, it only outputs gibberish so I have to disable VK_KHR_cooperative_matrix. I really hope I can have it work so I can compare it with DP4A commit you merged yesterday. By the way, DP4A by you works great! Many thanks for that!

[0cc4m]

There are some driver issues and unclear implementation details at play, it's not straightforward to fix.

0cc4m, I saw you guys' exchange last month on mesa website, have you guys discussed any progress or way forward since then?

No, I have not heard back from any mesa developers.

On my DG1, coopmat does not work at all, it only outputs gibberish so I have to disable VK_KHR_cooperative_matrix. I really hope I can have it work so I can compare it with DP4A commit you merged yesterday. By the way, DP4A by you works great! Many thanks for that!

It shouldn't output gibberish, but it's disabled by default due to the performance issues. The Vulkan interface to use accelerators like tensor cores and XMX is (at least in the coopmat1 case) very limited and makes implementations complicated. Nvidia is trying to fix that with the coopmat2 extension, but it will take a long time until that arrives in AMD and Intel implementations. For now I don't see a way of fixing this for Intel without more info.

[h9j6k]

It shouldn't output gibberish, but it's disabled by default due to the performance issues.

My bad I didn't mention the way I tested it was enabling it by commenting out return false on this line. Then if I start llama-server, GGML_LOG_DEBUG shows matrix cores: khr_coopmat instead none. But I only see gibberish result. So I have disabled it (keep it in line with master).

Update: have to use env variable PIPELINE_SELECT to start llama-server per https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25994

[0cc4m]

In my tests coopmat on Intel (A770) is slow, but gives correct results. If it doesn't for you, there might be an issue with your driver.

[h9j6k]

there might be an issue with your driver.

It could be. DG1 and DG2 (including A770) are known to have incomplete i915 support in vanilla kernel upstream. I tried intel oot driver quite a while back, it didn't work most of time (cpu hang, device lost etc.) I gave up on it soon afterwards.

Both DG1 and DG2 can be driven by xekmd and i915. But mesa xe driver support is experimental and it warns no bug report for experimental. When i915 and xe both work, i915 gives faster inference.

[morongosteve]

(Nods like I understand completely)

[qnixsynapse]

0cc4m, I saw you guys' exchange last month on mesa website

That would be me with 0cc4m. I was the one who poked Intel for an update for which one of them opened the issue you mentioned but there was no response after that unfortunately.

Tbh, I wish cm2 won't remain vendor tied to Nvidia and other vendors adopt it. cm2 has stuffs cm1 doesn't.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[codecnotsupported]

Wondering about this as well, anyone currently working on it?

[0cc4m]

This isn't up to us. The VK_KHR_cooperative_matrix implementation is there, but disabled on Intel because it performs worse than without it. mesa or Intel need to look into the driver implementation and optimize it or let us know that we need to treat Intel in some other way than AMD and Nvidia, where it works as expected.

[codecnotsupported]

This isn't up to us. The VK_KHR_cooperative_matrix implementation is there, but disabled on Intel because it performs worse than without it. mesa or Intel need to look into the driver implementation and optimize it or let us know that we need to treat Intel in some other way than AMD and Nvidia, where it works as expected.

SYCL supports XMX (Intel Arc's matrix core)
https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/experimental/sycl_ext_matrix/sycl_ext_oneapi_matrix.asciidoc

Wouldn't it be possible to change the SYCL backend to support XMX?

[0cc4m]

This issue is about Vulkan, I don't know much about SYCL or the SYCL backend.