Skip to content

Llama.cpp not working with intel ARC 770? #7042

Closed
@SergioVargasRamirez

Description

@SergioVargasRamirez

Hi,

I am trying to get llama.cpp to work on a workstation with one ARC 770 Intel GPU but somehow whenever I try to use the GPU, llama.cpp does something (I see the GPU being used for computation using intel_gpu_top) for 30 seconds or so and then just hang there, using 100% CPU (but only one core) as if it would be waiting for something to happen...

I am using the following command:

ZES_ENABLE_SYSMAN=0 ./main -m ~/LLModels/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_S.gguf -p "Alice lived in Wonderland and her favorite food was:" -n 512 -e -ngl 33 -sm none -mg 0 -t 32

it doesn't matter if I ignore the ZES_ENABLE_SYSMAN part.

now, the same biuld does work when -ngl 0. There I see the 32 cores be used and the model produces output.

If I run clinfo, I get the following output.

Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM)
 `-- Device #0: Intel(R) FPGA Emulation Device
Platform #1: Intel(R) OpenCL
 `-- Device #0: AMD Ryzen 9 5950X 16-Core Processor
Platform #2: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) Arc(TM) A770 Graphics

sycl-ls:

[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2  [2024.17.3.0.08_160000]
[opencl:cpu:1] Intel(R) OpenCL, AMD Ryzen 9 5950X 16-Core Processor             OpenCL 3.0 (Build 0) [2024.17.3.0.08_160000]
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO  [24.18.0]

ls-sycl-device sees three SYCL devices on is the GPU.

found 3 SYCL devices:
|  |                  |                                             |Compute   |Max compute|Max work|Max sub|               |
|ID|       Device Type|                                         Name|capability|units      |group   |group  |Global mem size|
|--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------|
| 0|    [opencl:gpu:0]|               Intel(R) Arc(TM) A770 Graphics|       3.0|        512|    1024|     32|    16225243136|
| 1|    [opencl:cpu:0]|AMD Ryzen 9 5950X 16-Core Processor            |       3.0|         32|    8192|     64|   134983352320|
| 2|    [opencl:acc:0]|               Intel(R) FPGA Emulation Device|       1.2|         32|67108864|     64|   134983352320|

Now I don't see the level-zero device, I had it at some point but had no opencl:gpu in exchange. With the level-zero device I had the same problem. The GPU will activate for 30s and go back to zero activity while ./main stays on guard for hours if I don't cancel.

I a running OpenSuse Tumbleweed and installed intel oneAPI locally using the online installer. I don't see compilation issues. I also compiled Neo and its requirements. All these packages are in my home but that doesn't seem to be the issue because previously installed intel packages (via zypper) where avail. system-wide with the same results.

I am really lost here because I don't seem to be getting any error, I am sure the GPU crashes but I don't know why or where to look for this info. So I would really appreciate your help on this. I can test anything you want (this is not a production system or else).

thanks in advance and best regards,

Sergio

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions