Add Linux NPU & GPU support to Lemonade Server

[jeremyfowers]

re-post of onnx/turnkeyml#305

Adding a discussion issue here to get feedback from the community about what they would use Linux support for in conjunction with Lemonade Server.

Something that would help would be if people would comment with what their use case is, what hardware they are running, what models they are interested in, etc. Having this written here would give us some concrete targets to go after.

[GreyXor]

Native NPU support in Linux would be a game changer for us.

• Linux XDNA NPU Support EricLBuehler/mistral.rs#1254
• NPU Linux support amd/gaia#9
• AMD Ryzen NPU support ollama/ollama#5186
• Running Your Model on XDNA under Linux AMD-AGI/Instella#1
• [Feature request] Any plans for AMD XDNA AI Engine support on Ryzen 7x40 processors? ggml-org/llama.cpp#1499

[ckuethe]

Quoting myself in onnx/turnkeyml#305 (comment)

At this moment, I'm using a ryzen cpu, 7900xtx, Ubuntu 24LTS, ROCm 6.3.3.60303-74~24.04, and whatever the latest versions of ollama and openwebui are. Docker does a nice job of keeping my machine from getting to cluttered.

models used? whatever seems interesting/useful for text processing (eg. code generation, document summaries, etc.), is easily imported into ollama, and fits in VRAM, eg.

• codegemma
• codellama
• codestral
• deepseek-r1
• deepcoder
• starcoder
• qwen-2.5-coder
• cogito
• phi4
• falcon3
• gemma3
• granite3.2
• granite-code
• dolphin-mixtral

Ollama 0.7.0 gained support for vision models which I've started playing with. Gemma, llava, qwen2.5vl, and granite3.2-vision all say fairly accurate things about some test pictures; llama3.2-vision repeatedly crashed.

I'm also up to ROCm 6.3.4.

[dennemark]

I am planning to use AMD Max+ 395 Strix Halo - Framework Desktop - with i.e. Ollama. And it would be a bummer not to habe Lemonade support in Linux. I would even cancel my preorder if Lemonade does not work, since it seems the performance of the APU mixht not be good enough for personal local LLM.

Thanks for opening the discussoon! :)

[jakespillstea]

I am also planning to use the Ryzen AI Max+ 395 in the Framework Desktop running Linux. I'm currently running Ollama or KoboldAI using CPU inference or RunPod.io. I would be using it for story writing, which frequently needs high context length, or basic chat use like organizing information or thoughts.

I frequently use the models fine-tuned by TheDrummer on HuggingFace. Having Lemonade support in Linux would be a game changer for energy-efficient, open source, and locally run inference in the community.

As an option, having the Lemonade server packaged & distributed in an OCI image (Docker) would make it incredibly portable and easy to deploy for Linux users.

Thank you!

[Delivator]

Bump! Just got my Ryzen AI 9 HX 370 notebook and was very excited to run local LLMs on the NPU but couldn't figure out how yet.

[CleverLittleMaker]

I have a Ryzen AI Max+ 395 (128GB) running Linux Ubuntu 25.04 currently.

My application is currently implemented as an interruptible Voice Chat assistant with tool use, vision, and MCP using Ollama. The goal is to develop embodied AI (i.e. running the LLMA to control a robotic 'body' at a high level). Unfortunately, this project is Linux only. Windows is not an option.

Since Ollama does not currently support the Ryzen AI Max+ 395 APU (because of the state of ROCm) I am open to other options such as lemonade if I can make them work for me.

[jeremyfowers]

Hi folks, I am glad to finally have an update on this front: some Linux support is coming in the next major release of Lemonade, v8.0.0, which should be available next week.

It will have Ubuntu support in Lemonade Server for Ryzen and Radeon GPUs via the llamacpp Vulkan backend. The NPU support will take a while longer to materialize, but the Vulkan backend is quite powerful (I use it a lot these days) so I hope that will help to tide people over.

[CleverLittleMaker]

Hi folks, I am glad to finally have an update on this front: some Linux support is coming in the next major release of Lemonade, v8.0.0, which should be available next week.

That's good news. :)

It will have Ubuntu support in Lemonade Server for Ryzen and Radeon GPUs via the llamacpp Vulkan backend. The NPU support will take a while longer to materialize, but the Vulkan backend is quite powerful (I use it a lot these days) so I hope that will help to tide people over.

A couple questions:

1. After I posted my comment 3 weeks ago, I was able tot get ollama to run GPU acceleratetd on my Strix Halo system using the AMD developer driver downloads. This is using ROCM not Vulkan (as far as I can tell). Is AMD not able to help you guys get Lemonade running using ROCM under Linux (since AMD is supporting this project)?

2. I was also under the impression that the necessary kernel level driver to support the NPU was included in the Linux 6.14 Kernel. Is this correct? If so, I hope this makes NPU support easy to implement.

Anyway, excitetd to see Lemonade Linux support. Keep up the good work!

Cheers from New Zealand. :)

[jeremyfowers]

Hey @CleverLittleMaker, its very cool to hear from New Zealand!

Re: ROCm support: we’re planning on doing ROCm after Vulkan. We just did Vulkan first because it was pretty easy, and we wanted to get GPU-only and Linux support out there ASAP!

Re: NPU on Linux: the upcoming Ryzen AI 1.5.0 release is supposed to have stronger LLMs-on-Linux support, which may enable us to support NPU-only on Lemonade on Linux. However, it will take longer to enable Hybrid Execution (NPU for prompt processing + GPU for token generation) because the GPU half of Hybrid is based on Microsoft DirectML, which is Windows-only. It will take the team more time to migrate off of DirectML and on to something cross-platform for Hybrid.

Anyway, excitetd to see Lemonade Linux support. Keep up the good work!

Cheers from Washington, DC!

[GreyXor]

Hey @CleverLittleMaker, its very cool to hear from New Zealand!

Re: ROCm support: we’re planning on doing ROCm after Vulkan. We just did Vulkan first because it was pretty easy, and we wanted to get GPU-only and Linux support out there ASAP!

Re: NPU on Linux: the upcoming Ryzen AI 1.5.0 release is supposed to have stronger LLMs-on-Linux support, which may enable us to support NPU-only on Lemonade on Linux. However, it will take longer to enable Hybrid Execution (NPU for prompt processing + GPU for token generation) because the GPU half of Hybrid is based on Microsoft DirectML, which is Windows-only. It will take the team more time to migrate off of DirectML and on to something cross-platform for Hybrid.

Anyway, excitetd to see Lemonade Linux support. Keep up the good work!

Cheers from Washington, DC!

Awesome news, looking forward for Ryzen AI 1.5.0

[RoyLi-Pvt]

Don't forget to support ROCm in WSL2~~~

[jeremyfowers]

Lemonade Server v8 just released with support for GPU acceleration in Ubuntu: https://pypi.org/project/lemonade-sdk/8.0.0/

The front README of the repo now has this compatibility table as well:

With regards to adding more supported configurations:

• NPU support in Linux and llama.cpp are a work in progress within AMD. When that support is published in the upstream libraries we will add it to Lemonade Server.
• The Lemonade team is planning to start work on ROCm support soon @RoyLi-Pvt.

I would love to get y'all's feedback on Lemonade Server v8 on Ubuntu!