Ollama support for AMD GPUs #692
Description
Describe the bug
The current implementation of .WithGPUSupport(OllamaGpuVendor.AMD) is insufficient.
The code currently applies .WithContainerRuntimeArgs("--device", "/dev/kfd"), but the section of the ollama documentation for AMD GPUs specifies two --device options. You need --device /dev/dri too.
Moreover the default tags won't use the AMD GPU. You need to use the rocm tag, or one of the version specific rocm tags, such as 0.6.0-rocm or 0.7.0-rc1-rocm
Regression
No response
Steps to reproduce
1. Install the dotnet AI template preview 2.
2. Create a new project with this template using the aspire option
3. Modify the AppHost/Program.cs to add GPU support
var ollama = builder.AddOllama("ollama")
.WithGPUSupport(OllamaGpuVendor.AMD)
...
4. Measure the speed.
To do so:
1. Start the project
2. Run `docker ps` to find the container
3. Then run `docker exec -it {CONTAINER_NAME} ollama run {MODEL_NAME} "hi" --verbose --keepalive 0m`
5. Modify the AppHost Program.cs again
var ollama = builder.AddOllama("ollama")
.WithGPUSupport(OllamaGpuVendor.AMD)
.WithContainerRuntimeArgs("--device", "/dev/dri")
.WithImageTag("0.6.0-rocm") // for a fair comparison since the default tag is 0.6.0 in this version
...
6. Measure the speed again
Expected behavior
Both should run at the same speed.
Screenshots
No response
IDE and version
Other
IDE version
VSCode Version: 1.100.2 Commit: 848b80aeb52026648a8ff9f7c45a9b0a80641e2e Date: 2025-05-14T21:47:40.416Z Electron: 34.5.1 ElectronBuildId: 11369351 Chromium: 132.0.6834.210 Node.js: 20.19.0 V8: 13.2.152.41-electron.0 OS: Linux x64 6.11.0-19-generic
Nuget packages
CommunityToolkit.Aspire.Hosting.Ollama (9.4.1-beta.289)
Additional context
Given that the tag is a combination of the version number and optionally -rocm, the best approach is far from obvious.
In the meantime the following extension method works for me by appending -rocm to the existing tag. A disadvantage is that you have to specify it after any call to .WithImageTag("...") or remember to add it manually.
public static IResourceBuilder<OllamaResource> WithAMDGPUSupport(this IResourceBuilder<OllamaResource> builder)
{
if (builder.Resource.TryGetLastAnnotation<ContainerImageAnnotation>(out var containerAnnotation))
{
if (containerAnnotation.Tag?.EndsWith("rocm") == false)
{
containerAnnotation.Tag += "-rocm";
}
}
return builder.WithContainerRuntimeArgs("--device", "/dev/kfd", "--device", "/dev/dri");
}Help us help you
Yes, but only if others can assist