Skip to content

fix(local-ai): enable NVIDIA GPU passthrough - #8462

Draft
bitsandbots wants to merge 1 commit into
nextcloud:mainfrom
bitsandbots:fix/local-ai-enable-nvidia-gpu-v2
Draft

fix(local-ai): enable NVIDIA GPU passthrough#8462
bitsandbots wants to merge 1 commit into
nextcloud:mainfrom
bitsandbots:fix/local-ai-enable-nvidia-gpu-v2

Conversation

@bitsandbots

Copy link
Copy Markdown

Summary

  • The local-ai community container only requests /dev/dri, which is the Mesa/AMD/Intel Vulkan device passthrough mechanism.
  • On NVIDIA-only hosts (no Mesa-compatible device), Vulkan has nothing to bind to and every backend (llama-cpp, stablediffusion-ggml, whisper) silently falls back to llvmpipe (CPU-only software rendering).
  • This makes image generation in particular unusably slow — CPU-based Stable Diffusion routinely took 2+ minutes per image in my testing, exceeding integration_openai's image_request_timeout (240s default) and causing Assistant sticker/image tasks to fail outright.
  • plex, jellyfin, and memories already solve this for their own GPU-accelerated use cases via "enable_nvidia_gpu": true, which DockerActionManager turns into Runtime: nvidia + the correct DeviceRequests when the host operator has set NEXTCLOUD_ENABLE_NVIDIA_GPU=true on the mastercontainer. local-ai.json was missing this flag.

Change

  • Add "enable_nvidia_gpu": true to community-containers/local-ai/local-ai.json, alongside the existing "devices": ["/dev/dri"] (same pattern as memories.json, so Mesa-only hosts are unaffected and NVIDIA hosts now get real acceleration).

Testing

Reproduced and fixed on an NVIDIA RTX 5070 host:

  • Before: vulkaninfo inside the container showed only llvmpipe (PHYSICAL_DEVICE_TYPE_CPU); a 512x512 SD1.5 image generation took 140s+ (cold) and still exceeded the timeout under normal load.
  • After (host operator sets NEXTCLOUD_ENABLE_NVIDIA_GPU=true on the mastercontainer + has nvidia-container-toolkit installed/configured): vulkaninfo correctly reports the NVIDIA GeForce RTX 5070 (PHYSICAL_DEVICE_TYPE_DISCRETE_GPU), and the same image generation completed in 4.5s once the backend was warm, with nvidia-smi confirming the stablediffusion-ggml process resident on the GPU.
  • Verified local-ai.json still validates against php/containers-schema.json (the enable_nvidia_gpu boolean property already exists in the schema, used by the other GPU-enabled community containers).

Notes

  • This is opt-in: nothing changes unless the host operator explicitly sets NEXTCLOUD_ENABLE_NVIDIA_GPU=true on the mastercontainer (per the existing hardware-acceleration docs in the main readme) and has the NVIDIA Container Toolkit installed. Mesa/AMD/Intel-only hosts are unaffected since /dev/dri passthrough is untouched.

🤖 Drafted with Claude Code

@bitsandbots
bitsandbots marked this pull request as ready for review July 17, 2026 18:57
@bitsandbots
bitsandbots force-pushed the fix/local-ai-enable-nvidia-gpu-v2 branch from 1bfb96e to fe0a1d6 Compare July 17, 2026 19:11
The local-ai community container only requests /dev/dri, which is
the Mesa/AMD/Intel Vulkan mechanism. On NVIDIA-only hosts this leaves
Vulkan with no usable device, so every backend silently falls back to
llvmpipe (CPU), causing image generation to take minutes and blow
past integration_openai's request timeout.

Add enable_nvidia_gpu, matching the plex, jellyfin, and memories
community containers, so AIO sets Runtime=nvidia and the proper
DeviceRequests when the host operator has NEXTCLOUD_ENABLE_NVIDIA_GPU
set. /dev/dri stays for Mesa-only hosts, matching memories' pattern
of combining both.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Cory <bitsandbots@gmail.com>
@bitsandbots
bitsandbots force-pushed the fix/local-ai-enable-nvidia-gpu-v2 branch from fe0a1d6 to 0dfefd2 Compare July 17, 2026 19:17
@szaimen

szaimen commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

@bitsandbots did you test and verified that this works locally?

@szaimen szaimen added 2. developing Work in progress enhancement New feature or request labels Jul 31, 2026
@szaimen szaimen modified the milestones: next, v13.5.0 Jul 31, 2026
@szaimen szaimen removed this from the next milestone Aug 18, 2026
@szaimen
szaimen marked this pull request as draft August 22, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2. developing Work in progress enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants