Skip to content

feat(localai): select the accelerator image variant during setup - #171

Merged
marko-boehm merged 2 commits into
releases/1.0.0from
feat/localai-gpu-variant
Aug 3, 2026
Merged

feat(localai): select the accelerator image variant during setup#171
marko-boehm merged 2 commits into
releases/1.0.0from
feat/localai-gpu-variant

Conversation

@marko-boehm

@marko-boehm marko-boehm commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

LocalAI was pinned to the CPU image on every host, with no GPU devices passed through. Setup
now detects what the host can run — NVIDIA (CUDA), AMD (ROCm), Intel (SYCL), Vulkan — and asks
which image to install, listing the driver prerequisites before the choice. Also raises the
LocalAI pin to v4.7.1.

Only the variant is stored (LOCALAI_IMAGE_VARIANT). The version stays pinned in
src/ai/localai/docker-compose.yml and the accelerator tag is recomposed from it on every
render, so papaia-ctl upgrade moves GPU installs forward exactly like CPU ones — storing a
full tag would have frozen the LocalAI version in the config dir via sticky reuse.

The image swap and device passthrough land in a generated
overrides/docker-compose.localai-gpu.override.yml, following the write-or-delete idiom of
generate_ssl_cert_override; the core compose file is never mutated. Selecting CPU deletes the
override again.

Three deliberate design points:

  • Every variant stays selectable, whatever detection found. Detection drives the label and
    the preselected default only — an operator may configure the stack before installing the
    driver or container runtime. Choosing an undetected variant prints a warning.
  • Detection and Compose emission are separate. Detection runs once, at setup time, and may
    fork nvidia-smi. The override generator runs on every render (and therefore on every
    papaia-ctl start) and is subprocess-free.
  • The probe is announced before it runs. It shells out to nvidia-smi and docker info,
    which is slow enough on a cold host that a silent wait reads as a hang.

Vulkan is the one variant whose device set is not fixed — upstream documents a different set
per vendor — so it emits the union of the device nodes actually present.

The wizard step looks like this:

LocalAI Image
  Detecting available GPU accelerators, this can take a few seconds ...

  The GPU images need the matching host prerequisites to be in place already:
    NVIDIA  proprietary driver + NVIDIA Container Toolkit
    AMD     ROCm kernel driver (provides /dev/kfd)
    Intel   none - the backend ships its own driver
    Vulkan  a Vulkan-capable GPU driver on the host
  The stack does not install any of these. Detection below reports only what
  this host exposes right now.

  1) NVIDIA GPU (CUDA)    NVIDIA GeForce RTX 4060 Laptop GPU, driver 610.88, CUDA 13  (recommended)
  2) AMD GPU (ROCm)       no AMD GPU detected
  3) Intel GPU (SYCL)     no Intel GPU detected
  4) Vulkan (generic)     no render node under /dev/dri
  5) CPU only             runs on any hardware
  Choose [1]:

Linked issues

Closes #169
Closes #170

Type of change

  • New feature (feat:)

Test plan

  • python -m pytest tests — 441 passed (25 new: detection, override generation, resolve
    pass, sticky defaults)
  • shellcheck --severity=warning tools/papaia-ctl tools/lib/sh/*.sh — clean
  • yamllint . — clean
  • Every wizard branch (1–5, word aliases, unrecognized input) exercised under
    set -euo pipefail; all return 0 and the expected variant
  • Live detection on an NVIDIA host: NVIDIA GeForce RTX 4060 Laptop GPU, CUDA 13
    recommends nvidia-cuda-13
  • End-to-end setup -y --localai-variant=autoLOCALAI_IMAGE_VARIANT=nvidia-cuda-13
    plus an override carrying localai/localai:v4.7.1-gpu-nvidia-cuda-13
  • Re-run without the flag → variant and override survive (sticky no-op)
  • Switch to intel → tag and /dev/dri update; switch to cpu → override removed
  • --localai-variant=bogus → exits 2 with the valid values listed
  • docker compose config against base + override resolves the expected image, devices,
    group_add and the NVIDIA reservation
  • Upgrade path: the override inherits whatever version the compose file pins (fixture pins
    v9.9.9, asserted in the tests)
  • Running stack on the CUDA 13 image: container healthy, OIDC login through Keycloak
    succeeds

Not covered: an inference run on AMD, Intel or Vulkan hardware — none available. For those
three the resolved docker compose config is the evidence.

Checklist

  • Title follows Conventional Commits
  • Documentation updated (README, deployment, configuration, troubleshooting, .env.example,
    models.txt, THIRD_PARTY_LICENSES.md; the dead LOCALAI_IMAGE hint is gone)
  • No secrets, credentials, or .env files committed
  • Self-reviewed the diff

Detect NVIDIA, AMD, Intel and Vulkan capability on the host and offer the
matching LocalAI image, instead of always installing the CPU one. Bump
LocalAI to v4.7.1.
The detection shells out to nvidia-smi and docker info. Running it silently
right after the LocalAI URL prompt reads as a hang.
@marko-boehm
marko-boehm merged commit b1d03ec into releases/1.0.0 Aug 3, 2026
6 checks passed
@marko-boehm
marko-boehm deleted the feat/localai-gpu-variant branch August 3, 2026 14:38
marko-boehm added a commit that referenced this pull request Aug 3, 2026
## Summary

Completes the curated `[1.0.0]` section in `CHANGELOG.md` before the
milestone is
integrated into `main`. Two gaps remained after #168 wrote the section:
the release
date was still a placeholder, and #171 was not represented at all —
neither the
accelerator-variant selection during setup nor the LocalAI bump that
came with it.

## Linked issues

None — release-artefact maintenance for milestone "v1.0.0 – Lean Core &
Extension Platform".

## Type of change
- [ ] Bug fix (`fix:`)
- [ ] New feature (`feat:`)
- [x] Documentation (`docs:`)
- [ ] Refactor / chore (`refactor:` / `chore:`)
- [ ] CI / build (`ci:` / `build:`)
- [ ] Breaking change

## What changed

- `## [1.0.0] - 2026-07-26` -> `## [1.0.0] - 2026-08-03` (actual
release-cut date)
- Features: added the LocalAI accelerator image variant (CPU / NVIDIA /
AMD / Intel / Vulkan) selected during setup and proposed by a host
hardware probe
- Maintenance: added the LocalAI bump to `v4.7.1`

The section's existing curation is kept as is: it lists user-visible
features,
maintenance and documentation only. Fixes to features that never shipped
outside
the milestone stay out.

## Test plan

- [x] `[1.0.0]` section now covers every user-visible change on
`releases/1.0.0` up to `b1d03ec`
- [x] Entries verified against the branch:
`src/ai/localai/docker-compose.yml` pins `localai/localai:v4.7.1`; the
variant selection lives in `tools/lib/gpu_detect.py` and
`tools/lib/sh/setup.sh`
- [x] Keep-a-Changelog structure and heading order unchanged
- [x] Documentation-only change, no code or configuration touched

## Checklist
- [x] Title follows [Conventional
Commits](https://www.conventionalcommits.org/) (e.g. `feat: …`,
`fix(scope): …`)
- [x] Documentation updated (README / docs / CHANGELOG) where applicable
- [x] No secrets, credentials, or `.env` files committed
- [x] Self-reviewed the diff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: bump LocalAI to v4.7.1 [Feature]: select the LocalAI image variant (CPU / NVIDIA / AMD / Intel / Vulkan) during setup

1 participant