v4.8.2

What's Changed

👒 Dependencies

• chore(deps): bump actions/stale from 10.4.0 to 11.0.0 by @dependabot[bot] in #11395
• chore(deps): bump actions/checkout from 4 to 7 by @dependabot[bot] in #11396

Other Changes

• feat(gallery): fall back to mirrors and a cached index when the primary source fails by @localai-bot in #11389
• chore(model-gallery): ⬆️ update checksum by @localai-bot in #11405
• docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #11397
• feat(swagger): update swagger by @localai-bot in #11398
• feat(gallery): default to index.localai.io with GitHub as a mirror by @localai-bot in #11409
• feat(nemo-speech-cpp): add the NVIDIA NeMo-Speech.cpp backend by @localai-bot in #11406
• fix(ci): remove unsupported cosign bundle flag by @localai-bot in #11413

Full Changelog: v4.8.1...v4.8.2

v4.8.1

What's Changed

Other Changes

• docs(blog): cover the terminal agent in the 4.8 post by @localai-bot in #11372
• fix(vram): contain malformed GGUF metadata by @richiejp in #11374
• docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #11377
• chore: ⬆️ Update 0xShug0/audio.cpp to 7efbb58def443722ea540d931dd3debee3e4d5e8 by @localai-bot in #11378
• chore: ⬆️ Update ikawrakow/ik_llama.cpp to cf1aa57e1a0fabfd015831718fc99d1aec01ada5 by @localai-bot in #11380
• chore: ⬆️ Update leejet/stable-diffusion.cpp to c6beeef35526c6dc94b74a7fb69f9d2e6a2a7a12 by @localai-bot in #11384
• chore: ⬆️ Update CrispStrobe/CrispASR to 21901d3f7c23554f072964828363e49ddbc2dc68 by @localai-bot in #11383
• fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again by @localai-bot in #11386
• fix(react-ui): stop traces page crash when switching trace tabs by @nandanadileep in #11387
• chore: ⬆️ Update antirez/ds4 to b0309611041655f4e45671cfd9c9886aff161406 by @localai-bot in #11381
• chore(model-gallery): ⬆️ update checksum by @localai-bot in #11382
• gallery: add Qwen3.5 9B HauhauCS variants by @localai-org-maint-bot in #11339
• gallery: add Qwen3.5 9B Defiant Fable variants by @localai-org-maint-bot in #11335
• feat(vllm-cpp): wire the full engine config surface through engine_args by @localai-bot in #11159
• fix(react-ui): restore 3D Studio results and history by @richiejp in #11393
• fix(cli): ignore a half-populated socket activation environment by @localai-bot in #11394

Full Changelog: v4.8.0...v4.8.1

v4.8.0

🎉 LocalAI 4.8.0 Release! 🚀

LocalAI 4.8.0 is out!

Twenty-two days, 386 pull requests, and three new modalities. This release introduces vllm.cpp, a C++20 engine maintained by the LocalAI team, which began as a vLLM port and now carries its own featureset, shipping as the vllm-cpp backend in alpha development builds. Around it: 3D generation as a new modality, a multi-family audio.cpp engine, gallery entries that install the build your hardware can actually run, and a deep reliability pass on distributed mode driven by production incidents.

Highlights:

• 🚀 vllm.cpp (alpha) - a C++20 engine maintained by the LocalAI team, which began as a vLLM port and keeps vLLM as its reference implementation: V1 serving architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) with no Python, PyTorch or ggml at inference. Measured at 1.045x vLLM on Qwen3.6-27B NVFP4 at concurrency 1, with token-for-token identical output. Loads safetensors and GGUF, enforces structured output in-engine, and runs on CPU, CUDA, Metal and Vulkan. The Apple Silicon build ships the MLX GEMM provider, measured at 1.5x to 2.2x on an M4. Shipping as alpha development builds: try it, do not depend on it.
• 🧊 3D generation - a new modality end to end: Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations, the trellis2cpp image-to-3D backend, and a UI page with a native GLB viewer and print remeshing.
• 🔊 audio.cpp - one backend process serving six audio endpoints across many model families, picked from the GGUF's own metadata: speech, transcription, VAD, diarization, source separation and sound generation.
• 🎛️ One model, many builds - a gallery entry can declare variants:, and LocalAI installs the largest build that your host can actually run. No more hunting through the gallery for the right quantization.
• ⚡ A much lighter web UI - gzip on the wire, immutable caching for hashed assets, and paginated trace endpoints: the React bundle is 3.48x smaller and the trace poll dropped from 21 MB to 7 KB.
• 📊 An Activity page - the stacked operations bar collapses to one line, and a new admin Activity page keeps the record of what installed, failed or was cancelled, instead of dropping it the moment it finished.
• 📦 Hugging Face artifact materialization - immutable snapshot resolution, authenticated downloads with real progress, and staged artifacts that remote workers can bind to.
• 🎚️ VRAM budgets - cap how much of a card LocalAI may use, per node, as a percentage (80%) or an absolute amount (12GB).
• 🗣️ Two new TTS engines - magpie-tts-cpp (NVIDIA Magpie Multilingual, 5 voices, 9+ languages) and moss-tts-cpp (48 kHz stereo with reference-audio voice cloning).
• 🌳 Sub-2-bit models - a new bonsai backend serves the 1-bit and ternary Bonsai quantizations of Qwen3 and Qwen3.6-27B.
• 🖧 Distributed mode hardening - the reaper no longer deletes rows for backends that are alive and busy, phantom replicas are cleaned up, and in_flight counters stop leaking.

Plus a Valkey vector store, systemd socket activation, persistent trace history, two security fixes, a documentation overhaul aimed squarely at onboarding, and a new localai.io.

---

📊 This release in numbers

Pull requests merged	386
Commits	392
Files changed	1,204 (+152,918 / -42,012)
Development window	22 days (2026-07-14 to 2026-08-05)
Contributors	25, of whom 11 first-time
New backends	7 (vllm-cpp, audio-cpp, trellis2cpp, valkey-store, bonsai, magpie-tts-cpp, moss-tts-cpp)
Gallery entries	1,221 to 1,515 (+294)

Where the work landed:

Area	Change
core/	+54,974 / -8,291 across 546 files
gallery/	+39,193 / -29,119 (variant ladders rewrite most of the index)
backend/	+25,668 / -1,305 across 256 files
pkg/	+9,678 / -385
.github/	+7,526 / -61
docs/	+4,285 / -2,461 (near-flat by design: the dedup pass removed as much as it added)
website/	+4,479 (new project site)
scripts/	+3,373 / -175

---

📌 TL;DR

Area	Summary
🚀 vllm.cpp (alpha)	An Apache-2.0 C++20 engine maintained by the LocalAI team and developed in its own repository, which began as a vLLM port, shipping as alpha development builds of the vllm-cpp backend over its stable C ABI v5. It uses vLLM as its reference implementation and benchmark, and implements vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) with no Python, PyTorch or ggml at inference. Safetensors + GGUF, in-engine structured output (JSON schema / regex / choice / GBNF). Chat and tool calling ride the llama.cpp autoparser path: full minja templates, tool_choice: auto as a lazy structural-tag constraint, 30 tool dialects and 7 reasoning parsers, streamed ChatDelta/ToolCallDelta. CPU amd64/arm64, CUDA 12/13 (Blackwell), L4T, Vulkan and Darwin Metal, the last with the MLX GEMM provider vendored in (1.5x to 2.2x on an M4).
🧊 3D generation	A new modality, wired end to end: Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations, the trellis2cpp image-to-3D backend over TRELLIS.2, and a UI page with a native GLB viewer, IndexedDB history and previewable print remeshing.
🔊 audio.cpp	New native C++ backend over audio.cpp, a multi-family ggml audio engine: one process serves /v1/audio/speech (supertonic, chatterbox, irodori-voicedesign), /v1/audio/transcriptions (citrinet, nemotron, forced-aligner), /v1/audio/vad, /v1/audio/diarize (sortformer), /audio/transform (htdemucs 4-stem separation, voice conversion, speech-to-speech) and /v1/sound-generation. Family comes from the GGUF's own audiocpp.model_spec.family key, so no per-model backend options. 13 gallery entries. CPU, CUDA 12/13, Vulkan, Metal.
📊 Activity page	The stacked operations bar becomes a permanent one-line strip (✕ now hides rather than cancels), with a new admin /app/activity page: in-progress detail with per-node breakdown, a "needs attention" lane with Cancel and Retry, and a bounded 50-entry record of what finished.
🗄️ Valkey vector store	New valkey-store backend adding Valkey Search as a vector store option.
🌐 New localai.io	The site splits into a project site at the root and docs under /docs/, with 214 generated redirect stubs so every published URL keeps working. Adds an engines page driven by YAML, a blog, an ecosystem band and ADOPTERS.md.
🎛️ Gallery variants	An entry may declare variants: referencing other entries. Install-time selection drops builds the host cannot run (IsBackendCompatible) or cannot fit (VRAM, or cgroup-aware RAM on CPU hosts), then picks the largest that fits. Override with variant on POST /models/apply, local-ai models install --variant, the install_model MCP tool, or the UI split-button. GET /api/models?has_variants=true narrows the list. Older clients ignore the key and install the entry as before.
📦 HF artifacts	Immutable snapshot resolution, authenticated downloads with progress, gallery install and preload materialization, runtime binding to staged artifacts, and UI progress reporting. Python backends reuse the Go download path.
⚡ HTTP performance	gzip middleware (--disable-http-compression, --http-compression-min-length), with streaming paths explicitly skipped. /assets/* served immutable, index.html no-cache. /api/traces and /api/backend-traces accept limit/offset/full and summarize by default, with GET /api/traces/{id} for the full record. React bundle 2,815,513 B to 807,918 B; backend-trace poll 21,131,097 B to 7,201 B.
🎚️ VRAM budget	LOCALAI_VRAM_BUDGET=80% or =12GB (also --vram-budget), on local-ai and local-ai worker. Standalone it is a hard per-process cap inherited by context-fit, GGUF warnings and the watchdog; distributed it is a placement ceiling the scheduler respects. Admin override via PUT/DELETE /api/nodes/:id/vram-budget and the set_node_vram_budget MCP tool. Unset means all detected VRAM.
🗣️ magpie-tts-cpp	New Go/purego backend over magpie-tts.cpp, a ggml port of NVIDIA Magpie TTS Multilingual 357M with NanoCodec embedded. 5 voices, 9+ languages, 22.05 kHz mono, one self-contained GGUF.
🗣️ moss-tts-cpp	New Go/purego backend over moss-tts.cpp for MOSS-TTS-Local v1.5. 48 kHz stereo, optional reference-audio voice cloning, no Python at inference.
🌳 bonsai	New backend on the PrismML llama.cpp fork, which is the only decoder for the Q1_0 and Q2_0 quant formats. Eight gallery entries across Bonsai 8B/27B and Ternary-Bonsai 8B/27B, from ~1.15 GB.
🖧 Distributed reliability	A busy backend is no longer reaped: the worker is asked directly over a new models.running subject, and the port-probe fallback distinguishes DeadlineExceeded (busy) from Unavailable (gone), requiring three consecutive misses. Frontend model stubs are dropped when no healthy replica remains, in_flight leaks are closed, and model-load deadlines scale with checkpoint size.
🛡️ Security	Inline GRPO reward code in POST /api/fine-tuning/jobs is refused unless the operator sets LOCALAI_TRL_ALLOW_INLINE_REWARD=true; t...

v4.7.1

What's Changed

Other Changes

• fix(config): only inject llama.cpp serving options on the llama.cpp path by @localai-bot in #10822

Full Changelog: v4.7.0...v4.7.1

v4.7.0

🎉 LocalAI 4.7.0 Release! 🚀

LocalAI 4.7.0 is out!

This release widens what LocalAI can generate and how you drive it: a UI-managed voice cloning library, local video and audio-driven avatar generation, and interleaved reasoning that travels with tool calls. It also lands new audio engines (one-pass diarized transcription, F5-TTS, true streaming TTS) and a batch of reliability fixes across auth, transcription, and the model gallery.

Highlights:

• 🎙️ Managed voice cloning - record or upload a consented reference in the UI, save it as a reusable profile, and reference it from any cloning-capable TTS backend with a stable localai://voice-profiles/<id> URI. No more hand-edited YAML or copying audio into model folders.
• 🎬 Local video & avatars - a new longcat-video backend brings text-to-video, image-to-video, and audio-driven talking-avatar generation, wired into the Studio UI with audio and reference-image controls.
• 🧠 Interleaved thinking with tool calls - an assistant turn can now carry reasoning and tool_calls together and keep the reasoning across the tool-result loop, with a reasoning_content inbound alias and Anthropic thinking block support.
• 🗣️ New audio engines - one-pass diarized transcription (moss-transcribe-cpp), F5-TTS voice cloning in CrispASR, and true streaming TTS in vibevoice-cpp (time-to-first-audio 2.38s vs 39.96s on CPU).
• 🎚️ Auto full context - context_size: -1 runs a model at its full trained context window, read from GGUF metadata per-model, with a VRAM-fit warning.
• 🖥️ Sharper control - pick which GPUs llama.cpp offloads to (devices:), and a new model-load cooldown stops a deterministically-failing model from respawning its backend on every poll and leaking VRAM.

Plus DFlash speculative-decoding gallery models, an OIDC fix for EC/PS/EdDSA-signed tokens, transcription language/translate settings that reach the backend, and the usual set of dependency updates.

---

📌 TL;DR

Area	Summary
🎙️ Voice Library	Admin-managed voice cloning profiles: record/upload consented reference audio in the UI, preview, and reference via localai://voice-profiles/<id>. New GET/POST/DELETE /api/voice-profiles, MCP tools (list/create/delete_voice_profile), and a typed tts.voice_cloning config. Cloning declared as a capability across 12+ TTS backends (self-discovered, no hardcoded backend names).
🎬 LongCat video & avatars	New longcat-video Python backend: text-to-video, image-to-video, and LongCat-Video-Avatar 1.5 audio-driven avatars. Gallery entries longcat-video and longcat-video-avatar-1.5; new known_input_modalities/known_output_modalities config fields; /video endpoint extended with staged audio. CUDA 12/13 x86_64 + CUDA 13 ARM64 images.
🧠 Interleaved thinking	Assistant turns carry reasoning + tool_calls together and preserve reasoning across the tool loop. reasoning_content accepted as an inbound alias; Anthropic Messages local path round-trips thinking blocks (gated on thinking: {type: "enabled"}).
🗣️ moss-transcribe-cpp	New Go backend over the C++/ggml MOSS-Transcribe-Diarize port: joint multi-speaker transcription + diarization + timestamps in one pass. Gallery model moss-transcribe-cpp-0.9b. Offline; 1.6-2.2x faster than reference on CPU, bit-exact on CUDA.
⚡ vibevoice streaming TTS	Real incremental streaming via vv_capi_tts_stream: TTFA 2.38s vs 39.96s batch (~17x) on CPU for VibeVoice-Realtime-0.5B.
🎨 F5-TTS	F5-TTS linked into CrispASR + f5-tts-crispasr gallery model (24kHz, voice cloning via voice:/voice_text: options).
🎚️ context_size: -1	Negative context_size (or LOCALAI_CONTEXT_SIZE=-1) resolves to the model's n_ctx_train from GGUF metadata, with a GPU-only VRAM-fit warning and a safe clamp so no backend ever sees a negative window.
🖥️ llama.cpp device selection	options: [devices:CUDA1,CUDA2] restricts offload to named GPUs (from --list-devices).
🛡️ Model-load cooldown	A failed load enters a cooldown (default 10s, geometric growth capped at 5m) so polling clients get 503 + Retry-After instead of respawning a crashing backend and leaking GPU memory.
🧠 Speculative decoding models	Four Qwen DFlash gallery entries (4B / 9B / 27B / 35B-A3B), each bundling target + drafter, spec_type:draft-dflash.

---

🚀 New Features & Major Enhancements

🎙️ Managed voice cloning profiles (Voice Library)

Record or upload a consented reference, save it as a reusable profile, and reference it from any cloning-capable TTS backend.

Voice cloning becomes a first-class, UI-driven workflow. Record or upload consented reference audio in the React UI, normalize it to WAV, preview it, and save it as a named profile. Reference the profile from the TTS UI or /v1/audio/speech with a stable localai://voice-profiles/<id> URI, no YAML editing and no copying audio into model directories.

• New REST surface: GET /api/voice-profiles, GET /api/voice-profiles/:id/audio, POST /api/voice-profiles (admin), DELETE /api/voice-profiles/:id (admin), surfaced in Swagger, /api/instructions, and the auth capability registry. MCP admin tools list_voice_profiles, create_voice_profile, delete_voice_profile.
• New typed config tts.voice_cloning (bool) opts custom model names in or out of Voice Library compatibility; false also rejects saved profile references with HTTP 400. Request precedence is voice -> tts.voice -> tts.audio_path; existing options still work with no breaking change.
• Compatibility is server-discovered from backend capabilities (the frontend hardcodes no backend names) and offers gallery models to install when none are present. Cloning is declared in the capability registry for vllm-omni, vibevoice-cpp, coqui, pocket-tts, qwen-tts, qwen3-tts-cpp, faster-qwen3-tts, fish-speech, neutts, chatterbox, voxcpm, omnivoice-cpp, and model-dependent crispasr.
• Cross-backend contract: reference WAV + exact transcript via params.ref_text. E2E verified on qwen3-tts-cpp (4.47s reference produced a non-silent 3.44s 24kHz WAV).

🔗 PRs: #10799

🎬 LongCat video and avatar generation

A new longcat-video Python backend brings local video generation to LocalAI: text-to-video, image-to-video, and LongCat-Video-Avatar 1.5 for audio-driven talking avatars. It is wired into the React Studio UI with audio and reference-image controls.

• Gallery entries longcat-video (text+image input, video output) and longcat-video-avatar-1.5 (text+image+audio input, video output). Backend images: CUDA 12 and CUDA 13 x86_64, plus CUDA 13 ARM64 (DGX Spark / NVIDIA ARM64 guidance included).
• Introduces declarative capability metadata: new known_input_modalities / known_output_modalities config fields so generic code discovers what a checkpoint accepts instead of branching on backend or checkpoint names. The HF importer emits matching self-describing recipe metadata.
• The /video endpoint gains staged audio and video-generation parameters, with distributed audio staging (input bounded to 128 MiB). Avatar mode supports multi-segment generation, BF16 or optional INT8 quantization, and distillation settings, documented in a new features/longcat-video.md page.

🔗 PRs: #10792

🧠 Interleaved thinking with tool calls

An assistant turn can now carry reasoning and tool_calls together, and the reasoning survives the tool-result loop across turns. This is uniform, tested, and documented behavior rather than a per-backend accident.

• OpenAI chat messages accept reasoning_content as an inbound alias for the canonical reasoning field (vLLM/DeepSeek/cogito-style clients emit it); emission is unchanged and the canonical field wins when both are present.
• The Anthropic Messages local path round-trips thinking blocks: inbound thinking blocks parse into the message reasoning, and a thinking block is emitted before tool_use on both non-streaming and streaming responses, gated on the request param thinking: {type: "enabled"}. The cloud-proxy passthrough path is untouched.
• Verified live: lfm2.5-8b-a1b and gemma-4-e2b return reasoning plus structured tool_calls in a single turn. A new features/interleaved-thinking.md doc is cross-linked from the model-configuration, text-generation, and functions guides.

🔗 PRs: #10744

🗣️ New audio engines: diarized transcription, F5-TTS, and streaming TTS

Three additions widen the audio surface:

• moss-transcribe-cpp (#10756): a new Go backend that dlopens the C++/ggml MOSS-Transcribe-Diarize port to do joint multi-speaker transcription, diarization, and timestamps in a single offline pass. Gallery model moss-transcribe-cpp-0.9b (default q5_k GGUF). The ggml port is byte-exact to the reference PyTorch and 1.6-2.2x faster on CPU, bit-exact on CUDA (verified on Blackwell / Jetson Thor). Builds across the full Linux matrix plus Darwin/Metal.
• F5-TTS in CrispASR (#10753): links the F5-TTS static runtime into the CrispASR build (SWivid, MIT; a 22-layer DiT flow-matching model with a built-in Vocos vocoder) and ships an f5-tts-crispasr gallery model. Produces 24kHz mono audio and auto-detects as f5-tts (no backend: selector). Voice cloning via `options: [voice:...

v4.6.2

What's Changed

👒 Dependencies

• chore(deps): bump actions/cache from 4 to 6 by @dependabot[bot] in #10704

Other Changes

• refactor: use slices.Contains to simplify code by @weifanglab in #10702
• fix(ci): shard single-arch backend matrix under GitHub's 256-job limit by @localai-bot in #10703
• chore(model gallery): add MiniCPM series models by @ZMXJJ in #10699

New Contributors

• @weifanglab made their first contribution in #10702
• @ZMXJJ made their first contribution in #10699

Full Changelog: v4.6.1...v4.6.2

v4.6.1

What's Changed

Other Changes

• fix(auth): log the real cause of OIDC/OAuth user-info failures by @localai-bot in #10679
• feat(api): add GET /v1/models/capabilities endpoint by @localai-bot in #10687
• chore: ⬆️ Update CrispStrobe/CrispASR to 1109cb3fcae2e242c2b3d42ec0e3fd6e813f2ce7 by @localai-bot in #10685
• chore: ⬆️ Update ggml-org/llama.cpp to 665892536dfb1b7532161e3182304bd35c33e768 by @localai-bot in #10681
• chore(model-gallery): ⬆️ update checksum by @localai-bot in #10686
• chore: ⬆️ Update vllm-metal (darwin) to v0.3.0.dev20260704102955 by @localai-bot in #10668
• fix(ui): center the home empty-state wizard by @localai-bot in #10691
• docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #10680
• chore: ⬆️ Update CrispStrobe/CrispASR to 09df654e304947f7521e1f52992ceacccf03c300 by @localai-bot in #10693
• chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to daedb763fd442e0916eb130a479fdd74947291c0 by @localai-bot in #10682
• chore: ⬆️ Update ServeurpersoCom/qwentts.cpp to 73fe0c67bbf0898ba2999535e0680a02a7f8537d by @localai-bot in #10683
• feat(agents): native Prometheus metrics for agent chat runs by @walcz-de in #10689
• chore: ⬆️ Update ggml-org/llama.cpp to 2da668617612d2df773f966e3b0ee22dc2beef7b by @localai-bot in #10694
• fix(reasoning): don't persist request-scoped reasoning_effort as an operator disable (#10622) by @Anai-Guo in #10623
• fix(startup): scope generated-content and upload dirs to the current user by @localai-bot in #10698
• fix(config): cap auto-derived context to fit VRAM by @localai-bot in #10696
• fix(llama-cpp): cap single-pass embedding batch to fit VRAM by @localai-bot in #10695

Full Changelog: v4.6.0...v4.6.1

v4.6.0

🎉 LocalAI 4.6.0 Release! 🚀

LocalAI 4.6.0 is out!

This is a reliability-focused release: AMD ROCm backends now run on-GPU at full speed, distributed model loads no longer wedge when a worker dies, and realtime sessions warm up predictably. It also brings conversation forking to the built-in chat UI, a Prometheus counter for PII/audit events, and an SSRF fix for the model gallery.

Highlights:

• 🔴 AMD ROCm runs correctly - ggml audio backends offload to the GPU, hipBLASLt kernel-tuning data is bundled (no more slow generic kernels), rocm-vllm installs the right wheel, and the ASIC ID table is found.
• 🎙️ Predictable realtime - sessions eagerly warm the whole pipeline (VAD, ASR, LLM, TTS) up front, so the first turn no longer pays per-model cold-start stalls, plus a new POST /backend/load API and "Load into memory" UI button.
• 🌿 Forking chat - retry any assistant answer, branch a new chat from any point, duplicate, or copy the whole conversation, directly in the built-in UI.
• 🛡️ Distributed hardening - a dead worker can no longer pin the model-load advisory lock (the ~15-minute wedge is gone), and orphaned backend workers self-terminate instead of holding VRAM.
• 📊 PII/audit metrics - PII detections/masks/blocks are exported as a Prometheus counter, so you can alert when the filter stops firing.
• 🔒 Gallery SSRF fix - POST /models/apply config-URL fetches are validated against private/loopback/metadata addresses.

Plus idempotent backend installs, tool-calling and reasoning fixes across the vLLM and Python/MLX backends, cloud-proxy compatibility with the newest reasoning models, and the usual set of dependency updates.

---

📌 TL;DR

Area	Summary
🔴 AMD ROCm reliability	ggml audio backends now compile with -DGGML_HIP=ON and link HIP (real GPU offload); hipBLASLt TensileLibrary data bundled + HIPBLASLT_TENSILE_LIBPATH exported; rocm-vllm installs from the AMD wheel index on Python 3.12; amdgpu.ids symlinked so the ASIC table is found.
🎙️ Realtime warm-up + load API	Sessions block-warm the full pipeline at start (errors surface up front); new POST /backend/load / POST /v1/backend/load, a "Load into memory" UI action, and a load_model MCP tool. Opt out per pipeline with disable_warmup: true.
🌿 Forking chat	Regenerate any assistant answer (not just the last), branch a new chat from any turn, duplicate a chat, or copy it as Markdown - all client-side in the React UI.
🛡️ Process & distributed lifecycle	A dead worker no longer pins the per-model PostgreSQL advisory lock (bounded load ceiling + context-scoped lock_timeout); backend workers self-terminate on parent death (LOCALAI_BACKEND_PARENT_WATCH); the watchdog stops logging optional Free() as an error.
⚙️ Idempotent backend installs	POST /backends/apply and the LOCALAI_EXTERNAL_BACKENDS boot loop no longer re-pull an already-installed backend unless force: true.
📊 PII/audit Prometheus counter	localai_pii_events_total{kind,origin,action,direction} on /metrics, complementing the /api/pii/events ring buffer.
🔒 Gallery SSRF hardening	Gallery config URL fetches run through ValidateExternalURL, blocking private, loopback, link-local, and cloud-metadata addresses.
🧩 Tool-calling & reasoning fixes	Non-streaming vLLM tool calls restored; MLX/Python backends decode tool-call arguments for chat templates and split closing-only </think> reasoning blocks.

---

🚀 New Features & Major Enhancements

🔴 AMD ROCm backends run correctly on-GPU

Four coupled fixes make ROCm/hipBLAS backends actually run on AMD hardware, and at full speed, instead of silently falling back to CPU or slow generic kernels:

• GPU offload for ggml audio backends (#10667): rocm-qwen3-tts-cpp, rocm-omnivoice-cpp, acestep-cpp, and vibevoice-cpp were building CPU-only because their Makefiles passed the no-op -DGGML_HIPBLAS=ON (upstream ggml only understands -DGGML_HIP=ON) and the CMake link loop omitted hip. They now use the same hipblas recipe as llama-cpp and link the HIP backend.
• hipBLASLt kernel-tuning data (#10660, #10672): the packager bundled rocBLAS data but not the parallel hipBLASLt TensileLibrary_lazy_gfx*.dat files, so every arch silently used slow kernels and logged Cannot read "TensileLibrary_lazy_gfx*.dat". The data is now bundled and HIPBLASLT_TENSILE_LIBPATH is exported by the llama-cpp and turboquant run.sh.
• rocm-vllm installs the right wheel (#10642, #10651): the backend was pulling the CUDA-only PyPI vllm (fatal ModuleNotFoundError: No module named 'vllm' on AMD). It now pins CPython 3.12 and installs vLLM from the ROCm wheel index (https://wheels.vllm.ai/rocm/).
• ASIC ID table found (#10624, #10627): the compute-only hipblas image lacks /opt/amdgpu/share/libdrm/amdgpu.ids, so every model load warned. Ubuntu's libdrm-common copy is now symlinked into place.

🔗 PRs: #10667, #10672, #10651, #10627

🎙️ Realtime: eager pipeline warm-up + a load-into-memory API

Realtime voice sessions now eagerly and blockingly warm the entire pipeline (VAD, transcription, LLM, TTS, sound detection, voice recognition) at session start instead of lazy-loading each sub-model on first use. The first turn no longer pays per-model cold-start stalls, and model-load errors surface up front at session start (as model_load_error) rather than mid-stream. Pipeline sub-models load concurrently, so a session warms in the time of its slowest stage, not the sum, and a failed stage names every broken model in a joined error.

This also adds a LocalAI-native POST /backend/load (and /v1/backend/load), the inverse of /backend/shutdown, exposed as a "Load into memory" UI action and a load_model MCP admin tool, so admins can pre-warm any model (including full pipelines) on demand. The --load-to-memory startup flag now routes through the same engine. Opt out per pipeline with disable_warmup: true.

🔗 PRs: #10662

🌿 Forking chat in the built-in UI

The React chat UI gains conversation-management tools: regenerate any assistant answer (not just the last), branch a new chat from any answer, duplicate a chat into an independent copy, or copy the whole conversation to the clipboard as Markdown. Retrying a mid-conversation answer correctly truncates the conversation before re-asking, both in the DOM and in the request payload (this also fixes a latent stale-closure bug where a mid-conversation retry sent the downstream turns back to the model). All client-side, no backend changes.

🔗 PRs: #10654

🛡️ Sturdier process and distributed lifecycle

• Dead-worker advisory-lock wedge (#10600): a distributed worker going mid-load could pin a per-model PostgreSQL advisory lock and fail every subsequent request to that model with 55P03 for ~15 minutes. The detached load context is now bounded by a model-load ceiling, the install wait honors cancellation via singleflight.DoChan, and lock_timeout is scoped to the caller's context budget instead of a deployment-global GUC.
• Parent-death safety net (#10639): if LocalAI is SIGKILLed before teardown, spawned backend workers used to get reparented to init and linger, holding VRAM and their port. Each backend now polls its parent PID and self-terminates on reparenting. Configurable via LOCALAI_BACKEND_PARENT_WATCH (default on, auto-off on Windows) and LOCALAI_BACKEND_PARENT_WATCH_INTERVAL (default 2s). C++ coverage is llama-cpp for now; Python covers all backends.
• Quieter watchdog (#10602, #10607): the optional Free() RPC returns gRPC Unimplemented for many backends and the federation proxy, so the watchdog no longer logs a misleading Error freeing GPU resources on eviction. A new grpcerrors.IsUnimplemented helper distinguishes it from genuine failures.
• Idempotent backend installs (#10643): POST /backends/apply and the LOCALAI_EXTERNAL_BACKENDS boot loop no longer re-download and re-extract an already-installed backend on every apply/boot. Pass "force": true (the UI's install button still does, doubling as "Reinstall").

🔗 PRs: #10600, #10639, #10607, #10643

📊 PII/audit events as a Prometheus counter

The PII middleware / MITM audit pipeline now emits a single monotonic counter, localai_pii_events_total{kind, origin, action, direction}, on /metrics, instrumented at the EventStore.Record choke point. Labels are cardinality-bounded (no pattern or user IDs). This complements the capacity-bound /api/pii/events ring buffer and, crucially, makes silent filter failure alertable: rate() on the counter detects that the PII filter stopped firing after a deploy.

🔗 PRs: #10641

🔒 Gallery SSRF hardening

POST /models/apply with an empty id fetches the supplied url directly; in a default Docker setup (no API key) any reachable client could probe internal services or cloud-metadata (169.254.169.254) and exfiltrate a slice via the job error. Gallery config fetches now run through the existing ValidateExternalURL guard (the same one protecting the CORS proxy and media downloads), blocking private, loopback, link-local, unspecified, and metadata addresses. Only plain http(s):// is validated; huggingface://, github:, oci://, ollama://, and file:// are untouched.

🔗 PRs: #10673

---

🐛 Bug Fixes (recap)

• fix(vllm): restore non-streaming tool-call extraction that regressed after #10351 (a capability flag was mistaken for run state) - #10638
• fix(python-backends): decode tool-call arguments for chat templates (unbreaks MLX/Qwen3.5 agent loops) and split reasoning when a model em...

v4.5.6

What's Changed

👒 Dependencies

• chore(deps): bump actions/cache from 4 to 6 by @dependabot[bot] in #10593

Other Changes

• docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #10560
• fix(distributed): missing agent NATS permission by @ALameLlama in #10549
• feat(distributed): SyncedMap component + migrate finetune/quant/agent-tasks to cross-replica state by @localai-bot in #10542
• chore(fish-speech): drop the darwin/metal build target by @localai-bot in #10561
• fix(config): fall back to DefaultContextSize for unparseable GGUFs; pin NVFP4 gallery context_size by @localai-bot in #10563
• ci(vibevoice): skip the ASR transcription e2e on release tag builds by @localai-bot in #10567
• fix(gallery): match mmproj/model quant as a whole token so F16 no longer selects BF16 (#10559) by @localai-bot in #10564
• fix(distributed): return empty backend list for agent nodes instead of failing backend.list (#10545) by @localai-bot in #10565
• feat(distributed): add LOCALAI_DISTRIBUTED_SHARED_MODELS to skip staging on shared volumes (#10556) by @localai-bot in #10566
• chore: ⬆️ Update leejet/stable-diffusion.cpp to 9956436c925a367daeab097598b1ea1f32d3503f by @localai-bot in #10533
• fix(openresponses): bound resume-stream buffer and enforce response ownership by @localai-bot in #10569
• chore: ⬆️ Update ggml-org/whisper.cpp to 0ae02cdb2c7317b50991367c165736ce42ed96ac by @localai-bot in #10532
• chore: ⬆️ Update CrispStrobe/CrispASR to 6514c9da00b03a2f0f1b49a43fae4f3a01a41844 by @localai-bot in #10535
• chore: ⬆️ Update ggml-org/llama.cpp to 0ed235ea2c17a19fc8238668653946721ed136fd by @localai-bot in #10536
• fix(ik-llama): port multimodal path to mtmd API and bump to f96eaddb (#10534) by @localai-bot in #10568
• feat(backends): add voice-detect + face-detect ggml backends (replace Python insightface/speaker-recognition) by @localai-bot in #10441
• fix(kokoro): add explicit click dep so spacy CLI works on intel build by @localai-bot in #10572
• fix(launcher): robust binary download/upgrade (resume, rate-limit, UX) by @localai-bot in #10575
• fix(distributed): missing agent NATS permissions by @ALameLlama in #10571
• fix(fish-speech): allow invalid_reference_casting so tokenizers builds on darwin by @localai-bot in #10573
• fix(oci): retry layer downloads on transient network errors by @localai-bot in #10579
• chore(model-gallery): ⬆️ update checksum by @localai-bot in #10585
• chore: ⬆️ Update leejet/stable-diffusion.cpp to c1790754d31bec0731ed5fddc9d5b9ff22ee19cd by @localai-bot in #10584
• chore: ⬆️ Update CrispStrobe/CrispASR to 6b50f76e59700665358a1aabf5295597fa318e06 by @localai-bot in #10583
• chore: ⬆️ Update ggml-org/llama.cpp to dbdaece23de9ac63f2e7ca9e6bfcdc4fc156a3fa by @localai-bot in #10582
• chore: ⬆️ Update mudler/voice-detect.cpp to 3d510772357538c5182808ac7de2278b84824e24 by @localai-bot in #10581
• chore: ⬆️ Update mudler/face-detect.cpp to 06914b077d52f90d5421299138e7be6bdd06b5e8 by @localai-bot in #10580
• chore: ⬆️ Update vllm-metal (darwin) to v0.3.0.dev20260628073537 by @localai-bot in #10562
• chore(recon): re-pin voice/face-detect to squashed release commits (+ graph-cache fix) by @localai-bot in #10591
• fix(sglang): parse tool_call function arguments before applying the chat template by @pos-ei-don in #10558
• feat(realtime): Semantic VAD EOU token by @richiejp in #10444
• fix(openai): stop max_tokens streaming retry loop on reasoning models (#9716) by @Dennisadira in #10448
• fix(import): derive model name from selected GGUF for repo-root URIs by @Dennisadira in #10589
• fix(functions): avoid quadratic-time debug logging in CleanupLLMResult / ParseFunctionCall by @pos-ei-don in #10592
• chore: ⬆️ Update leejet/stable-diffusion.cpp to 3b6c9ca97cfcda8e68e719e6670d06379fcbe943 by @localai-bot in #10594
• chore: ⬆️ Update ggml-org/llama.cpp to 6f4f53f2b7da54fcdbbecaaa734337c337ad6176 by @localai-bot in #10595
• chore: ⬆️ Update localai-org/privacy-filter.cpp to 595f59630c69d361b5196f2aba2c71c873d0c13c by @localai-bot in #10596
• chore: ⬆️ Update CrispStrobe/CrispASR to 3b93758f9725d400eca82976f895e4cec3f31260 by @localai-bot in #10597
• chore: ⬆️ Update ikawrakow/ik_llama.cpp to f74a6fb87b315b2c3154166e075360e15021a61d by @localai-bot in #10598
• fix(import): strip file:// scheme from model path for local imports by @Dennisadira in #10599
• fix(tests): align openresponses test model name with GGUF-derived naming (#10589) by @localai-bot in #10609
• fix(macos): staple the notarization ticket to the .app, not just the dmg by @localai-bot in #10606
• fix(watchdog): persist UI-saved Check Interval across restarts (#10601) by @localai-bot in #10605
• feat(config): default swa_full:true for sliding-window-attention models by @localai-bot in #10611

New Contributors

• @ALameLlama made their first contribution in #10549

Full Changelog: v4.5.5...v4.5.6

v4.5.5

What's Changed

Other Changes

• fix(backends): repair release CI build/test breaks (kokoros, fish-speech, llama-cpp-quantization, sglang) by @localai-bot in #10547
• chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-bot in #10544
• fix(backends): whisper darwin run.sh loads whichever fallback lib exists (.so/.dylib) by @localai-bot in #10553

Full Changelog: v4.5.4...v4.5.5