feat(omnivoice-cpp): add OmniVoice TTS backend (file + streaming, voice cloning + voice design)#10310
Merged
Conversation
…ice ov_* ABI Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…s with tests Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| // writeWAV24k writes samples as a finalized 24 kHz mono 16-bit WAV at dst. | ||
| func writeWAV24k(dst string, samples []float32) error { | ||
| f, err := os.Create(dst) |
| // float32 slice in [-1,1] for use as reference audio. OmniVoice expects 24 kHz; | ||
| // callers should supply 24 kHz reference clips. | ||
| func readWAVAsFloat(path string) ([]float32, error) { | ||
| f, err := os.Open(path) |
| } | ||
| var refPtr unsafe.Pointer | ||
| if len(ref) > 0 { | ||
| refPtr = unsafe.Pointer(&ref[0]) |
|
|
||
| var n int32 | ||
| ptr := CppTTS(req.Text, lang, instruct, refPtr, len(ref), refText, seed, | ||
| boolToInt(o.opts.denoise), unsafe.Pointer(&n)) |
| return fmt.Errorf("omnivoice: synthesis failed") | ||
| } | ||
| defer CppPCMFree(ptr) | ||
| src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // C-allocated PCM, copied out before free |
| return fmt.Errorf("omnivoice: synthesis failed") | ||
| } | ||
| defer CppPCMFree(ptr) | ||
| src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // C-allocated PCM, copied out before free |
| if nSamples <= 0 || samples == nil || streamChan == nil { | ||
| return 1 // continue | ||
| } | ||
| src := unsafe.Slice(samples, int(nSamples)) |
| } | ||
| var refPtr unsafe.Pointer | ||
| if len(ref) > 0 { | ||
| refPtr = unsafe.Pointer(&ref[0]) |
… cloning reference The model config tts.audio_path (ModelOptions.AudioPath) and tts.voice now provide a default voice-cloning reference used when a request omits Voice, so a cloned voice can be pinned in the model YAML instead of passed per request. A per-request voice still overrides. Paths resolve relative to the model dir. Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Mirrors the whisper/vibevoice convention: a -development meta aggregating the master-tagged image variants (the production meta and per-variant prod+dev image entries already existed; only the development meta aggregator was missing). Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
omnivoice-cpp, a Go + purego TTS backend wrapping ServeurpersoCom/omnivoice.cpp (GGML-based; Qwen3 612M backbone + RVQ audio codec, 24 kHz mono, 646 languages). Mirrors the structure of the existingqwen3-tts-cpp/crispasrbackends.Capabilities wired:
TTS) and streaming TTS (TTSStream) - OmniVoice'son_chunkcallback is bridged through gRPCTTSStream. The backend emits a self-describing WAV (header chunk first, then PCM16LE chunks), since the gRPCchan []bytepath never setsMessage.voice= reference audio path (decoded to float in Go and passed asref_audio_24k),params.ref_text= transcript.instructions= attribute keywords (gender, age, pitch, style, volume, emotion).How it's structured
cpp/gomnivoicecpp.cpp) exposing flatomni_*symbols over OmniVoice'sov_*C ABI (no by-value structs across the purego boundary). Links the upstreamomnivoice-corestatic lib.Load/TTS/TTSStreamoverbase.SingleThread; a single sharedpurego.NewCallback(serialized bySingleThread) avoids leaking a callback per request. The streaming callback fires synchronously on the calling thread (verified against upstreampipeline-tts.cpp), so the purego path is safe.Registration surfaces
backend/index.yaml: meta&omnivoicecpp(capabilities map, nouri) + 20 image entries (production + development), tag-suffixes aligned with the CI matrix..github/backend-matrix.yml: 12 build variants (cpu/cuda12/cuda13/intel-sycl/metal/rocm/vulkan/l4t), allDockerfile.golang.bump_deps.yamlpinsOMNIVOICE_VERSION.Makefilewiring; pref-only importer entry (core/http/endpoints/localai/backend.go) + a guard test confirming a bare GGUF URI never auto-resolves toomnivoice-cpp.omnivoice-cpp(Q8_0 default, ~945 MB) andomnivoice-cpp-hq(BF16), real sha256 from the HF LFS pointers.features/text-to-audio.mdsection +reference/compatibility-table.mdrow.Testing
make lintreports 0 issues;go vetclean.TTSwrites a valid 24 kHz WAV, andTTSStreamemits a WAV header chunk followed by PCM chunks during MaskGIT decode (rc=0). Env-gated (OMNIVOICE_MODEL/OMNIVOICE_CODEC) so CI unit runs are unaffected.Notes
omnivoice-codecRVQ pre-compute (.rvq) path is deferred; reference audio is passed as WAV per call.Assisted-by: claude:claude-opus-4-8 [Claude Code]