feat(qwen3-tts-cpp): migrate to ServeurpersoCom/qwentts.cpp (streaming, speakers, voice design)#10316
Merged
Conversation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
…esign Map TTSRequest onto qwentts.cpp: instructions->instruct, voice->named speaker or clone-reference path, params map->ref_text + sampling. Add TTSStream over the qt chunk callback. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Register CppPCMFree before the n<=0 guard so a non-null buffer with zero samples cannot leak (the C contract returns NULL on failure, so this is defensive). Raised in code review. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
…0 & Q4_K_M Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Add resolveRequest unit specs (config audio_path used as the clone reference when Voice is empty; per-request audio Voice overrides it; a named-speaker Voice does not trigger cloning) plus a real-inference e2e that clones from audio_path (confirmed ref_spk_emb=yes in the pipeline). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| // writeWAV24k writes samples as a finalized 24 kHz mono 16-bit WAV at dst. | ||
| func writeWAV24k(dst string, samples []float32) error { | ||
| f, err := os.Create(dst) |
| // float32 slice in [-1,1] for use as cloning reference audio. qwentts expects | ||
| // 24 kHz; callers should supply 24 kHz reference clips. | ||
| func readWAVAsFloat(path string) ([]float32, error) { | ||
| f, err := os.Open(path) |
| } | ||
| var refPtr unsafe.Pointer | ||
| if len(ref) > 0 { | ||
| refPtr = unsafe.Pointer(&ref[0]) |
| return fmt.Errorf("failed to load qwen3-tts model (error code: %d)", ret) | ||
| var n int32 | ||
| ptr := CppTTS(req.Text, lang, instruct, speaker, refPtr, len(ref), refText, | ||
| s.seed, s.temperature, s.topK, s.topP, s.repPen, s.maxNew, unsafe.Pointer(&n)) |
| if n <= 0 { | ||
| return fmt.Errorf("qwen3-tts: synthesis produced no samples") | ||
| } | ||
| src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // C-allocated PCM, copied out before free |
| if n <= 0 { | ||
| return fmt.Errorf("qwen3-tts: synthesis produced no samples") | ||
| } | ||
| src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // C-allocated PCM, copied out before free |
| if nSamples <= 0 || samples == nil || streamChan == nil { | ||
| return 1 // continue | ||
| } | ||
| src := unsafe.Slice(samples, int(nSamples)) |
| } | ||
| var refPtr unsafe.Pointer | ||
| if len(ref) > 0 { | ||
| refPtr = unsafe.Pointer(&ref[0]) |
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Migrates the
qwen3-tts-cppbackend frompredict-woo/qwen3-tts.cpptoServeurpersoCom/qwentts.cpp(MIT), in place. Same backend name, meta-backend, and image variants - only the upstream library and backend internals change. The new upstream is an independent pure-C++/GGML port of the same Alibaba Qwen3-TTS 12Hz model with a clean C99 ABI (qt_*) purpose-built for language bindings, and it is structurally identical to our siblingomnivoice-cppbackend (same author).Why
The previous upstream was file-output only, exposed a C++ class API (forcing a hand-written shim), shipped a single 0.6B Chinese-centric model, and had no license (all-rights-reserved by default).
qwentts.cppis better on every axis we care about.New capabilities
stream: trueon/v1/audio/speech) via the upstream chunk-callback APIvoice=serena,vivian,ryan,aiden,eric(sichuan),dylan(beijing), ...instructions(e.g. "male, young adult, moderate pitch")voice= a 24kHz reference.wavpath, optionalref_text(in-context)How
cpp/goqwen3ttscpp.{h,cpp}): flattens the struct-basedqt_*ABI into flatqt3_*C functions (purego cannot marshal the structs). Synthesis mode (base/customvoice/voicedesign) is auto-detected from the talker GGUF; the shim passes speaker/instruct/ref through and surfacesqt_last_error().base.SingleThread;Loadresolves talker + auto-discovers the*tokenizer*.ggufsibling;TTS(buffered) +TTSStream(one process-lifetimepurego.NewCallbackrouted through a mutex-guarded channel). Request mapping (no proto change):instructions→instruct,voice→named-speaker-or-clone-path (by extension),ref_text+ sampling via the params map.MethodTTSStreamadded to the capability registry;backend/index.yamlmetadata updated; gallery rewritten to 10 curated models onServeurperso/Qwen3-TTS-GGUF(0.6B/1.7B x base/customvoice/voicedesign x Q8_0/Q4_K_M).Testing
.sovariants build;qt3_load+ full talker→codec pipeline synthesize a WAV viaTTS, andTTSStreamemits chunked frames starting with a RIFF/WAVE header. golangci-lint: 0 issues.Breaking change
The GGUF format differs - the old
endo5501/qwen3-tts.cppF16 weights are not compatible. Re-install aqwen3-tts-cpp*model from the gallery (nowServeurperso/Qwen3-TTS-GGUF) to upgrade. Seedocs/release-notes/qwen3-tts-cpp-qwentts-migration.md.Assisted-by: Claude:claude-opus-4-8 [Claude Code]
🤖 Generated with Claude Code