feat(sherpa-onnx): add Kokoro TTS + multilingual Piper voices#10309
Merged
Conversation
Wire the Kokoro model family into the sherpa-onnx backend (which only
supported VITS/Piper before) and add gallery voices for Italian, English,
Spanish, French and German plus a multilingual Kokoro model.
- csrc/shim.{c,h}: kokoro_* config setters (model/voices/tokens/data_dir/
dict_dir/lexicon/lang/length_scale) mirroring the VITS path, with the
matching frees in tts_config_free.
- backend.go: loadTTS now detects a Kokoro model (a voices.bin beside the
ONNX) and routes to configureKokoroTTS, otherwise configureVitsTTS.
Kokoro picks up espeak-ng-data, the jieba dict and the per-language
lexicons (only one English variant, to avoid tens of thousands of
duplicate-word warnings at load); the language= option hints the lang.
- backend_test.go: functional test for isKokoroModel detection.
- gallery: 5 Piper VITS voices (it_IT-paola, en_US-amy, es_ES-davefx,
fr_FR-siwis, de_DE-thorsten) + kokoro-multi-lang-v1.0, served through
sherpa-onnx-tts.yaml with native streaming TTS.
Verified by building the backend and synthesizing with a real Piper and
Kokoro model (31/31 specs pass, including real-model synth smokes).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 gofmt golangci-lint go-test
Collaborator
Author
Real end-to-end verificationRan the prebuilt
What this confirms beyond the unit test:
Coverage note: 3 of the 6 entries (it/de Piper + Kokoro) were fully synthesized. The remaining |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Kokoro TTS support to the sherpa-onnx backend (it previously only wired the VITS/Piper config path) and ships a set of multilingual gallery voices.
Backend
csrc/shim.{c,h}: newsherpa_shim_tts_config_set_kokoro_*setters (model / voices / tokens / data_dir / dict_dir / lexicon / lang / length_scale) mirroring the existing VITS setters, plus the matching frees intts_config_free.backend.go:loadTTSnow detects a Kokoro model (avoices.binnext to the ONNX) and routes toconfigureKokoroTTS; everything else stays on the refactoredconfigureVitsTTS. Kokoro picks upespeak-ng-data, the jiebadict/and the per-language lexicons. Only one English lexicon variant is passed (US preferred) to avoid tens of thousands of "duplicated word" warnings at load; thelanguage=model option hints the Kokoro language.backend_test.go: functional unit test forisKokoroModel.Gallery
Six new entries served via
sherpa-onnx-tts.yaml(native streaming TTS), archives auto-extracted:vits-piper-it_IT-paola-sherpavits-piper-en_US-amy-sherpavits-piper-es_ES-davefx-sherpavits-piper-fr_FR-siwis-sherpavits-piper-de_DE-thorsten-sherpakokoro-multi-lang-v1.0-sherpaWhy
The backend could only run VITS/Piper models, so the rich Kokoro multilingual model (which covers Italian and many other languages in a single model) was unusable. This also gives first-class non-English TTS voices that pair with the realtime streaming-TTS pipeline.
Testing
Built the sherpa-onnx backend from source and synthesized with a real Piper model (it_IT-paola) and the real Kokoro multi-lang model via a local synth smoke spec: both produce valid WAVs. Full suite: 31/31 specs pass,
gofmtandgolangci-lint(new-from-merge-base) clean.Assisted-by: Claude:claude-opus-4-8 gofmt golangci-lint go-test