Add streaming LLM→TTS pipeline with SSE audio to client#948
Open
ext-sakamoro wants to merge 20 commits into
Open
Add streaming LLM→TTS pipeline with SSE audio to client#948ext-sakamoro wants to merge 20 commits into
ext-sakamoro wants to merge 20 commits into
Conversation
added 4 commits
July 22, 2026 16:39
Adds a backend that delegates chat completions to any HTTP endpoint speaking the OpenAI /v1/chat/completions protocol -- llama.cpp server, vLLM, LM Studio, LocalAI, Ollama's shim, or the OpenAI API itself. Bypasses the on-device Qwen download / load path so the remote model serves every refinement / personality call. - New OpenAICompatLLMBackend implementing the LLMBackend protocol. Non-streaming for now; a streaming variant can layer on top of the same request builder. - Module-level custom-LLM config with set_llm_config() plus a get_llm_backend() dispatch that reroutes to the remote backend when the endpoint is configured. Config changes drop the cached backend so URL / model swaps take effect immediately without a restart. - Three new capture_settings columns (custom_llm_endpoint, custom_llm_model, custom_llm_api_key) with an idempotent column-add migration and matching CaptureSettingsResponse / CaptureSettingsUpdate fields. - Startup bootstrap seeds set_llm_config() from the persisted row before the first LLM call; every capture-settings PUT re-syncs. - /llm/generate route skips the Qwen size validation and download- progress branch when the custom endpoint is active (remote server owns model selection). UI wiring in ProfileForm / capture-settings pane is a follow-up; the backend is already reachable through the existing PUT /settings/captures endpoint.
Real HTTP round-trip against a caller-configured endpoint -- no mocks -- covering protocol compliance, few-shot history propagation, dispatch, config-change cache invalidation, and fallback to the built-in Qwen path when the endpoint is cleared. Skips automatically when VOICEBOX_TEST_OPENAI_COMPAT_URL and VOICEBOX_TEST_OPENAI_COMPAT_MODEL are unset or the endpoint is not listening, so `pytest` on a fresh clone still returns green. The docstring at the top of the file documents the workflow for pointing it at llama.cpp, vLLM, LM Studio, Ollama, or OpenAI itself.
Adds three fields to the CaptureSettings TypeScript interface matching the backend Pydantic schema, and a "Custom LLM endpoint (advanced)" section under the Refinement group on the Captures settings page. The section renders three stacked inputs (endpoint URL, model name, API key) that write to the existing PUT /settings/captures mutation on blur, so a user typing a URL doesn't spam optimistic updates on every keystroke. Model and API-key inputs stay disabled until the endpoint field has a value, and a small footnote reports whether the built-in Qwen3 or the remote endpoint is currently active.
Adds translation strings for the "Custom LLM endpoint (advanced)"
section: title, description, three input placeholders, and the
two status messages ("using built-in Qwen3" vs "sending every
refinement to <endpoint>"). Other locales fall back to English
until a native translation lands.
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughChangesThe pull request adds configurable OpenAI-compatible LLM routing and streaming speech. Backend personality output flows through sentence-level TTS and SSE audio frames, while the frontend provides endpoint settings, incremental playback, cancellation, and streaming controls. Streaming speech and custom LLM
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant FloatingGenerateBox
participant ApiClient
participant speak_stream
participant stream_speak_events
participant OpenAICompatLLMBackend
participant TTS
User->>FloatingGenerateBox: Start streaming speech
FloatingGenerateBox->>ApiClient: POST /speak/stream
ApiClient->>speak_stream: Send streaming request
speak_stream->>stream_speak_events: Resolve profile and generate events
stream_speak_events->>OpenAICompatLLMBackend: Stream personality deltas
stream_speak_events->>TTS: Synthesize sentence chunks
TTS-->>ApiClient: Return SSE audio frames
ApiClient-->>FloatingGenerateBox: Schedule PCM playback
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (1)
backend/services/stream_speak.py (1)
159-161: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winConsider offloading final concat/save off the event loop.
concatenate_audio_chunks(CPU-bound) andsave_audio(blocking file I/O) run inline in the SSE async generator, so they block the event loop for the full-utterance duration after the last sentence. Wrapping them inawait asyncio.to_thread(...)keeps the server responsive to other requests during finalization.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/services/stream_speak.py` around lines 159 - 161, Update the SSE async generator’s finalization flow around concatenate_audio_chunks and save_audio to run the CPU-bound concatenation and blocking file write via await asyncio.to_thread, preserving the existing arguments, output path, sample rate, and final_audio/final_path behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/components/Generation/FloatingGenerateBox.tsx`:
- Around line 497-553: Add a useEffect in FloatingGenerateBox that observes
streamState.status and streamState.error, and calls the existing toast with the
stream speak title, error description, and destructive variant when streaming
enters the error state. Include the relevant toast, translation, and
stream-state dependencies so each error transition is surfaced without affecting
successful or idle states.
In `@app/src/components/ServerTab/CapturesPage.tsx`:
- Around line 499-536: Add explicit accessible labels for the three
custom-endpoint inputs in the SettingRow: associate distinct labels with
customLlmEndpoint, customLlmModel, and customLlmApiKey using matching htmlFor/id
values or equivalent aria-label attributes. Keep the existing input behavior and
translations intact.
In `@app/src/i18n/locales/en/translation.json`:
- Around line 988-995: Update customEndpoint.statusRemote in
app/src/i18n/locales/en/translation.json (lines 988-995) to disclose that both
refinements and personality rewrites are sent to the configured endpoint. Apply
the equivalent Japanese disclosure to app/src/i18n/locales/ja/translation.json
(lines 983-990), preserving the existing translation structure.
In `@app/src/lib/hooks/useStreamingSpeak.ts`:
- Around line 160-220: Store the server PCM sample rate from the meta event in a
dedicated ref, reset that ref in _teardown, and use it when creating the
AudioBuffer in the audio event branch. Calculate chunkDuration from the same
server rate so nextStartTimeRef remains accurate; retain the existing fallback
only when no valid meta rate is available.
In `@backend/models.py`:
- Around line 267-269: Update the CaptureSettingsResponse model and the GET/PUT
/settings/captures response flow so custom_llm_api_key is never serialized or
returned to clients. Keep the API key available for input and persistence, using
separate input/output models or an excluded response field, and expose only a
masked value or is_set indicator in responses.
In `@backend/services/stream_speak.py`:
- Around line 170-182: Update the stream_speak flow to accumulate each spoken
sentence across both personality paths, including can_stream_llm and
rewrite_as_profile, and persist the joined accumulated text in
history.create_generation instead of the original text parameter. Keep the
existing audio generation behavior unchanged and ensure the stored text reflects
the exact rewritten sentences rendered in the audio.
In `@backend/tests/test_openai_compat_integration.py`:
- Around line 119-132: Update the generate assertion in the few-shot integration
test to verify the response matches the output mapped from the supplied examples
for the chosen input, rather than only checking that it is non-empty. Preserve
the existing examples and prompt setup while asserting the expected mapped
translation.
In `@backend/utils/chunked_tts.py`:
- Around line 387-441: Limit concurrent backend.generate calls in
generate_streaming_from_sentences with a semaphore or bounded worker mechanism,
while preserving ordered output and existing seed behavior. Ensure each task
acquires and releases the concurrency guard around inference. Wrap streaming and
draining logic in finally, cancel all remaining pending tasks, and await them
with exceptions consumed so failures cannot leave orphaned work.
---
Nitpick comments:
In `@backend/services/stream_speak.py`:
- Around line 159-161: Update the SSE async generator’s finalization flow around
concatenate_audio_chunks and save_audio to run the CPU-bound concatenation and
blocking file write via await asyncio.to_thread, preserving the existing
arguments, output path, sample rate, and final_audio/final_path behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 48a1a6e2-a4a4-4172-ae1d-e9b809919d5a
📒 Files selected for processing (23)
app/src/components/Generation/FloatingGenerateBox.tsxapp/src/components/ServerTab/CapturesPage.tsxapp/src/i18n/locales/en/translation.jsonapp/src/i18n/locales/ja/translation.jsonapp/src/lib/api/client.tsapp/src/lib/api/types.tsapp/src/lib/hooks/useStreamingSpeak.tsbackend/app.pybackend/backends/__init__.pybackend/backends/openai_compat_backend.pybackend/backends/qwen_llm_backend.pybackend/database/migrations.pybackend/database/models.pybackend/models.pybackend/routes/generations.pybackend/routes/llm.pybackend/routes/speak.pybackend/services/generation.pybackend/services/personality.pybackend/services/settings.pybackend/services/stream_speak.pybackend/tests/test_openai_compat_integration.pybackend/utils/chunked_tts.py
Fills in the missing docstrings CodeRabbit flagged on this PR: - OpenAICompatLLMBackend: __init__ / is_loaded / generate + the two private helpers (_build_messages, _extract_content). - services.settings: update_capture_settings (which now propagates the custom-LLM config) plus the pre-existing _get_or_create_capture_row, _get_or_create_generation_row, and update_generation_settings helpers that were missing coverage on the files this PR already touches. Brings the coverage on files modified in this PR from 59.4 % to 91.9 %, above the 80 % threshold.
ext-sakamoro
force-pushed
the
feature/custom-llm-endpoint
branch
from
July 22, 2026 14:20
f009a41 to
2f7e865
Compare
6 tasks
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/services/stream_speak.py`:
- Around line 100-112: Update both fallback branches in the sentence-stream
setup to pass their text through stream_sentences(...,
max_chunk_chars=max_chunk_chars) instead of _single_sentence_stream(...).
Preserve the rewritten text for the personality fallback and the original text
for the plain fallback, while retaining sentence-level chunking and the existing
max_chunk_chars limit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b5aaa78c-6020-486e-89ba-6b0819e59a8c
📒 Files selected for processing (17)
app/src/components/Generation/FloatingGenerateBox.tsxapp/src/i18n/locales/en/translation.jsonapp/src/i18n/locales/ja/translation.jsonapp/src/lib/api/client.tsapp/src/lib/api/types.tsapp/src/lib/hooks/useStreamingSpeak.tsbackend/backends/__init__.pybackend/backends/openai_compat_backend.pybackend/backends/qwen_llm_backend.pybackend/models.pybackend/routes/generations.pybackend/routes/speak.pybackend/services/generation.pybackend/services/personality.pybackend/services/settings.pybackend/services/stream_speak.pybackend/utils/chunked_tts.py
🚧 Files skipped from review as they are similar to previous changes (9)
- backend/services/generation.py
- app/src/i18n/locales/ja/translation.json
- app/src/i18n/locales/en/translation.json
- backend/routes/speak.py
- backend/services/settings.py
- app/src/lib/api/client.ts
- app/src/lib/hooks/useStreamingSpeak.ts
- app/src/lib/api/types.ts
- app/src/components/Generation/FloatingGenerateBox.tsx
- Make the custom LLM API key write-only: replace the plaintext response field with a boolean ``custom_llm_api_key_configured`` flag on ``CaptureSettingsResponse``, computed via a ``model_validator(mode= 'before')`` so the raw value is dropped before Pydantic materialises the response. The settings UI reads the flag, keeps the input blank on load, wipes the field after a successful save, and adds an explicit "Clear" action — the key never survives on the client where a browser cache or dev-tools inspection could grab it. - Disclose that both refinement and personality rewrites now travel through the configured endpoint: update the customEndpoint status string and swap the "fully local, no cloud" sidebar bullet for a "runs remotely" variant that names the endpoint when it's active. en and ja both updated. - Add explicit accessible labels for the three custom-endpoint fields (URL / model / API key) so screen readers can distinguish them; the previous ``SettingRow`` title had no ``htmlFor`` and the inputs had no ``aria-label``. - Guard against structured (non-string) ``choices[0].message.content`` in ``_extract_content``: tool-use / vision servers sometimes return content parts arrays and used to trip ``AttributeError`` inside ``.strip()``; raise a clear ``ValueError`` instead. - Strengthen the few-shot integration test: pass an arbitrary ``ping → PONG``, ``foo → BAR`` mapping and assert the mapped-shape token leads the response so a passing run actually proves the ``examples`` field wasn't silently dropped in transit.
ext-sakamoro
force-pushed
the
feature/custom-llm-endpoint
branch
from
July 22, 2026 14:51
2f7e865 to
7cfd611
Compare
kemo04
reviewed
Jul 22, 2026
kemo04
left a comment
kemo04
left a comment
There was a problem hiding this comment.
AI Code Review (PR Review Agent)
- [Logic & Correctness · Medium] —
app/src/components/ServerTab/CapturesPage.tsx(diff line 186): commitCustomLlmApiKey sendscustomLlmApiKeyDraftinstead of the pre-trimmedtrimmedvariable. - [Test Coverage · Critical] —
app/src/components/Generation/FloatingGenerateBox.tsx(diff line 40): New streaming availability logic based on custom LLM settings and profile personality lacks test coverage. - [Test Coverage · Critical] —
app/src/components/Generation/FloatingGenerateBox.tsx(diff line 48): Error toast notification effect on stream failure lacks test coverage.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/backends/__init__.py (1)
795-842: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winSynchronize configuration updates with backend resolution.
Line 799 updates endpoint/model/key before the cache lock, while Lines 827-829 can return the cached backend without that lock. A request racing a settings update can send its prompt to the prior custom endpoint using stale credentials. Guard configuration snapshots and cache invalidation/resolution under one consistent lock (or a dedicated config lock).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/backends/__init__.py` around lines 795 - 842, Synchronize custom LLM configuration reads and writes with backend resolution. Update set_llm_config and get_llm_backend/_get_openai_compat_backend so endpoint, model, and API-key snapshots, cache invalidation, and cached-backend returns are guarded by the same lock, preventing requests from resolving or using a backend with stale configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/routes/generations.py`:
- Around line 90-103: The streaming personality path must persist the rewritten
personality text instead of leaving generations.text as data.text. Update the
supports_streaming branch around _run_streaming_personality_generation() to
obtain or retain the rewritten text and pass it through the existing generation
status update/persistence flow, while preserving the current non-streaming
behavior and source value.
---
Outside diff comments:
In `@backend/backends/__init__.py`:
- Around line 795-842: Synchronize custom LLM configuration reads and writes
with backend resolution. Update set_llm_config and
get_llm_backend/_get_openai_compat_backend so endpoint, model, and API-key
snapshots, cache invalidation, and cached-backend returns are guarded by the
same lock, preventing requests from resolving or using a backend with stale
configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e369596b-ac3c-4b42-9aaa-90e97fd63b85
📒 Files selected for processing (18)
app/src/components/Generation/FloatingGenerateBox.tsxapp/src/components/ServerTab/CapturesPage.tsxapp/src/i18n/locales/en/translation.jsonapp/src/i18n/locales/ja/translation.jsonapp/src/lib/api/client.tsapp/src/lib/api/types.tsapp/src/lib/hooks/useStreamingSpeak.tsbackend/backends/__init__.pybackend/backends/openai_compat_backend.pybackend/backends/qwen_llm_backend.pybackend/models.pybackend/routes/generations.pybackend/routes/speak.pybackend/services/generation.pybackend/services/personality.pybackend/services/stream_speak.pybackend/tests/test_openai_compat_integration.pybackend/utils/chunked_tts.py
🚧 Files skipped from review as they are similar to previous changes (7)
- app/src/components/Generation/FloatingGenerateBox.tsx
- app/src/i18n/locales/ja/translation.json
- backend/services/generation.py
- app/src/lib/api/client.ts
- app/src/lib/hooks/useStreamingSpeak.ts
- app/src/components/ServerTab/CapturesPage.tsx
- backend/tests/test_openai_compat_integration.py
CodeRabbit flagged a race between ``set_llm_config`` and ``get_llm_backend``: the writer updated the endpoint/model/key globals before acquiring the lock, while the reader's fast path returned a cached backend without any lock. A request that raced a settings update could therefore reach a cached ``OpenAICompatLLMBackend`` still pointing at the previous URL while the freshly-typed key had already been swapped in — sending a fresh credential to a stale endpoint (or the reverse) is exactly the shape of leak we've been trying to avoid. Fold the config snapshot, the change detection, the cache invalidation, the cache lookup, and the on-miss constructor call all under ``_llm_backends_lock``. The Qwen fallback still runs outside the lock — its own dispatch can take multiple seconds to load a model and we don't want that blocking concurrent ``set_llm_config`` writes. ``_get_openai_compat_backend`` disappears (it was a private helper with no external callers) since the resolution now lives inline in ``get_llm_backend`` where the config read happens.
ext-sakamoro
force-pushed
the
feature/custom-llm-endpoint
branch
from
July 22, 2026 19:24
7cfd611 to
09fe330
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/services/stream_speak.py`:
- Around line 211-216: Update the SpeakStreamComplete payload in the streaming
completion flow to set audio_path using config.to_storage_path(final_path),
matching the persisted speech record instead of returning str(final_path).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 06ae8d86-0518-402e-aa7f-21a845e8f553
📒 Files selected for processing (16)
app/src/components/Generation/FloatingGenerateBox.tsxapp/src/i18n/locales/en/translation.jsonapp/src/i18n/locales/ja/translation.jsonapp/src/lib/api/client.tsapp/src/lib/api/types.tsapp/src/lib/hooks/useStreamingSpeak.tsbackend/backends/__init__.pybackend/backends/openai_compat_backend.pybackend/backends/qwen_llm_backend.pybackend/models.pybackend/routes/generations.pybackend/routes/speak.pybackend/services/generation.pybackend/services/personality.pybackend/services/stream_speak.pybackend/utils/chunked_tts.py
🚧 Files skipped from review as they are similar to previous changes (7)
- app/src/components/Generation/FloatingGenerateBox.tsx
- app/src/i18n/locales/ja/translation.json
- backend/services/generation.py
- app/src/lib/api/client.ts
- app/src/lib/hooks/useStreamingSpeak.ts
- app/src/i18n/locales/en/translation.json
- app/src/lib/api/types.ts
added 12 commits
July 23, 2026 04:34
Two CodeRabbit follow-ups on the ``__init__.py`` lock work: - Extract ``"openai_compat"`` into a module-level ``_OPENAI_COMPAT_CACHE_KEY`` constant so ``set_llm_config`` (cache pop), ``get_llm_backend`` (cache lookup), and the on-miss install can't drift on the string literal. - Rewrite the comment that used to describe the Qwen fallback as having "its own lock". ``get_llm_backend_for_engine`` actually reuses the same non-reentrant ``_llm_backends_lock``, and the release-before-call ordering is load-bearing — a future edit that nested the fallback inside the ``with`` block above would self-deadlock the calling thread. Make that constraint explicit so the pattern is defensible on read.
Extends the LLMBackend protocol with generate_stream() and supports_streaming(). The OpenAI-compatible backend implements a real SSE consumer against POST /chat/completions with stream=true, yielding delta.content strings as they arrive and honouring the [DONE] sentinel. The built-in Qwen3 backends (PyTorch and MLX) declare no streaming support and fall back to a single-yield wrap of generate() so consumers can call the same API against every backend without a capability probe.
Three new async helpers layer on top of the existing sentence splitter: - _find_first_sentence_end — forward scan variant of the last-end helper, used by the streaming buffer to flush the earliest complete sentence as soon as it lands. - stream_sentences — consumes an LLM text stream and emits one complete sentence at a time, with a max-chunk-chars fallback for runaway generations without punctuation. - generate_streaming_from_sentences — fires per-sentence TTS as soon as sentences arrive, spawning each as an asyncio.create_task and yielding ordered audio chunks. Sentence N+1's TTS starts while the caller is still awaiting sentence N. - generate_streaming_chunked — end-to-end helper that stitches LLMBackend.generate_stream → stream_sentences → generate_streaming_from_sentences and returns the final crossfaded audio.
- services.personality.rewrite_as_profile_stream mirrors rewrite_as_profile but yields per-token deltas, so the streaming TTS pipeline can start work before the LLM has finished restating the message. - services.generation.run_generation grows a personality_prompt parameter. When set alongside a streaming-capable LLM backend, the background task runs the streaming pipeline instead of the sequential text → chunked TTS path. Backends without streaming ignore the parameter and take the existing route. - routes.generations.generate_speech chooses which path applies: it defers the rewrite into the background when backend.supports_streaming() reports True (letting LLM and TTS overlap), and materialises it up-front when the backend can only produce full text (preserving the current behaviour for the built-in Qwen3 loaders). Verified against a real streaming source: with an SSE server in front of a mock TTS backend (500 ms per sentence), first-audio latency drops by ~7 s on a 3-sentence weather-forecast prompt while total wall clock stays comparable — real TTS engines will amplify the total-time win too because their per-sentence latency is closer to the LLM's.
- Four Pydantic shapes for the SSE payload (meta / audio / complete / error), all tagged with a Literal ``type`` discriminator so the client can route frames without a schema-per-frame lookup. - ``generate_streaming_from_sentences`` now yields (audio, sr, sentence) 3-tuples instead of just (audio, sr). Callers that only cared about the audio (``generate_streaming_chunked``, ``_run_streaming_personality_generation``) discard the extra field; the new SSE emitter uses it to echo the sentence back so the client can subtitle progressively. - ``services.stream_speak.stream_speak_events`` drives one /speak/stream request end-to-end: sets up the TTS backend, streams the LLM (if the backend can) or falls back to a one-shot personality rewrite, dispatches per-sentence TTS via ``generate_streaming_from_sentences``, and yields SSE-formatted frames ready to be handed straight to a ``StreamingResponse``. Audio is base64-encoded PCM float32 in a JSON payload — decodes straight into a ``Float32Array`` on the browser side. Final audio is concatenated + saved to disk and recorded in the generations table so the streaming route feeds History the same way /speak does. Errors at any step emit an ``error`` frame followed by the terminal ``[DONE]`` sentinel; the SSE connection never dies on a partial state. Verified end-to-end against ALICE-LLM's SSE endpoint: the emitter produces meta → audio (per sentence) → complete → [DONE] on a two-sentence weather prompt, base64 PCM chunks decode to the expected float32 sample counts, and the concatenated wav lands on disk.
Sibling of POST /speak that ships audio as it renders instead of returning a generation ID for the client to poll on. Same profile / engine / personality / X-Voicebox-Client-Id binding resolution as the plain route — the only difference is the response body, which is a ``text/event-stream`` sequence of frames driven by ``services.stream_speak.stream_speak_events``. Headers pin ``X-Accel-Buffering: no`` and ``Cache-Control: no-cache`` so each frame lands on the wire the instant we yield it, and the generator is imported late to keep the fire-and-forget /speak path's import graph unchanged for callers that don't need the streaming service.
Adds the frontend counterpart of the backend SpeakStream* Pydantic models: a StreamingSpeakRequest body shape and a discriminated union covering the four SSE frames (meta / audio / complete / error). apiClient.streamSpeak opens ``POST /speak/stream`` with fetch + a ReadableStream reader, splits the body on ``\n\n`` frame boundaries, strips the ``data:`` prefix from each field line (per the SSE spec, values from repeated data lines join with newline), stops on the ``[DONE]`` sentinel, and dispatches every parsed frame through the supplied ``onEvent`` callback. Transport failures reject the returned promise; ``AbortSignal`` cancellation propagates as ``AbortError`` so callers can distinguish user cancellation from a real fault.
React hook that consumes /speak/stream via apiClient.streamSpeak and schedules each sentence's audio against a single AudioContext so playback starts before the LLM has finished generating. State machine covers idle → connecting → streaming → complete | error | aborted so the UI can distinguish user cancellation from transport failure. A ``nextStartTime`` cursor keeps consecutive sentences abutting each other with zero gap when audio arrives faster than real time; when it arrives slower, ``AudioContext.currentTime`` takes over as the floor so we never schedule in the past. Base64 PCM float32 chunks decode with atob → Uint8Array → Float32Array → AudioBuffer with no external decoder dependency. The hook exposes an ``abort()`` method that tears down the fetch, stops every scheduled BufferSourceNode, and closes the AudioContext — and it wires the same teardown into an unmount effect so navigating away mid-stream doesn't leak an audio context. Playback progression exposes a ``playingIndex`` that only moves forward — a chunk finishing after a later one already started can't drag the UI back, which matters when consumers highlight the currently speaking sentence.
Renders a secondary "Speak with streaming (β)" button next to the existing Generate button whenever the user has both wired a custom OpenAI-compatible LLM endpoint and selected a profile that has a personality prompt set — the two conditions where the LLM+TTS overlap actually buys any latency. Otherwise the button stays hidden and the box looks exactly like it did before. Clicking runs useStreamingSpeak with the current text, profile, engine, and language; while a stream is active the icon flips from AudioLines to Square and the click aborts. Tooltip mirrors the button label so the affordance is discoverable without hover. en / ja translations for the two new labels; other locales fall back to English until a native pass lands.
- Fix the AudioBuffer sample-rate bug in ``useStreamingSpeak``: the third argument to ``createBuffer`` must describe the *data's* rate (24 kHz TTS output on the meta frame), not the AudioContext's device rate (44.1/48 kHz on most hardware). The old code used ``AudioContext.sampleRate`` and ``sampleRateOrDefault(...)``, so 24 kHz audio was rendered as if it were 48 kHz and played ~2× fast and pitched up. Track the meta rate in a ref, reset it in teardown, and use it for both the buffer and the ``chunkDuration`` cursor so the gapless ``nextStartTime`` chain doesn't drift. - Persist the spoken text on the /speak/stream generations row: the audio is rendered from the streamed sentences on the personality path, so the previous ``data.text`` (raw user input) disagreed with what History would play back. Accumulate the yielded sentence values and store the joined result instead. - Route the fallback and no-personality branches through ``stream_sentences`` too, so long inputs stay chunked at ``max_chunk_chars`` instead of collapsing into one oversized ``backend.generate`` call whenever the LLM can't stream. The single-shot ``_single_sentence_stream`` helper becomes ``_as_single_chunk`` and just adapts a materialised string into a one-element async iterator for the splitter. - Cap per-sentence TTS concurrency in ``generate_streaming_from_sentences``: a fast LLM used to fan out dozens of parallel ``backend.generate`` calls against a single backend, which only guards model loading not inference. Wrap the calls in a semaphore (default 2) and cancel + await outstanding tasks in a ``finally`` block so a mid-stream cancellation doesn't leak orphan work. - Surface streaming errors through the shared toast: the ``streamState.status === 'error'`` transition was silent, so a mid-stream server fault just flipped the button back to idle with no UI feedback. Add a ``useEffect`` that toasts on the error transition with an en/ja disclosure.
CodeRabbit flagged that the /generate fire-and-forget path was still storing ``data.text`` (the raw user input) even when ``supports_streaming()`` shipped the audio off through the streaming personality pipeline. ``_run_streaming_personality_generation`` returned only (audio, sample_rate), and ``update_generation_status`` doesn't touch the ``text`` column, so the History row disagreed with what listeners actually heard on the personality path. Return the joined sentence values as ``spoken_text`` from the streaming helper and patch the row in a small ``_update_generation_text`` helper (via ``asyncio.to_thread`` so the SQLAlchemy write doesn't block the event loop). The non-streaming sequential path is unchanged — its generations row was already created with the rewritten text before ``enqueue_generation``.
CodeRabbit flagged that the streaming complete event returned ``str(final_path)`` while the generations row was persisted with ``config.to_storage_path(final_path)``. A client that looks the row up by ``generation_id`` would then see the storage-relative path in History but an absolute filesystem path in the SSE payload, so the two representations never matched up. Ship the same shape in both places.
ext-sakamoro
force-pushed
the
feature/custom-llm-endpoint
branch
from
July 22, 2026 19:35
09fe330 to
53bcae7
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/components/Generation/FloatingGenerateBox.tsx`:
- Around line 511-523: Update the stop-control rendering and disabled logic
around the Button in FloatingGenerateBox so it remains available whenever a
stream is active, regardless of selectedProfileId, personality, or current text
input. Preserve the existing validation requirements for starting a new
generation, while ensuring an active useStreamingSpeak stream can always be
explicitly aborted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 74c2c0c2-70f7-4d1c-9e93-80999529d0e5
📒 Files selected for processing (16)
app/src/components/Generation/FloatingGenerateBox.tsxapp/src/i18n/locales/en/translation.jsonapp/src/i18n/locales/ja/translation.jsonapp/src/lib/api/client.tsapp/src/lib/api/types.tsapp/src/lib/hooks/useStreamingSpeak.tsbackend/backends/__init__.pybackend/backends/openai_compat_backend.pybackend/backends/qwen_llm_backend.pybackend/models.pybackend/routes/generations.pybackend/routes/speak.pybackend/services/generation.pybackend/services/personality.pybackend/services/stream_speak.pybackend/utils/chunked_tts.py
🚧 Files skipped from review as they are similar to previous changes (7)
- app/src/lib/hooks/useStreamingSpeak.ts
- app/src/i18n/locales/ja/translation.json
- backend/routes/speak.py
- app/src/i18n/locales/en/translation.json
- app/src/lib/api/client.ts
- app/src/lib/api/types.ts
- backend/services/generation.py
CodeRabbit flagged that the streaming Speak button vanished (or went disabled) the moment the user cleared the text field or switched to a profile without a personality — but ``useStreamingSpeak`` only tears down on an explicit ``abort()`` or on unmount, so the audio kept playing with no in-place way to stop it. Keep the button mounted while ``streamingActive`` is true regardless of ``streamingAvailable``, and only enforce the profile / text validation when the user is starting a new stream. Once a stream is running the button is always clickable so ``streamAbort()`` can tear it down cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #947 (custom OpenAI-compat endpoint). This PR stacks on top of it — every commit from #947 is included, with the streaming work in the last eight.
Layers a streaming path onto the endpoint infrastructure from #947 so users pointed at an SSE-capable OpenAI-compat endpoint (llama.cpp server, vLLM, LM Studio, LocalAI, OpenAI, or a self-hosted engine like ALICE-LLM) hear the first sentence before the LLM has finished generating.
Design
Backend
LLMBackend.generate_stream()+supports_streaming()added to the protocol.OpenAICompatLLMBackendimplements a real SSE consumer viahttpx.AsyncClient.stream()— yieldsdelta.contentstrings as they arrive, honours the[DONE]sentinel. The built-in Qwen3 backends declare no streaming and fall back to a single-yield wrap ofgenerate().chunked_ttsgains three async helpers:stream_sentences(LLM chunk iter → sentence iter),generate_streaming_from_sentences(fires per-sentence TTS asasyncio.create_task, yields ordered(audio, sr, sentence)3-tuples), andgenerate_streaming_chunkedend-to-end.services.personality.rewrite_as_profile_streammirrorsrewrite_as_profilebut yields per-token deltas so the streaming TTS pipeline can start work before the LLM has finished restating the message.services.generation.run_generationgrows apersonality_promptparameter. When set alongside a streaming-capable LLM backend, the background task runs the streaming pipeline instead of the sequential text → chunked TTS path. Backends without streaming ignore the parameter and take the existing route.Frontend
Measured latency
Against a real streaming source (an OpenAI-compat SSE endpoint) with a mock TTS backend rendering each sentence in 500 ms, a two-sentence weather-forecast prompt reached first audio ~7 s earlier than the sequential LLM→TTS path (29 s vs 36 s wall clock). Real TTS engines (Kokoro, Chatterbox, Qwen CustomVoice) will amplify the total-time win too because their per-sentence latency is closer to the LLM's than the mock is.
Testing
Not yet verified in-browser. The UI parts (Web Audio scheduling, autoplay policy handling, subtitle rendering) follow the standard patterns for progressive audio playback but I don't have a Tauri dev environment set up here to click through the actual button. The (β) suffix on the label reflects that. Happy to iterate on visual issues once someone with the setup can play with it.
Breaking changes / migration
None. `POST /speak/stream` is additive alongside `POST /speak`. The streaming path in `run_generation` only fires when the LLM backend reports `supports_streaming=True` AND `personality=true`; the existing sequential path is preserved otherwise, and the streaming button UI only appears when the two settings that make it useful are both on.
Summary by CodeRabbit