Repository navigation
voice: TTS output has large lag and drops text — slow non-streaming worker (3.8s/49ch, 14s/209ch) + per-sentence fallback cancels prior audio #12460
Description
Activity
Live confirmation of the 'huge pauses between sentences' symptom on the current deploy:
- Silence-then-speech: the initial 'no sound / stuck speaking' was the AudioContext-unlock bug (voice: 'AutoBot speaking' shows but no sound (and indicator stuck) — AudioContext not unlocked in WS streaming path; isSpeaking set before playback #12503); once unlocked, audio plays.
- The pauses are worker-synthesis latency, not a transport problem: WS
/api/voice/streamupgrades cleanly (nginx logs101) and is reused within the 30s idle window (_connectTtsWsreturns the open socket) — it is NOT reconnecting per sentence. - Measured worker throughput is ~real-time or slower: ~3.8s to synthesize a 49-char sentence (~146 KB WAV ≈ 3.3s of audio), ~14s for 209 chars. So there is no synthesis buffer — each sentence's audio finishes before the next sentence is synthesized, producing the audible gaps. First-audio latency is ~4s.
autobot-tts-workerruns CPU-only (backend health reportsnpu_acceleration: false).
This reinforces the fix directions already in this issue: stream audio out of the worker as it is produced, run synthesis on NPU/GPU (or a lighter/faster model), and/or synthesize-ahead (buffer N sentences) so the next chunk is ready before the current finishes.
Reopening — the user reports AutoBot's speech stuttering during playback, observed live 2026-08-04. The mechanism has changed since this issue was closed (the worker streams now, per #13215), but the user-visible outcome — unusable audio — is back, and it is measurable.
19 of 19 syntheses today ran slower than real-time. Playback needs >= 1.0x to stream without gaps.
0.09x 0.14x 0.15x 0.18x(x4) 0.19x(x2) 0.21x(x3) 0.22x 0.24x 0.25x(x2) 0.32x 0.44x 0.83x <1.0x: 19 >=1.0x: 0Worked examples:
Generated: 3360 ms of audio in 15025 ms (0.22x) Generated: 4320 ms of audio in 9826 ms (0.44x) Generated: 8080 ms of audio in 32398 ms (0.25x)At 0.21x the player drains ~5s of buffer for every 1s produced, so it starves and resumes repeatedly —
which is exactly the reported stutter. Contributing factor: load average 19.50 / 21.28 / 22.65 on 22
cores, sustained. The TTS process itself is only ~6.3% CPU, so it is starved rather than hogging.Average generation step timeclimbs 161 ms -> 294 ms -> 308 ms within a single session.Why this is throughput, not architecture
#13215 fixed time-to-first-audio (blocking
/tts/synthesize-> streaming). That fix holds. This is the next constraint: once streaming starts, the worker cannot sustain the stream. No amount of earlier first-audio helps if generation runs at a fifth of playback speed.Suggested directions
- Buffer a lead-in proportional to the measured real-time factor before starting playback, so a 0.2x worker produces a late-but-continuous stream rather than an immediate-but-broken one.
- Establish why the box sits at load ~20/22 cores sustained — the TTS process is only ~6.3% CPU, so it is losing the scheduler contest, not causing the load. The ChromaDB crash loop (bug(ai-stack): chromadb service fails to start — possible permission or venv issue #4090, 645 restarts, one exec attempt every 10s) is part of that churn.
- Emit the real-time factor as a metric and alert below 1.0x — the log already computes it every synthesis, and nothing consumes it.
Related: #13215 (first-audio latency, fixed), #10561 (voice pre-warm), #4090 (host load).
PR #13736 delivers two of the three directions from the reopen. Leaving this open — the third is host-side, and the fix itself needs host evidence before it can be called done.
1. Lead-in proportional to the measured real-time factor — done.
The lead-in is derived rather than guessed. For an utterance of
Daudio-seconds produced atraudio-seconds per wall-second, playback that starts withBseconds buffered stays gapless iff consumed time never exceeds produced audio —t <= B + r*tfor allt <= D— binding att = D, soB >= (1 - r) * D.Dis estimated from the utterance text;ris measured across chunk arrivals and carried between utterances, since it is a property of the worker and its host rather than of the sentence.Guards, because a pre-roll that always fires would just re-break #13215:
- no measured rate yet, or a worker at/above 1.0x → zero delay, first-audio latency untouched
- audio already scheduled ahead counts toward the lead-in, so a follow-on sentence inherits the previous one's remaining playout
- the lead-in is capped at 8s and backed by a release timer — a stalled stream degrades to "late", never "silent"
tts_endalways flushes the tail; a short utterance that never reaches its target is still spoken in full- barge-in drops held audio, so interruption stays immediate
3. Emit the factor as a metric and alert below 1.0x — done.
autobot_tts_realtime_factor(histogram + last-value gauge),autobot_tts_synthesis_below_realtime_totalandautobot_tts_first_chunk_seconds, labelled by worker route, measured intts_client.stream_or_synthesizeacross whichever route served the utterance. WARNING logged below 1.0x.alerts-tts-throughput.ymlalerts on a sustained below-real-time ratio and on a collapsed median factor.2. Host load ~20/22 cores with the TTS process at ~6.3% CPU — not addressed.
That is starvation, not saturation, and it is host-side rather than a code change in this repo. It overlaps the crash-loop churn already tracked in #4090. Flagging rather than silently dropping it.
Why this stays open
The measurement and the alert are code and are tested, but the acceptance claim here is a host-behaviour one — "the stutter is gone". That needs the real-time factor observed on the live deploy after this lands, not merge evidence. Verification evidence in the PR covers 32 frontend and 61 backend tests, lint, types and both closure gates.
5 remaining items
Filed the throughput question as #13766 — the TTS worker runs
torch 2.13.0+cpu(cuda False) with no accelerator path, on a host whereautobot-npu-worker.serviceis active alongside it.That is the why behind the numbers in this issue: median real-time factor of exactly 1.0x on an idle 22-core box is what CPU-only neural synthesis costs here.
It does not change the disposition of this issue. Pre-roll plus the
TTSRealTimeFactorCollapsedalert remain the right answer regardless of how #13766 is decided — if the worker stays on CPU they are the permanent fix, and if it gains an accelerator they are still the safety net for when it falls behind.Host evidence — measured today
The remaining gate on this issue was host evidence, not merge evidence. Measured against the running TTS worker (
model_loaded,engine_degraded: false), real-time factor = audio-seconds / wall-seconds:2026-08-04 (reopen) 2026-08-09 (now) host load / cores 19.50 / 21.28 / 22.65 on 22 1.85 / 1.93 / 1.63 on 22 syntheses at or above 1.0x 0 of 19 sentence-length: 1.04x-1.05x range 0.09x - 0.83x 0.76x - 1.05x chars audio wall RTF 12 1.12s 1.15s 0.97x 81 4.64s 4.42s 1.05x 192 10.00s 9.54s 1.05x 12 1.20s 1.57s 0.76x 81 4.08s 4.19s 0.97x 192 10.80s 10.34s 1.04x Full table and method:
docs/audit/tts_realtime_factor_host_measurement_2026_08_09.md(PR #13768).Reading
The collapse was load-driven, exactly as the reopen hypothesised. The TTS process sat at ~6.3% CPU while the box ran at load ~20 — it was losing the scheduler contest, not saturating. With the host at ~1.9/22 it keeps up, and the sentence-length utterances the chat path actually dispatches run at or about real time.
Short clips still read below 1.0x (0.76x at 12 chars) because model warm-up dominates a one-second clip. That is not a throughput problem, and it is why the merged telemetry measures the factor on steady-state generation and reports warm-up separately as time-to-first-audio.
Closing this
All three directions from the reopen are resolved:
- Lead-in proportional to the measured factor — merged in fix(voice): pre-roll audio against a below-real-time TTS worker + make the real-time factor observable (#12460) #13736 (
c761d651). Correctly a no-op at the current ~1.0x; it engages only when the worker falls behind. - Host load — resolved; load is down from ~20 to ~1.9 on 22 cores. This was bug(ai-stack): chromadb service fails to start — possible permission or venv issue #4090's territory and no longer reproduces.
- Factor emitted as a metric with an alert below 1.0x — merged in fix(voice): pre-roll audio against a below-real-time TTS worker + make the real-time factor observable (#12460) #13736.
The defect as reported is no longer reproducible on the host, and the mitigation against recurrence is merged.
One thing deliberately not claimed
The deployed backend still runs pre-merge code —
autobot_tts_realtime_factoris absent from its metrics endpoint and the Prometheus query returns empty. So the pre-roll and theTTSSynthesisBelowRealTimealert are merged but not yet live; deploying needs the builtin code-sync, which requires credentials this session does not hold. Tracked in #13767 rather than held open here, since it is a deployment-verification step and not part of this defect.- Lead-in proportional to the measured factor — merged in fix(voice): pre-roll audio against a below-real-time TTS worker + make the real-time factor observable (#12460) #13736 (
- added a commit that references this issue
on Aug 8, 2026 - added 9 commits that reference this issue
on Oct 3, 2026
Summary
Chat voice output (TTS) has large latency before speech starts and does not speak all of the assistant's text. Two compounding root causes: (1) the TTS worker synthesis is slow and non-streaming, and (2) the frontend per-sentence dispatch drops text under its fallback/cursor logic.
Evidence — worker latency (confirmed, measured)
POST /api/voice/synthesizereturns the full WAV in one shot (no streaming), and synthesis is slow:~68 ms/char, roughly linear. So a single paragraph blocks ~14 s before any audio, and even a short sentence is ~4 s. Because the response is a single blob (not chunked/streamed), nothing plays until the whole utterance is synthesized.
Frontend mechanism (code paths)
components/chat/ChatInterface.vue(#1319) speaks the assistant reply sentence-by-sentence during streaming viauseVoiceOutput.ts:Lag before first audio: a sentence is only dispatched once it is >=
_MIN_TTS_SENTENCE_CHARS(20) AND terminated by./!/?. So the first audio waits for the first complete >=20-char sentence, then the multi-second worker synthesis above. The single-shotspeak()path additionallyawaits the entire audio blob before playback (await response.blob()-> play), so no early audio.Not all text spoken:
speakStreaming()falls back tospeak(text, true)when the WS is unavailable.speak()begins withif (isSpeaking.value) _stopCurrentAudio()and_speakController?.abort()— so several per-sentence fallback calls in quick succession cancel each other, leaving only the last sentence audible._lastSpokenIdx += s.lengthusing the sentence without trailing whitespace, while_extractCompleteSentencesinternally consumes the full multi-char whitespace run (match[0], e.g. newlines / double spaces). Over a multi-paragraph reply the cursor drifts and slices can drop or duplicate text..(lists, bullet points, code blocks, a reply ending without punctuation) is only spoken by the end-of-streamremainderflush; if the!isTypingbranch mis-fires or the cursor was mis-advanced, the tail is dropped.Impact
Voice output is effectively unusable for normal replies: multi-second delay before speech, and partial/truncated narration of the message.
Requested (file only — no remediation)
/voice/synthesizestream audio rather than return one blob so playback can start on the first chunk.speak(), which aborts the in-flight utterance — the current fallback guarantees only the last sentence is heard._lastSpokenIdxby the exact consumed span (including inter-sentence whitespace) so long/multi-paragraph replies are not mis-sliced.Notes
autobot-tts-worker) is active on its port.