fix(voice): stream first audio and survive a worker without the stream route (#13215, #12886)

[mrveiss]

Thinking Path

These two issues were filed as "slow" and "404", but reading the code they are one defect seen from two ends, and #13215's stated premise is partly wrong.

#13215 says the streaming route "sits unused". It does not. api/voice_stream.py:98 has consumed synthesize_stream() since #12501 — the WebSocket voice path, which is what speaks chat replies. What never used it is the HTTP path (api/voice.py).

That matters, because it explains the measurement. Chain, verified in code:

1. A worker older than the backend 404s /tts/synthesize/stream (bug(tts): synthesis 404s — deployed worker lacks /tts/synthesize/stream because autobot-tts-worker has NO resolve path AND NO drift visibility (silent skew on every backend update); /tts/clone-voice is a dead call that exists nowhere #12886).
2. synthesize_stream() raises, voice_stream.py catches it and emits {"type": "error", "message": "TTS synthesis failed: TTS worker error 404: ..."} — the exact string reported in bug(tts): synthesis 404s — deployed worker lacks /tts/synthesize/stream because autobot-tts-worker has NO resolve path AND NO drift visibility (silent skew on every backend update); /tts/clone-voice is a dead call that exists nowhere #12886.
3. The client (useVoiceOutput.ts) only falls back to HTTP when the socket fails. A per-utterance error frame is logger.warn-ed and nothing else. The streaming path produces silence, not slowness.
4. So every reply that is heard at all arrives via the HTTP fallback — which used the whole-blob route. That is the 4.7–10.1s.

So the naive fix (point api/voice.py at the streaming route) makes it strictly worse: it converts the one path that still worked into a second 404. The fix has to be capability-aware, which is why these ship together.

I could not verify the deployed worker's route table — that needs a live host, and #12886's history is five rounds of "merged, still not on the host". This change deliberately does not depend on that: it is correct whether or not the route is served.

#13207 is not in this PR. It is multimodal_processor — a different subsystem, zero file overlap — and it is a behaviour change (threshold 0.7→0.8, timeout 30→15) that needs an owner's decision on which value is intended. Per the closure gate, partial delivery must not close it, so it stays open. Detail in a comment on #13215.

What Changed

services/tts_client.py — capability probe + fallback (the core).
synthesize_stream() now raises TTSStreamUnsupported (not bare RuntimeError) on 404/405, distinguishing "the worker's route table predates the backend" from "synthesis failed". New stream_or_synthesize() prefers the streaming route and degrades to the whole-utterance route when it is absent. The 404 arrives before any chunk, so the fallback cannot duplicate or interleave audio; a guard re-raises rather than restarting if anything was already emitted. The negative result is cached for AUTOBOT_TTS_STREAM_PROBE_TTL (module constant, default 5 min) so it costs one failed round trip per window, not one per utterance, and re-probes afterwards — a worker updated in place resumes streaming without a backend restart. A 500 is not treated as skew and is never retried against a different endpoint.

api/voice_stream.py — the WS path uses stream_or_synthesize. A stale worker now speaks instead of going silent. This is the #12886 user-facing symptom, fixed independently of deploy state.

api/voice.py — /synthesize and /speak take stream: bool = Form(False). Default is unchanged (audio/wav whole blob), so every existing caller keeps its contract. With stream=true the response is a StreamingResponse of the worker's own [4-byte length][WAV] framing, forwarded unchanged.

Framing rather than concatenation is required, not stylistic: the chunks are separate, independently-decodable WAV files. Concatenated into one audio/wav body only the first RIFF header is valid, so decodeAudioData would decode the first ~250ms and silently drop the rest of the sentence.

useVoiceOutput.ts — requests stream=true and, when the framing header is present, splits frames across read boundaries (a frame can straddle two reads, including mid-header) and schedules each mini-WAV on the existing gapless timeline as it arrives. Falls back to the blob path when the header is absent. isSpeaking is deliberately not force-cleared on the framed path — chunks are still sounding when the response ends; the per-source onended owns that transition.

Also regenerated api.ts for the new field. Note: the generator bakes the generating machine's CWD/install path and unset-env placeholders into schema defaults, so a local regeneration produces unrelated churn; those hunks were reverted to the committed values and filed as #13298.

Verification

Regression tests were written first and fail on the pre-fix tree:

$ pytest api/voice_latency_test.py -q
FAILED test_streaming_synthesize_reaches_first_audio_far_sooner
  AssertionError: streaming time-to-first-audio 0.511s is not materially
  better than blocking 0.520s
FAILED test_streaming_response_carries_every_chunk_in_order
FAILED test_ws_path_still_speaks_when_worker_lacks_stream_route
  AssertionError: stale worker still surfaced an error frame:
  [{'type': 'tts_start', ...},
   {'type': 'error', 'message': 'TTS synthesis failed: TTS worker error 404: Not Found'},
   {'type': 'tts_end'}]
3 failed, 1 passed

That third failure message is #12886's reported symptom reproduced in-process from the repo alone.

After:

$ pytest api/voice_latency_test.py services/tts_client_test.py \
         api/voice_stream_test.py tests/test_voice_503_guard.py -q
22 passed

$ pytest api/api_endpoint_migrations_test.py ... -q     22 passed, 2111 skipped
$ pytest tests/services/tts_contract_test.py \
         tests/services/drift_checker_test.py \
         tests/test_updater_refreshes_tts_worker_12886.py -q      37 passed

The latency claim is measured, not asserted. A fake worker paces a 0.5s utterance as 5 chunks; time-to-first-byte is read off the ASGI http.response.body messages:

MEASURED time-to-first-audio: blocking=0.533s  streaming=0.112s   (4.8x)

which is the expected 1/5. Note starlette's TestClient buffers the whole response before returning, so it cannot observe TTFB at all — the test drives the ASGI app directly, and still exercises the real route, form parsing and StreamingResponse. An "it called synthesize_stream" assertion would pass even if the route buffered the whole utterance, i.e. it would not catch the actual defect.

Frontend, same before/after discipline:

$ npx vitest run useVoiceOutput.streaming.test.ts        (pre-fix)
  × requests the chunked variant and schedules each frame separately
  × plays the first chunk before the response has finished arriving
  × splits frames correctly when they arrive misaligned across reads
  ✓ falls back to whole-blob playback when the response is not framed
  3 failed | 1 passed

$ npx vitest run useVoiceOutput.streaming.test.ts \
      useVoiceOutput.playback.test.ts useVoiceOutput.test.ts \
      useVoiceConversation.test.ts                        22 passed
$ npx vue-tsc --noEmit -p tsconfig.app.json               clean
$ npx eslint <changed files>                              clean

Not verified — stated plainly rather than claimed. I did not execute anything against a live deployment, so I cannot confirm the deployed worker now serves /tts/synthesize/stream. That is exactly why this change does not depend on it: on a current worker the WS and HTTP paths stream, and on a stale one both degrade to working whole-blob audio instead of a 404 or silence. The live route probe from #12886's verification block remains the check to run before treating the deploy half as proven.

Closes #13215, #12886

Discovered and filed, not fixed here: #13298 (generated-types gate is machine-dependent), #13299 (a per-utterance WS error frame is only logged, so the sentence is dropped and the HTTP fallback never fires — this PR removes the common cause of that frame, not the handling gap).

Model Used

claude-opus-5

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

[mrveiss]

PR #13297 — awaiting CI + review.

[codecov]

Codecov Report

❌ Patch coverage is 88.57143% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
autobot-frontend/src/composables/useVoiceOutput.ts	88.57%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!