Problem
autobot-backend/api/voice_stream.py:100 hard-codes _TTS_PIPELINE_DEPTH = 3 (added in #6752 to absorb jitter via N-chunks-ahead pipelining).
But the TTS worker (tts-worker.py:285-287) calls Pocket TTS via loop.run_in_executor(None, _run_synthesis, ...). The default executor's threads run Python code that holds the GIL during synthesis (model inference is Python-level). Concurrent requests effectively serialize on the GIL/model lock. depth=3 queues 3 simultaneous synth tasks that take 3× synth time total, not 1× as true pipelining would.
Empirically: 3 concurrent synth = ~27s vs 3 sequential = ~11s in informal benchmark.
Hypothesis on right default
- depth=1 — pure sequential, no pipelining; risk of audible gap if synth time > playback time of one chunk
- depth=2 — synthesize chunk N+1 while client plays chunk N. Worker still GIL-serialized but the client gets continuous audio because chunk N+1 lands while N still plays.
- depth=3 — schedules chunk N+2 as well, which only adds queue contention with no gain.
Recommended default: 2. Make env-configurable so ops can tune without redeploy:
_TTS_PIPELINE_DEPTH = int(os.getenv("AUTOBOT_TTS_PIPELINE_DEPTH", "2"))
Acceptance criteria
Out of scope (file follow-ups)
- Benchmark across audio lengths and pick optimal default empirically
- Investigate whether Pocket TTS releases GIL during inference (would change calculus)
- Consider running multiple TTS worker processes for true parallelism
Why now
User reports voice latency: "it takes very long to respond with voice". Even shaving 1s off multi-sentence responses is noticeable. PR #6810 unblocks the "TTS works first time only" bug; this PR addresses the latency complaint.
Problem
autobot-backend/api/voice_stream.py:100 hard-codes
_TTS_PIPELINE_DEPTH = 3(added in #6752 to absorb jitter via N-chunks-ahead pipelining).But the TTS worker (tts-worker.py:285-287) calls Pocket TTS via
loop.run_in_executor(None, _run_synthesis, ...). The default executor's threads run Python code that holds the GIL during synthesis (model inference is Python-level). Concurrent requests effectively serialize on the GIL/model lock. depth=3 queues 3 simultaneous synth tasks that take 3× synth time total, not 1× as true pipelining would.Empirically: 3 concurrent synth = ~27s vs 3 sequential = ~11s in informal benchmark.
Hypothesis on right default
Recommended default: 2. Make env-configurable so ops can tune without redeploy:
Acceptance criteria
_TTS_PIPELINE_DEPTHreadsAUTOBOT_TTS_PIPELINE_DEPTHenv var, default 2docs/developer/AUTOBOT_REFERENCE.mdOut of scope (file follow-ups)
Why now
User reports voice latency: "it takes very long to respond with voice". Even shaving 1s off multi-sentence responses is noticeable. PR #6810 unblocks the "TTS works first time only" bug; this PR addresses the latency complaint.