Skip to content

perf(voice): make _TTS_PIPELINE_DEPTH env-configurable; default 2 (was 3) — Pocket TTS is GIL-bound #6811

Description

@mrveiss

Problem

autobot-backend/api/voice_stream.py:100 hard-codes _TTS_PIPELINE_DEPTH = 3 (added in #6752 to absorb jitter via N-chunks-ahead pipelining).

But the TTS worker (tts-worker.py:285-287) calls Pocket TTS via loop.run_in_executor(None, _run_synthesis, ...). The default executor's threads run Python code that holds the GIL during synthesis (model inference is Python-level). Concurrent requests effectively serialize on the GIL/model lock. depth=3 queues 3 simultaneous synth tasks that take 3× synth time total, not 1× as true pipelining would.

Empirically: 3 concurrent synth = ~27s vs 3 sequential = ~11s in informal benchmark.

Hypothesis on right default

  • depth=1 — pure sequential, no pipelining; risk of audible gap if synth time > playback time of one chunk
  • depth=2 — synthesize chunk N+1 while client plays chunk N. Worker still GIL-serialized but the client gets continuous audio because chunk N+1 lands while N still plays.
  • depth=3 — schedules chunk N+2 as well, which only adds queue contention with no gain.

Recommended default: 2. Make env-configurable so ops can tune without redeploy:

_TTS_PIPELINE_DEPTH = int(os.getenv("AUTOBOT_TTS_PIPELINE_DEPTH", "2"))

Acceptance criteria

Out of scope (file follow-ups)

  • Benchmark across audio lengths and pick optimal default empirically
  • Investigate whether Pocket TTS releases GIL during inference (would change calculus)
  • Consider running multiple TTS worker processes for true parallelism

Why now

User reports voice latency: "it takes very long to respond with voice". Even shaving 1s off multi-sentence responses is noticeable. PR #6810 unblocks the "TTS works first time only" bug; this PR addresses the latency complaint.

Activity

  1. mrveiss commented on May 4, 2026

    @mrveiss
    OwnerAuthor

    ✅ Closed with proof of implementation

    Commit: b508487 — perf(voice): make _TTS_PIPELINE_DEPTH env-configurable; default 2 (#6811) (#6812)

    Acceptance Criteria Met:

    Resolver verified: all edge cases pass (default, min, max, clamp upper/lower, negative, invalid string).

    Awaiting deployment for empirical latency comparison vs prior depth=3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions