💡 Cloud Voice Provider Pipeline with Spatial Audio Support

[don-petry]

Summary

Extend the STT/TTS abstraction layer to support cloud voice providers (e.g., Deepgram, ElevenLabs, Azure Speech) as drop-in replacements for the MVP's Web Speech API, and explore spatial audio processing to improve voice capture in noisy environments. This unlocks higher-quality voice recognition, more natural avatar voices, and better noise handling for the target home-office and coffee-shop usage scenarios.

Market Signal

Voice interface usage in AI applications has grown 65% year-on-year according to 2026 VUI design research. Spatial Hearing AI is emerging as the foundation of the 2026 voice stack — moving beyond simple direction-of-arrival to pinpoint speakers in 3D space and separate them from overlapping voices (Kardome). ElevenLabs and Deepgram are the dominant voice API providers, with sub-200ms latency for streaming TTS. D-ID and LiveAvatar both ship real-time avatar voice with lip-sync quality that sets user expectations. The PRD's 3-second first-response target is achievable only with streaming cloud TTS — Web Speech API quality varies significantly across platforms.

User Signal

The PRD explicitly requires STT/TTS to be "configurable via abstraction layer — swappable without app-level code changes" (Implementation Considerations). FR7 requires transcription to begin within 500ms of silence detection. NFR1b requires first spoken response within 3 seconds via streaming TTS. The architecture doc identifies "Audio lifecycle management" as a cross-cutting concern. Web Speech API quality is inconsistent across macOS/Windows, making cloud providers essential for the cross-platform promise.

Technical Opportunity

The architecture defines SpeechToText and TextToSpeech as port interfaces in the Voice bounded context (renderer/speech/), with MVP implementations using Web Speech API. The ports-and-adapters pattern means cloud providers are literally designed as drop-in adapters. The IPC streaming pattern (event-based via contextBridge) already supports the streaming data flow needed for cloud STT/TTS. The renderer process handles voice, so no main-process changes are needed — just new adapter implementations behind existing interfaces.

Assessment

Dimension	Score	Rationale
Feasibility	high	Architecture explicitly designed for this; ports already defined; adapter pattern is straightforward
Impact	high	Voice quality directly affects core UX promise; cross-platform consistency is a real pain point
Urgency	med	Web Speech API works for MVP validation; cloud providers become critical at scale and for Windows parity

Suggested Next Step

Spike on ElevenLabs streaming TTS and Deepgram streaming STT as the first two cloud adapters. Benchmark latency against Web Speech API on macOS and Windows to quantify the quality gap. Design the provider selection UX (automatic vs. user-configurable in settings).

---

Proposed by the BMAD Analyst (Mary) on 2026-04-05.

[github-actions[bot]]

Research Enhancement — Cloud Voice Provider Pipeline with Spatial Audio Support

Sharpened Problem Framing

The voice pipeline is TalkTerm's primary interaction channel — and its quality ceiling is currently capped by Web Speech API, which has inconsistent quality across platforms (notably poor on Windows), no voice cloning, and no streaming viseme data for future lip-sync. The real problem is not "add cloud providers" — it is achieving the NFR targets (500ms STT start, 3s first spoken response) reliably across macOS and Windows while keeping the door open for avatar lip-sync via viseme streams.

Prior Art & Competitive Signal (2026)

Provider	STT Latency	TTS Latency	Key Differentiator	Pricing
ElevenLabs Scribe v2 Realtime	~150ms first-partial	—	90+ languages, real-time	$0.39/hr
ElevenLabs Flash v2.5	—	~75ms (model-only)	Lowest TTS latency; streaming viseme data available	$0.05/1K chars
Deepgram Flux	Sub-300ms streaming	—	Purpose-built for voice agents; endpointing + turn-detection signals	Pay-per-use
Azure Speech SDK	~200ms	~100ms	Viseme output for lip-sync (100+ languages); enterprise compliance	Tiered
Web Speech API (current)	Variable	Variable	Free, no API key needed	Free

Key finding: ElevenLabs and Deepgram are complementary — Deepgram for STT (best endpointing for barge-in) and ElevenLabs for TTS (lowest latency + viseme data for lip-sync). This "best of both" stack is explicitly recommended by both vendors.

Spatial Audio & Noise Handling Landscape

The spatial audio angle is worth reframing. The real user need is noise resilience in home offices and coffee shops, not spatial audio per se:

Technology	Approach	Integration Path
Krisp	On-device AI noise cancellation; works with any app	OS-level solution — TalkTerm could recommend it, not embed it
NVIDIA Broadcast	RTX GPU Tensor core noise suppression	Free for RTX users; OS-level, same as Krisp
Windows 11 Voice Focus	Built-in AI noise reduction on supported hardware	No integration needed — OS handles it
Deepgram Flux endpointing	Server-side voice activity detection + turn detection	Integrated into the STT adapter — better barge-in than local VAD
Kardome Spatial Hearing	3D acoustic scene mapping, speaker isolation	Enterprise SDK; likely overkill for single-user desktop app

Recommendation: Drop the "spatial audio" framing. Instead, focus on cloud STT providers with strong endpointing (Deepgram Flux) for barge-in reliability, and recommend OS-level noise cancellation (Krisp/NVIDIA Broadcast/Voice Focus) rather than building it in.

Options Table

Option	Pros	Cons	Effort
A. ElevenLabs TTS + Deepgram STT (both streaming)	Best-in-class latency; viseme data for lip-sync; proven voice-agent combo	Two vendor dependencies; ongoing API costs for users (BYOK)	Med
B. Azure Speech SDK (unified STT + TTS)	Single vendor; built-in viseme output; enterprise compliance	Higher latency than ElevenLabs; heavier SDK	Med
C. Provider-agnostic adapter layer (ship A first, support B later)	Future-proof; lets users choose by cost/quality preference	Slightly more upfront design work for the adapter interface	Med-High

Recommendation: Option C — but the SpeechToText and TextToSpeech port interfaces already exist in the architecture. Implement ElevenLabs TTS + Deepgram STT as the first cloud adapters (Option A), then add Azure as a second implementation. The port-adapter pattern means each new provider is isolated work.

Risks & Unknowns

• BYOK cost stacking: Users already pay for Claude API; adding voice API costs (ElevenLabs + Deepgram) creates a third cost stream. Consider making cloud voice opt-in with clear cost estimates, and keep Web Speech API as the free default.
• Latency budget: The 3s first-response target includes Claude API response time + TTS. If Claude takes 2s to stream the first sentence, TTS has only 1s — ElevenLabs Flash v2.5 at ~75ms model latency fits, but network overhead matters.
• Voice cloning / custom voices: ElevenLabs supports voice cloning for per-persona avatar voices (FR1-FR5). This is a powerful differentiator but raises ethical concerns — worth flagging in the product brief.
• Viseme data format: ElevenLabs and Azure use different viseme schemas. The adapter layer should normalize to a common Viseme type that the avatar lip-sync system consumes.

Phased Validation Path

1. Phase 1: Implement ElevenLabs TTS adapter behind existing TextToSpeech port. Benchmark latency vs Web Speech API on macOS + Windows. Ship as opt-in provider in settings.
2. Phase 2: Implement Deepgram STT adapter behind SpeechToText port. Focus on endpointing quality for barge-in (FR37). Compare with Web Speech API barge-in reliability.
3. Phase 3: Add normalized viseme output to the TTS adapter interface. Wire to avatar state machine for lip-sync (connects to Discussion 💡 3D Avatar Evolution with Lip-Sync and Emotional Expression #58).

Success Metrics

• Latency: Cloud TTS first-audio ≤500ms (p95) vs Web Speech API baseline
• STT accuracy: Cloud STT word error rate ≤5% vs Web Speech API baseline on same test corpus
• Cross-platform parity: Latency variance between macOS and Windows ≤20% (vs current Web Speech API variance)
• Barge-in reliability: False-positive barge-in rate ≤2% with Deepgram endpointing