Skip to content

feat(voice): WebRTC Realtime mode in voice conversation UI #7345

Description

@mrveiss

Background

useVoiceConversation.ts currently supports three modes: walkie-talkie, hands-free, full-duplex. All three pipe audio → backend Whisper → LLM → TTS, giving roughly 2-3s end-to-end latency per turn.

With the backend SDP proxy (#7342), MCP-to-Realtime tool bridge (#7343), and toolset bundles (#7344) in place, the frontend can run a fourth mode where a RTCPeerConnection streams audio directly to OpenAI Realtime, the model speaks/listens with sub-300ms turn-taking, and tool calls flow over the data channel back through AutoBot.

Goal

A new 'realtime-webrtc' ConversationMode with:

  • Browser RTCPeerConnection to OpenAI Realtime
  • SDP exchange via the backend proxy (no direct OpenAI traffic from the browser)
  • oai-events data channel for session.update, response.*, and tools/call
  • Tool list fetched from backend (feat(voice): MCP tools as Realtime function tools — sub-second tool calling in voice mode #7343) and registered via session.update after connection
  • Tool calls round-tripped through POST /api/voice/realtime/tools/call
  • Native barge-in (model interruption) handled by the RTC layer

Acceptance Criteria

  • New composable autobot-frontend/src/composables/useRealtimeVoice.ts (separation of concerns from useVoiceConversation.ts)
  • ConversationMode type extended: 'walkie-talkie' | 'hands-free' | 'full-duplex' | 'realtime-webrtc'
  • Connection lifecycle: getUserMedia → create peer + data channel → createOffer → POST to /api/voice/realtime/session → setRemoteDescription(answer) → register tools via session.update
  • Tool-call dispatcher: oai-events response.function_call_arguments.done → POST /api/voice/realtime/tools/call → emit result back over data channel
  • Mode toggle in voice UI (settings or per-session)
  • Microphone permission handling consistent with existing voice modes (uses the same secure-context check at useVoiceConversation.ts:48-52)
  • Disconnect cleanup: close peer, stop tracks, abort in-flight tool calls
  • Vitest covering: mode transition, SDP exchange (mocked fetch), tool registration message shape, tool-call routing, disconnect cleanup
  • Live test: in dev environment, voice round-trip < 500ms (manual verification with screen-recording or RTC stats)
  • No regression in the three existing voice modes (existing tests still pass)

Non-Goals

  • Replacing the existing voice modes (Realtime is opt-in for low-latency contexts)
  • Cost monitoring / token accounting in the UI (filed separately if needed)

Depends On

Files Affected

  • New: autobot-frontend/src/composables/useRealtimeVoice.ts (~250 LOC)
  • New: autobot-frontend/src/composables/__tests__/useRealtimeVoice.test.ts
  • Modify: autobot-frontend/src/composables/useVoiceConversation.ts (extend ConversationMode, add branch in state machine)
  • Modify: voice settings/mode-toggle component (locate via grep for ConversationMode in src/components/voice/)

Discovery Source

Filed during cross-codebase architecture research; AutoBot has the harder UX problem (3 modes, VAD, barge-in) already solved — adding WebRTC is the highest-leverage latency win available.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions