You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(voice): WebRTC Realtime mode in voice conversation UI #7345
useVoiceConversation.ts currently supports three modes: walkie-talkie, hands-free, full-duplex. All three pipe audio → backend Whisper → LLM → TTS, giving roughly 2-3s end-to-end latency per turn.
With the backend SDP proxy (#7342), MCP-to-Realtime tool bridge (#7343), and toolset bundles (#7344) in place, the frontend can run a fourth mode where a RTCPeerConnection streams audio directly to OpenAI Realtime, the model speaks/listens with sub-300ms turn-taking, and tool calls flow over the data channel back through AutoBot.
Goal
A new 'realtime-webrtc'ConversationMode with:
Browser RTCPeerConnection to OpenAI Realtime
SDP exchange via the backend proxy (no direct OpenAI traffic from the browser)
oai-events data channel for session.update, response.*, and tools/call
Modify: autobot-frontend/src/composables/useVoiceConversation.ts (extend ConversationMode, add branch in state machine)
Modify: voice settings/mode-toggle component (locate via grep for ConversationMode in src/components/voice/)
Discovery Source
Filed during cross-codebase architecture research; AutoBot has the harder UX problem (3 modes, VAD, barge-in) already solved — adding WebRTC is the highest-leverage latency win available.
Background
useVoiceConversation.ts currently supports three modes:
walkie-talkie,hands-free,full-duplex. All three pipe audio → backend Whisper → LLM → TTS, giving roughly 2-3s end-to-end latency per turn.With the backend SDP proxy (#7342), MCP-to-Realtime tool bridge (#7343), and toolset bundles (#7344) in place, the frontend can run a fourth mode where a
RTCPeerConnectionstreams audio directly to OpenAI Realtime, the model speaks/listens with sub-300ms turn-taking, and tool calls flow over the data channel back through AutoBot.Goal
A new
'realtime-webrtc'ConversationModewith:RTCPeerConnectionto OpenAI Realtimeoai-eventsdata channel forsession.update,response.*, andtools/callsession.updateafter connectionPOST /api/voice/realtime/tools/callAcceptance Criteria
autobot-frontend/src/composables/useRealtimeVoice.ts(separation of concerns fromuseVoiceConversation.ts)ConversationModetype extended:'walkie-talkie' | 'hands-free' | 'full-duplex' | 'realtime-webrtc'getUserMedia→ create peer + data channel →createOffer→ POST to/api/voice/realtime/session→setRemoteDescription(answer)→ register tools viasession.updateoai-eventsresponse.function_call_arguments.done→POST /api/voice/realtime/tools/call→ emit result back over data channelNon-Goals
Depends On
Files Affected
autobot-frontend/src/composables/useRealtimeVoice.ts(~250 LOC)autobot-frontend/src/composables/__tests__/useRealtimeVoice.test.tsautobot-frontend/src/composables/useVoiceConversation.ts(extendConversationMode, add branch in state machine)ConversationModeinsrc/components/voice/)Discovery Source
Filed during cross-codebase architecture research; AutoBot has the harder UX problem (3 modes, VAD, barge-in) already solved — adding WebRTC is the highest-leverage latency win available.