-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Open
Labels
maintainerLane: High-risk, cross-system changesLane: High-risk, cross-system changesmemoryLayer: Memory creation, syncing, storageLayer: Memory creation, syncing, storagep1Priority: Critical (score 22-29)Priority: Critical (score 22-29)
Description
Follow-up to PR #4784 (pusher memory leak fix — reduced pod memory ~80%). A deep audit of routers/pusher.py, routers/transcribe.py, and downstream utils found three remaining leak/backpressure patterns.
Current Behavior
- Unbounded audiobuffer growth (
routers/pusher.py:316-439):audiobuffer/trigger_audiobufferextend on every audio chunk but only cleared whenhas_audio_apps_enabledoraudio_bytes_webhook_delay_secondsis truthy. Users with no audio apps leak ~57MB/hour per connection. - Unbounded thread spawning (
utils/conversations/process_conversation.py:686-731): 7+ rawthreading.Thread().start()per conversation completion — no pooling, no rate limiting. Under sustained load, hundreds of concurrent threads pile up, each retaining a fullConversationobject. - Long external integration timeouts (
utils/app_integrations.py): 30s/15s/10s timeouts with thread-join blocking. A single slow app webhook blocks the entire pipeline, causing backpressure on upstream queues.
Expected Behavior
Audio buffers should only accumulate when there's a consumer. Background tasks should use a bounded thread pool. External integration timeouts should be short enough to prevent pipeline stalls.
Subtasks
- Fix 1: Guard audiobuffer — only extend when audio apps/webhook enabled (5-line fix in
pusher.py) - Fix 2: Replace per-conversation
threading.Thread()withThreadPoolExecutor(max_workers=32)inprocess_conversation.py - Fix 3: Reduce timeouts — 30s→10s (external), 15s→5s (audio bytes), 10s→5s (realtime)
Files to Modify
| File | Fix | Change |
|---|---|---|
backend/routers/pusher.py |
#1 | Guard audiobuffer.extend() / trigger_audiobuffer.extend() |
backend/utils/conversations/process_conversation.py |
#2 | ThreadPoolExecutor + executor.submit() |
backend/utils/app_integrations.py |
#3 | Reduce timeout= values |
Impact
Prevents additional memory growth paths not covered by PR #4784. Reduces thread count under load and limits backpressure from slow external apps.
Found during deep audit with Codex.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
maintainerLane: High-risk, cross-system changesLane: High-risk, cross-system changesmemoryLayer: Memory creation, syncing, storageLayer: Memory creation, syncing, storagep1Priority: Critical (score 22-29)Priority: Critical (score 22-29)