Skip to content

fix: drain media before BYE on local hangup to prevent voicemail clipping - #811

Open
rkfshakti wants to merge 3 commits into
livekit:mainfrom
rkfshakti:fix/hangup-media-drain
Open

rkfshakti wants to merge 3 commits into
livekit:mainfrom
rkfshakti:fix/hangup-media-drain

Conversation

@rkfshakti

Copy link
Copy Markdown

Fixes livekit/livekit#4737

Problem

When an agent ends a call right after wait_for_playout() returns — EndCall RPC, end_call tool, or removing the SIP participant — the last word of the utterance is clipped on the callee's end (voicemail scenario). The audio is already past the agent: it is buffered in the room mixer (5-frame ~100ms input buffer), the codec encoder, and in-flight RTP. wait_for_playout() only tracks the local queue, so it cannot see this tail.

The SIP bridge then sends BYE and closes media immediately, so whatever is still buffered never clears the wire. A trailing pause masks the bug because it gives the tail time to flush; an abrupt end-of-speech hangup clips the last word. The reporter's workaround (2s pre-hangup sleep) confirms the mechanism.

Fix

Add a configurable drain window hangup_drain_time (default 500ms): on a locally-initiated clean hangup, keep feeding media to the peer for the window before sending BYE. The window is bounded (default 500ms) and only applies to:

  • EndCall RPC (reason rpc)
  • ctx cancel / agent end-call (reason hangup)
  • participant removed from room (reason removed)
  • cancelled invite (inbound, reason cancelled)

Remote BYE (the peer hung up), errors, timeouts, media failures, and pre-connect failures are not delayed. Set hangup_drain_time: -1 to disable.

Test

TestOutboundHangupDrainsMediaBeforeBYE: on an established outbound call, an EndCall RPC must not emit BYE before the drain window has elapsed. Fails on the old code (BYE sent before the media drain window elapsed), passes with the fix. All media-port and outbound tests pass; the pre-existing auth-test failures on clean main are unrelated (they fail identically without this change).

devin-ai-integration[bot]

This comment was marked as resolved.

rkfshakti added a commit to rkfshakti/sip that referenced this pull request Aug 25, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti

Copy link
Copy Markdown
Author

Addressed the Devin review finding (cancelled inbound call delayed): drainOnHangup now gates on the media actually being bridged (c.started broken) and "cancelled" is dropped from the local-hangup set — a remote CANCEL before 200 OK is not a local hangup and gets no drain. Also constrained the media-port test range to 10000-20000 so CI can't land on a privileged port (<1024, seen on TestSetOfferReportsUnknownProvider: 'bind: permission denied').

@codecov

codecov Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.87500% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.25%. Comparing base (0460b40) to head (fd7db18).
⚠️ Report is 391 commits behind head on main.

Files with missing lines Patch % Lines
pkg/sip/inbound.go 66.66% 4 Missing and 1 partial ⚠️
pkg/config/config.go 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #811      +/-   ##
==========================================
+ Coverage   65.25%   69.25%   +3.99%     
==========================================
  Files          51       43       -8     
  Lines        6588     8736    +2148     
==========================================
+ Hits         4299     6050    +1751     
- Misses       1915     2163     +248     
- Partials      374      523     +149     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 2, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from a871b74 to 16668d2 Compare September 2, 2026 15:57
devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread pkg/sip/inbound.go
Comment thread pkg/sip/inbound.go
Comment thread pkg/sip/outbound.go
rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 5, 2026
Review feedback on livekit#811:

- inbound/outbound drain now uses select{ctx.Done, time.After} instead of
  time.Sleep so a context cancellation cuts the drain window short
  (inbound close() runs under WithoutCancel, so this preserves the full
  drain for clean hangups while allowing cancellation to pre-empt it).

- outboundCall.close() releases c.mu for the drain window: Close,
  CloseWith and CloseWithTimeout invoke close() while holding c.mu, and
  sleeping under the lock would block Participant() readers and a
  concurrent shutdown for the whole drain duration.
@rkfshakti

Copy link
Copy Markdown
Author

Pushed cab53c5 addressing both review comments:

  1. Cancellable drain — the drain window is now select { case <-ctx.Done(): case <-time.After(drain): } in both inbound and outbound paths, so context cancellation cuts the drain short instead of a blind time.Sleep.

  2. No sleeping under c.mu — outboundCall.close() releases c.mu before the drain wait and re-acquires it after, since Close/CloseWith/CloseWithTimeout all enter with the lock held. A 500ms drain no longer blocks Participant() readers or concurrent shutdown.

Verified: go build ./pkg/sip/ clean, go test ./pkg/sip/ -run 'TestOutbound|Drain|TestInbound' passes; the -race TestService_* failures reproduce on the unmodified head too (pre-existing, unrelated to this PR).

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@rkfshakti

Copy link
Copy Markdown
Author

Thanks @genseric-ghiro for the review. I merged the latest main and addressed each thread:

  • preserved the original caller context so cancellation/deadlines can cut the optional drain short
  • moved the BYE ordering comment next to the actual teardown call
  • serialized close ownership without holding c.mu during the drain or while competing closers wait
  • added concurrent-hangup regression coverage

The latest head (6bc104d) is mergeable, all checks are green, and all review threads are resolved. Please let me know if anything else is needed — I’m happy to contribute more.

rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 12, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from 6bc104d to bac9dc1 Compare September 12, 2026 17:16
devin-ai-integration[bot]

This comment was marked as resolved.

rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 13, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from bac9dc1 to fa26463 Compare September 13, 2026 17:01
rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 15, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from fa26463 to e2d85c0 Compare September 15, 2026 17:31
rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 16, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from e2d85c0 to 10384ca Compare September 16, 2026 16:46
rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 19, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from 10384ca to 6acf3b5 Compare September 19, 2026 15:53
rkfshakti added a commit to rkfshakti/sip that referenced this pull request Sep 23, 2026
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from 6acf3b5 to a17e816 Compare September 23, 2026 16:32
…ping

When an agent ends a call (EndCall RPC, end_call tool, or participant
removed from the room) right after wait_for_playout() returns, the last
word of the utterance is clipped on the callee's end. The audio is
already past the agent — it is buffered in the room mixer (~100ms input
buffer), the encoder, and in-flight RTP — but the SIP bridge sends BYE
and closes media immediately, so the tail never clears the wire (issue

Add a configurable hangup drain window (hangup_drain_time, default
500ms): on a locally-initiated clean hangup, keep feeding media for the
window before sending BYE. Remote BYE, errors, timeouts, and
pre-connect failures are not delayed.

Regression test: an EndCall RPC on an established outbound call must not
emit BYE before the drain window has elapsed. Fails on the old code
('BYE sent before the media drain window elapsed'), passes with the
fix.
…nprivileged range

Devin review feedback on PR livekit#811:

1. inbound drainOnHangup: a remote CANCEL before the call is answered
   hit the 'cancelled' branch and slept the drain window with no media
   bridged. Gate the drain on the media actually being bridged
   (c.started broken) and drop 'cancelled' from the local-hangup set.
   Remote CANCEL is not a local hangup.

2. newTestMediaPort bound a random even port in 1-65535, which can land
   on a privileged port (<1024) and fail in CI with 'bind: permission
   denied' (seen on TestSetOfferReportsUnknownProvider). Constrain the
   test range to 10000-20000.
Devin review feedback on PR livekit#811:

The outbound drainOnHangup returned the full drain window for
'hangup'/'rpc'/'removed' regardless of connection state, so a CANCEL
arriving during ringing waited HangupDrainTime with no media bridged.
Gate the drain on c.started (media bridged) — same gate the inbound
path got in the sibling commit — and add a regression test asserting
the gate flips on the started fuse, not the reason set.
@rkfshakti
rkfshakti force-pushed the fix/hangup-media-drain branch from a17e816 to fd7db18 Compare September 27, 2026 16:38
@rkfshakti

Copy link
Copy Markdown
Author

Hi @davidzhao @Cloudish9 — following up on this one. I've pushed the last open review finding: the outbound drainOnHangup now gates on media actually being bridged (c.started), matching the inbound gate from earlier — so a CANCEL during ringing is no longer delayed by the drain window. Regression test added (TestOutboundPreConnectHangupSkipsDrain); the full TestOutbound|Drain|TestInbound suites pass locally, with the branch rebased onto current main.

Quick context: I'm working from a fork (rkfshakti/sip), so a maintainer CI run is needed to kick off the E2E legs — that's the only gate left on my end. Everything else is green and mergeable. Given the repo's fork PR backlog, I'd love to focus on closing this one out — happy to split it into smaller pieces if that helps review. Thanks!

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread pkg/sip/inbound.go
// wire before the BYE tears the call down. Without this, the last
// word of a voicemail is clipped on abrupt hangups (issue #4737).
c.log().Debugw("draining media before hangup", "drain", drain)
time.Sleep(drain)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Shutdown strands draining inbound calls

If shutdown starts during time.Sleep, Shutdown cannot terminate the inbound call because done is already set. Server.Stop then closes SIP transport without waiting for BYE, leaving the peer connected.

Learn more

An inbound call marks itself done before the new drain window. Shutdown calls Shutdown on every registered call, but that method re-enters close, which returns immediately once done is set. Server.Stop then closes SIP transport while the original close is asleep. The original close later attempts CloseWithStatus, too late to reliably send BYE on that transport.

Example: A local EndCall begins a 500 ms drain. At 100 ms, service shutdown finds the call, gets an immediate return from Shutdown, and closes the SIP listener. At 500 ms the call attempts BYE; the peer may remain connected until its own timeout.

Recommended fix: Coordinate the in-progress close with Shutdown and Server.Stop. Interrupt the drain on shutdown and wait for the original close to finish sending BYE before closing signaling transport.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Outbound SIP: last word of voicemail clipped on hangup

2 participants