Add end-to-end session lifecycle benchmarks - #1114
Conversation
BM-CONN decomposes connect() into its internal phases, which answers "where does connect time go" but not "how long does a user wait for the thing they asked for". BM-SESSION measures the milestones themselves, across two participants in one room: connect, peer visibility, publish to subscription, first frame, data round trip, and disconnect. Every metric comes from a single session per iteration, so the phase sums hold per-run rather than only in aggregate -- percentiles of separate metrics come from different runs and cannot be subtracted, which is easy to get wrong when reading the connect-phase table. Frames come from a buffer-backed track fed synthetic pixel buffers rather than a camera, so the benchmark runs headless without device permissions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 2 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| return try await withThrowingTaskGroup(of: Double.self) { group in | ||
| group.addTask { | ||
| try await withCheckedThrowingContinuation { continuation in | ||
| let fired: Double? = self._state.mutate { state in | ||
| if let fired = state.fired { return fired } | ||
| state.waiters.append(continuation) | ||
| return nil | ||
| } | ||
| if let fired { continuation.resume(returning: fired) } | ||
| } | ||
| } | ||
| group.addTask { | ||
| try await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000)) | ||
| throw BenchmarkTimeout() | ||
| } | ||
| defer { group.cancelAll() } | ||
| return try await group.next()! |
There was a problem hiding this comment.
🟡 Lifecycle timeouts hang forever
When an event never arrives, wait leaves its checked continuation suspended after timeout. The benchmark hangs forever instead of reporting failure.
Prompt for agents
Rework EventLatch.wait in Benchmarks/LiveKitBenchmark/BenchmarkSupport/SessionObservers.swift so timeout and task cancellation remove and resume the registered waiter. Cancelling the task-group child cannot cancel a checked continuation, and withThrowingTaskGroup waits for that child before leaving scope. Consider storing identifiable waiter records and resolving each continuation exactly once from fire, timeout, or cancellation. Ensure timeout actually returns BenchmarkTimeout and does not leave stale continuations in StateSync.
Was this helpful? React with 👍 or 👎 to provide feedback.
| try await publisher.connect(url: config.url, | ||
| token: tokenGen.generate(roomName: roomName, identity: "bench-pub")) | ||
| benchmark.measurement(mConnect, Int(nowMs() - connectStart)) | ||
|
|
||
| // peer_visible_ms — from the subscriber starting its connect to the publisher seeing it. | ||
| let peerStart = nowMs() | ||
| try await subscriber.connect(url: config.url, | ||
| token: tokenGen.generate(roomName: roomName, identity: "bench-sub")) |
There was a problem hiding this comment.
🟡 Failed sessions leave rooms connected
After either connection succeeds, any later thrown error skips both disconnect calls. Failed runs leave active rooms and media resources behind.
Prompt for agents
Add failure-safe asynchronous teardown around runSession in Benchmarks/LiveKitBenchmark/SessionLifecycleBenchmark.swift. Once either Room connects, every exit path must disconnect both publisher and subscriber, including failures from the second connect, publication, lifecycle waits, and data publishing. Preserve the existing disconnect_ms measurement for the successful publisher teardown and avoid measuring cleanup performed only after an error.
Was this helpful? React with 👍 or 👎 to provide feedback.
Correction and completed matrixThe results in the description were measured on
Three things this settles that the two-column table did not: 1. The stack does not touch dual PC. main+dual vs stack+dual is within noise on all seven metrics (561→556, 209→210, 220→222, 193→192, 415→412, 125→124, 6→5). Every change in the stack is single-PC-gated, and this is a much broader confirmation than the single 2. The connect win is the stack, not single PC. main+single is 554 — statistically the same as main+dual's 561. Only stack+single reaches 489. Single PC on its own buys nothing on connect; offer-with-join is the entire gain, and it only runs on the single-PC path. 3. The Net on |
1 participant
BM-CONNdecomposesconnect()into its internal phases — good for "where does connect time go", not for "how long does a user wait for the thing they asked for".BM-SESSIONmeasures the milestones themselves, across two participants in one room.Metrics
All wall-clock milliseconds, all from one session per iteration so the sums hold per-run:
connect_msroom.connect()call → returnpeer_visible_msconnect()call → publisher'sparticipantDidConnectpub_to_sub_mspublish(videoTrack:)call → subscriber'sdidSubscribeTrackfirst_frame_msdidSubscribeTrack→ first frame at the subscriber's rendererpublish_to_first_frame_mspublishcall → first frame (end to end)data_rtt_msdisconnect_msroom.disconnect()call → returnTwo variants,
BM-SESSION-001-DualPCandBM-SESSION-002-SinglePC, mirroringBM-CONN.Frames come from a buffer-backed track fed synthetic pixel buffers rather than a camera, so this runs headless with no device permissions.
First results (LiveKit Cloud staging, 15 iterations)
connect_mspeer_visible_mspub_to_sub_msfirst_frame_mspublish_to_first_frame_msdata_rtt_msdisconnect_msThe reason to have this benchmark, in one row: single PC buys 67 ms on connect and gives back 86 ms on publish→subscribe, netting exactly zero on time-to-first-frame. The connect-phase benchmark cannot see that, because the cost lands after
connect()returns.The
pub_to_subregression is theMediaSectionsRequirementround trip: in single PC the client must add receive m-sections and renegotiate before a subscription completes, where dual PC has the server offer on a separate transport. That is precisely what preallocating receive sections was meant to remove (#1112, currently blocked server-side).Usage
--filterneeds the exact full name. Note that switching branches underBenchmarks/(a path dependency) can leave a stale.buildreferencing files from the other branch;rm -rf Benchmarks/.buildclears it.Definitions worth confirming
These are modelled on a metric set from another SDK's harness, and three definitions are judgement calls I'd rather align than guess:
peer_visible_msincludes the subscriber's ownconnect(). Measuring only the propagation after connect returns is also defensible and would be a much smaller number.first_frame_msstarts atdidSubscribeTrackand attaches the renderer at that moment, so it includes encoder start and the first keyframe — hence ~100–200 ms rather than single digits. Starting the clock at renderer attach instead would report near-zero.disconnect_msis just the call duration (~5 ms); it does not wait for server-side teardown to be observable.Happy to re-cut any of them.
🤖 Generated with Claude Code