Skip to content

#538 Stamp stream video by delivery slot, not render frame number - #723

Merged
iamfatness merged 1 commit into
mainfrom
fix/538-slot-clock
Sep 30, 2026
Merged

iamfatness merged 1 commit into
mainfrom
fix/538-slot-clock

Conversation

@iamfatness

Copy link
Copy Markdown
Owner

Found by the first real-YouTube #703 run on beta-2026-09-30-ec70a7b: the exact packets sent to YouTube had video about 66 ms early against audio, while a local stand-in run of the same build was clean.

Root cause

Compositor frame numbers count rendered frames, but the program buffer delivers on a fixed slot grid. A render stall leaves slots without a new frame. In these runs it was a take opening the media clip at stream start, a stall of about 100 ms. Every later frame is then delivered that many slots later than its number says.

Stream video PTS was frameNumber / fps, so it silently absorbed the stall. Audio is sample-counted, so for the rest of the session video sat N frames early. The Program recording stamps each frame by delivery time, so it was unaffected. The clap gates anchor after startup and nothing stalled later, so they never saw it.

This existed before Slice 8, because video always used frame numbers. A mid-show render stall would cause the same offset.

How it was proven. A temporary frame-number label was drawn into Program pixels and a per-frame trace of K = timeline − frame/fps was logged. Neither is in this PR.

  • In bad sessions the labels still matched their PTS exactly, so the encoder labelling was right.
  • K stepped once by exactly the error:
Run RTMP − record K step after anchor
good +0.8 / +0.5 ms 0
bad −98.9 ms +100.0 ms (6 slots, delivery slots 26–31 missed)
bad −81.6 ms +83.3 ms (5 slots)

About half of all sessions started at core launch were bad.

Fix

  • ProgramSlotClock: each sender tracks K from the buffered frames it receives. Their timeline is the exact delivery deadline; unbuffered frames are ignored.
  • Anchor K: the audio packet now carries its anchor frame's timeline, so the writer knows which K the audio anchored on.
  • Correction: before queueing, each encoded frame is shifted by the whole slots K has moved since the anchor. The shift never decreases, so PTS stays monotonic, and a stall becomes a video PTS gap instead of an A/V offset.
  • Evidence: streamClock.videoSlotShift / videoSlotShifts, plus the log line [stream-clock] delivery grid moved N slot(s) at frame F.

Evidence (Windows Release, final build)

  • Repro sessions: 8 on the core-launch repro, RTMP−record 0.1–2.2 ms. 3 of them had 4-slot stalls; each was logged and corrected. Before the fix, stalled sessions measured −63 to −99 ms.
  • Native suite: 1351 passed, 0 failed. Two new tests: slot steps only on whole-slot moves, and the measured 6-slot stall is 100 ms uncorrected and within one sample corrected.
  • Shared-stream gates: block-rtmp, reconnect-rtmp and bitrate all pass.
  • 30 s gap gate: RTMP−Record 1.7 ms.
  • Stream confidence: measure decoded YouTube A/V sync against local monitoring #703 same-run harness, --local-standin ×2: RTMP−recording +1.5 and +2.2 ms.
  • 15-minute drift gate: 300/300 cues paired, RTMP−Record 0.4 ms, drift change 0.3 ms.

The real-YouTube leg of #703 still needs a live event; the first attempt's playback capture failed, most likely because the event ended when the encoder stopped.

🤖 Generated with Claude Code

Compositor frame numbers count RENDERED frames; the program buffer delivers
on a fixed slot grid. A render stall (a take opening media at stream start)
leaves slots without a new frame, and every later frame is delivered that
many slots later than its number says. Frame-number video PTS absorbed the
stall, so the stream sat 4-6 frames early against sample-counted audio for
the rest of the session: RTMP-record -63 to -99 ms in about half of all
sessions started at core launch, including the first real-YouTube #703 run.
The recording, stamped by delivery time, was unaffected.

Proven with a temporary frame-number label drawn into Program pixels (pixels
matched their PTS, so the encoder was right) and a per-frame trace of
K = timeline - frame/fps, which stepped by exactly the error.

ProgramSlotClock tracks K from the buffered frames each sender receives; the
writer shifts each encoded frame by the whole slots K moved since the audio
anchor (never decreasing), so a stall becomes a video PTS gap instead of a
permanent A/V offset. Evidence: streamClock.videoSlotShift/videoSlotShifts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamfatness
iamfatness merged commit c205e66 into main Sep 30, 2026
20 checks passed
@iamfatness
iamfatness deleted the fix/538-slot-clock branch September 30, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants