WHIP: Convert OBS WHIP to RTMP failed for no RTCP SR packets.

[winlinvip]

Reproduce steps:

1. Run SRS by rtc2rtmp.conf
2. Run OBS and publish WHIP to SRS.
3. Play HTTP-FLV, failed. http://localhost:8080/players/srs_player.html

No sequence header for RTMP or HTTP-FLV or HLS.

Issues

It appears that some issues need to be resolved in order to support the transmuxing of WebRTC to RTMP. The following is a pcap file captured by Wireshark, which includes ICE and RTP packets produced by OBS WHIP:

whip-ice-rtp.pcapng.zip

1. There is no RTCP Sender Report (SR) available to synchronize the audio and video timing. By default, SRS will drop the packet. For more information, refer to RTC: Fix rtc to rtmp sync timestamp using sender report. #2470
2. The SPS and PPS are sent as separate NALUs, with each NALU contained in an individual RTP packet. This causes a failure in constructing the sequence header from the RTP packet.
3. The IDR frame is divided into multiple slices, which must be combined to form a video frame for RTMP.

SR NTP Sync

Let's examine an example to understand how Chrome sends RTCP SR and how SRS synchronizes the timestamp.

srs.log.zip

Refer to issue #2470 for using NTP in RTCP SR to synchronize the timestamp. At startup, the timestamp has not yet been synchronized:

[2023-09-05 18:07:07.050] Audio async rate=0, rtp=4098464126
[2023-09-05 18:07:07.061] Video async rate=0, rtp=2819409850

It takes approximately 5 seconds to receive the SR and synchronize the timestamp, which is:

[2023-09-05 18:07:12.592] Video async rate=90, rtp=2819907460, corrected=3065188167
[2023-09-05 18:07:12.609] Audio async rate=48, rtp=4098731006, corrected=3065188190

(2819907460-2819409850)/90. = 5529.0
(4098731006-4098464126)/48. = 5560.0

Note: We also use timebase for calculating timestamps, with 90k for video and 48k for audio.

About 30s later, the timestamp:

[2023-09-05 18:07:37.313] Video async rate=90, rtp=2822132440, corrected=3065212889
[2023-09-05 18:07:37.329] Audio async rate=48, rtp=4099917566, corrected=3065212908

(2822132440-2819409850)/90. = 30251.0
(4099917566-4098464126)/48. = 30280.0

About 1m, the timestamp:

[2023-09-05 18:08:07.201] Video async rate=90, rtp=2824822270, corrected=3065242775
[2023-09-05 18:08:07.210] Audio async rate=48, rtp=4101351806, corrected=3065242787

(2824822270-2819409850)/90. = 60138.0
(4101351806-4098464126)/48. = 60160.0

About 2m, the timestamp:

[2023-09-05 18:09:07.430] Audio async rate=48, rtp=4104242366, corrected=3065303005
[2023-09-05 18:09:07.444] Video async rate=90, rtp=2830243960, corrected=3065303016

(2830243960-2819409850)/90. = 120379.0
(4104242366-4098464126)/48. = 120380.0

About 12m, the timestamp:

[2023-09-05 18:19:07.531] Audio async rate=48, rtp=4133047166, corrected=3065903092
[2023-09-05 18:19:07.544] Video async rate=90, rtp=2884251520, corrected=3065903100

(4133047166-4098464126)/48. = 720480.0
(2884251520-2819409850)/90. = 720463.0

About 17m, the timestamp:

[2023-09-05 18:24:51.393] Video async rate=90, rtp=2915198560, corrected=3066246956
[2023-09-05 18:24:51.396] Audio async rate=48, rtp=4149553406, corrected=3066246963

(2915198560-2819409850)/90. = 1064319.0
(4149553406-4098464126)/48. = 1064360.0

Please see this post for detail.

[littleq0903]

thanks @winlinvip for working on this.

[winlinvip]

OBS WHIP cannot convert to RTMP due to multiple issues, the biggest one being that OBS WHIP lacks RTCP SR, making it impossible to synchronize audio and video timing. Although there are some ways to work around this, such as using the sampling rate to calculate timestamps, it is expected that there will be issues with audio and video synchronization, and it cannot be completely bypassed. Therefore, let's put this issue aside for now and wait for OBS WHIP to update and support RTCP SR before considering it.

PS: Using the sampling rate to calculate timestamps can be risky on some mobile devices where the sampling rate is not accurate. However, since OBS generally runs on PCs, this solution has the potential to solve the problem, but it requires development and testing.

TRANS_BY_GPT4

[winlinvip]

Update:

OBS version 30.0.2 now includes RTCP SR, and the issue with timestamp synchronization has been resolved.

(lldb) x/70xb data
0x63100003c800: 0x80 0xc8 0x00 0x06 0xab 0xc9 0xed 0xdf
0x63100003c808: 0x5f 0x72 0xa6 0x57 0xf1 0xed 0x58 0x46
0x63100003c810: 0xeb 0xe0 0xab 0x1b 0xcb 0x59 0xd5 0x71
0x63100003c818: 0x49 0x43 0xa9 0xd3 0xae 0xec 0x12 0x82
0x63100003c820: 0xd3 0x9e 0x23 0x10 0xfa 0xf7 0x3b 0x84
0x63100003c828: 0xfd 0x92 0x54 0x29 0x84 0x5c 0x94 0xe6
0x63100003c830: 0xd9 0x3e 0x49 0xcc 0x84 0x38 0xae 0x82
0x63100003c838: 0x80 0x00 0x00 0x01 0xf4 0x71 0xab 0xea
0x63100003c840: 0xeb 0xdb 0xe2 0x7a 0x60 0x53

(lldb) x/56xb unprotected_buf
0x63100003c800: 0x80 0xc8 0x00 0x06 0xab 0xc9 0xed 0xdf
0x63100003c808: 0xe9 0x25 0x28 0x01 0x76 0x36 0x80 0x00
0x63100003c810: 0x74 0xc0 0xa2 0x07 0x00 0x00 0x00 0x33
0x63100003c818: 0x00 0x00 0x4f 0x01 0x81 0xca 0x00 0x06
0x63100003c820: 0xab 0xc9 0xed 0xdf 0x01 0x10 0x51 0x6c
0x63100003c828: 0x59 0x58 0x4b 0x4f 0x72 0x30 0x4e 0x4a
0x63100003c830: 0x6b 0x5a 0x76 0x31 0x6d 0x73 0x00 0x00

SRS has successfully decrypted and processed the data.

SRS has added a log entry indicating that the timestamp has been synchronized. See 84d1a33

[2023-12-14 15:39:33.194][WARN][80102][93j9973x][35] RTC: Discard for no-sync, Video ssrc=2882137568, seq=26689, ts=3523645489, state=-1
[2023-12-14 15:39:35.237][WARN][80102][93j9973x][35] RTC: Accept for sync, Audio ssrc=2882137567, seq=10520, ts=2630565740, state=0

Approximately 2 seconds into the stream, the timestamp will be fully synchronized.

PS: Currently, there is an issue with SPS/PPS that is preventing the conversion from WHIP to RTMP. The issue is being addressed.

TRANS_BY_GPT4