FFmpeg: Add Whip Muxer support for subsecond latency streaming #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
winlinvip
force-pushed
the
feature/rtc-muxer
branch
7 times, most recently
from
598aecc to
cf69b01
Compare
winlinvip
changed the title
winlinvip
changed the title
winlinvip
force-pushed
the
feature/rtc-muxer
branch
8 times, most recently
from
48d93eb to
92c4187
Compare
winlinvip
force-pushed
the
feature/rtc-muxer
branch
13 times, most recently
from
b4da552 to
2ad58fe
Compare
…cessary ClientHello.
1. Fix OpenSSL build error. 2. Support OpenSSL 1.0.1k and newer versions. 3. Support WHIP authorization via Bearer HTTP header. 4. Change the option default value from 1500 to 1200, to make Pion work. 5. Detect the minimum required OpenSSL version, should be 1.0.1k and newer. 6. Quickly check the SDP answer by taking a glance at the first few bytes.
1. Merge ICE and DTLS ARQ max retry options into a single handshake timeout. 2. Utilize DTLS server role to prevent ARQ, as the peer DTLS client will handle ARQ. 3. Replace IO from DTLSContext with a callback function. 4. Measure and analyze the time cost for each step in the process. 5. Implement DTLS BIO callback for packet fragmentation using BIO_set_callback. 6. Generate private key and certificate prior to ICE for faster handshake. 7. Refine DTLS MTU settings using SSL_set_mtu and DTLS_set_link_mtu. 8. Provide callback for DTLS state, returning errors when DTLS encounters issues or closes. 9. Consolidate ICE request/response handling and DTLS handshake into a single function.
1. Refine WHIP muxer name. 1. Refine SRTP key macros. 1. Refine logging context. 1. Refine SSL error messages. 1. Refine DTLS error messages. 1. Refine RTC error messages. 1. Use AV_RB8 to read integer from memory. 1. Update DTLS curve list to X25519:P-256:P-384:P-521. 1. Refine SRTP profile name for FFmpeg and OpenSSL. 1. Replace magic numbers with macros and extract to functions. 1. Alter log levels from INFO to VERBOSE, except for final results. 1. Use typedef SRTPContext. 1. Refine the ICE STUN magic number. 1. Reposition the on_rtp_write_packet function. 1. Refer to Chrome definition of RTP payload types. 1. Replace magic numbers with macros for RTP and RTCP payload types. 1. Rename to WHIP muxer. 1. Add TODO for OPUS timestamp issue. 1. Refine comments, do not hardcode H.264. 1. Define SDP session id and creator IP as macros. 1. Refine fixed frame size 960 to rtc->audio_par->frame_size. 1. Use h264_mp4toannexb to convert MP4/ISOM to annexb. 1. Address occasional inaccuracies in OPUS audio timestamps. 1. Correct marker setting after utilizing BSF. 1. Remove dependency on avc.h after using BSF.
…64_mp4toannexb filter only processing MP4 ISOM format.
1. Change the CommonName from ffmpeg.org to lavf. 2. Rename rtcenc.c to whip.c, rtc to whip. 3. Replace av_get_random_seed by AVLFG. 4. Add TODO to support libtls, mbedtls, and gnutls.
winlinvip
force-pushed
the
feature/rtc-muxer
branch
from
df62b77 to
63e3a55
Compare
|
Updated by #18: Feature: refactor DTLS to merge it into tls_openssl.c. This is a big step toward merging the WHIP patch into FFmpeg. I initialized this patch but was unable to finish it. However, after the efforts of FFmpeg developers and maintainers, especially the work from Jack Lau, Steven Liu, Sean Der, Mypopy Zhao, a new patch has been created and completed the last improvements for FFmpeg WHIP. With this patch, we are now ready to submit it to FFmpeg, and we will have a rudimentary WHIP implementation in FFmpeg. Regarding to the new patch, there will still be lots of work to do. We need your help to test and review this patch. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is updated by #18, #19, #24, #43, #44, #45.
This PR has been squashed into a single commit on the patch/whip/v1 branch for easier submission to the FFmpeg community for code review.
WHIP RFC9725 stands for the WebRTC-HTTP ingestion protocol, which is a sub-second streaming protocol designed by encoders and publishers. It is widely supported by various tools and media servers, allowing it to interact with other WebRTC clients, ingest streams to media servers, and is compatible with all modern browsers.
Unfortunately, most WHIP implementations are highly complex and require modern C++11 or C++14, or RUST. This complexity makes it impossible to integrate with FFmpeg, which requires C.
However, if FFmpeg were to incorporate WHIP support, it could be a game-changer for both FFmpeg and the WebRTC ecosystem, particularly for certain IoT or small devices that are too small to run modern languages but can run FFmpeg.
To meet FFmpeg's requirements, this PR contains just C code. And we have rewritten the WHIP and WebRTC protocol stack using only around 3k lines of C code.
Content
Usage
Migrated to #49
Usage: FFmpeg + SRS
Migrated to #37
Usage: FFmpeg + Janus
Migrated to #38
Usage: FFmpeg + Pion
Migrated to #39
Usage: FFmpeg + Millicast
Migrated to #46
Usage: FFmpeg + TRTC
Migrated to #47
Usage: FFmpeg + Cloudflare
Migrated to #41
Usage: FFmpeg + AWS-IVS
Migrated to #42
OpenSSL
Migrated to #33
Debugging
Migrated to #34
Asan and Fate
Migrated to #35
Simulate Packet Loss
Migrated to #36
Known Issues
Migrated to #28
Latency
Migrated to #29
Load Certificate File
Migrated to #30
Authorization
Migrated to #31
Contributors
Migrated to #32
Links