Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FFmpeg: Add Whip Muxer support for subsecond latency streaming #1

Open
wants to merge 60 commits into
base: feature/whip
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
ae07b06
WHIP: Add WebRTC WHIP muxer.
winlinvip Apr 16, 2023
f090d64
WHIP: Only support h264+opus codec.
winlinvip Apr 16, 2023
edaf213
WHIP: Generate and exchange offer server to get answer.
yangrtc Apr 21, 2023
83202f7
WHIP: Support parse ice from answer.
winlinvip Apr 22, 2023
8a63f74
WHIP: Refine code.
winlinvip Apr 22, 2023
67bccbf
WHIP: Support ICE handshake by binding.
winlinvip May 1, 2023
9db6961
WHIP: Support fast retransmit for binding request.
winlinvip May 1, 2023
1496684
WHIP: Update the dependency, requires openssl.
winlinvip May 2, 2023
0522db8
WHIP: Support DTLS handshake by openssl
winlinvip May 2, 2023
4ee4ca5
WHIP: Setup SRTP with key material generated by DTLS
winlinvip May 3, 2023
67f9942
WHIP: Refine code.
winlinvip May 3, 2023
ff5ef96
WHIP: Support mux and send audio by RTP
winlinvip May 3, 2023
98fca74
WHIP: Support mux and send video by RTP
winlinvip May 4, 2023
9ecbef0
WHIP: Support audio or video only
winlinvip May 4, 2023
40c385a
WHIP: Support video annexb format.
winlinvip May 4, 2023
fc0974a
WHIP: Refine code.
winlinvip May 5, 2023
fae7b40
WHIP: Dispose resource when write trailer.
winlinvip May 5, 2023
5e400cd
WHIP: Check alloc fail and return EMOMEM.
winlinvip May 5, 2023
d9c7dc7
WHIP: Always dispose resource by WHIP.
winlinvip May 5, 2023
235aaa0
WHIP: Refine macros.
winlinvip May 6, 2023
34fd9f9
WHIP: Support set parameters by options.
winlinvip May 6, 2023
2bfdb67
WHIP: Eliminate the unused write_trailer.
winlinvip May 6, 2023
e3fdedd
WHIP: Refine code.
winlinvip May 6, 2023
b4574d6
WHIP: Check for null string to avoid strlen crash, scanned by ASAN
winlinvip May 11, 2023
248243a
WHIP: Improve macro and options comments with more background details
winlinvip May 11, 2023
d801061
WHIP: Remote password should be used to generate the message integrity.
winlinvip May 12, 2023
a1a8335
Use AVPrint to generate the offer and answer for WebRTC SDP.
winlinvip May 12, 2023
ff72920
WHIP:Support send sctp
duiniuluantanqin May 12, 2023
fc3a4a2
WHIP: Refining SRTP context selection.
winlinvip May 12, 2023
45e6ece
WHIP: Refine code.
winlinvip May 13, 2023
6f965d6
Add bidirectional ICE binding, DTLS with ECDSA, and compatibility imp…
cloudwebrtc May 13, 2023
3fd6767
Add missing quotation string to error message
winlinvip May 15, 2023
68ef7ad
WHIP: Extract DTLSContext from RTContext.
winlinvip May 17, 2023
c7b146b
WHIP: Reorder functions, nothing changed.
winlinvip May 17, 2023
2dedc86
WHIP: Remove the macro that suppresses warnings for the SSL deprecate…
winlinvip May 18, 2023
1054bed
WHIP: Merge write header to init.
winlinvip May 18, 2023
b31e0c0
WHIP:Support baseline/main/high profile without B frames (#2)
duiniuluantanqin May 19, 2023
b3f5c27
WHIP: Fix the SSL deprecated warning by replacing EC_KEY_new with EVP…
winlinvip May 23, 2023
85c5680
WHIP: Refine the code to be shorter.
winlinvip May 23, 2023
71ee877
WHIP: Increase the base timeout and thereby reduce the number of unne…
winlinvip May 23, 2023
726bb67
WHIP: Refine ARQ for DTLS with bug fixed.
winlinvip May 23, 2023
6e1fbb5
WHIP: Update muxers.texi for RTC.
winlinvip May 29, 2023
86b361c
WHIP: Fix bugs causing build OpenSSL error, to work with Pion
winlinvip May 30, 2023
ea74e3c
Optimize DTLS Handshake and ICE Handling for Improved Performance (#3)
winlinvip Jun 7, 2023
680ed49
Roundtrip3: Refinements Based on Derek Buitenhuis' Comments (#4)
winlinvip Jun 9, 2023
2ac4100
WHIP: Do not hardcode codec for SDP. (#5)
winlinvip Jun 9, 2023
5819297
WHIP: Parse profile and level from extradata.
winlinvip Jun 9, 2023
0c4a158
WHIP: Insert SPS and PPS before IDR frames in annexb format due to h2…
winlinvip Jun 10, 2023
812932e
WHIP: Failed immediately if there is no profile and extradata present.
winlinvip Jun 10, 2023
aaf26ef
WHIP: Refine names and comments.
winlinvip Jun 19, 2023
66a064f
WHIP: Free buffer leak of pb for RTP muxer.
winlinvip Jun 20, 2023
9c7a091
WHIP: Use options for passing parameters to the HTTP or UDP context.
winlinvip Jun 21, 2023
de33fcc
WHIP: Enhance security by using BearToken for delete API.
winlinvip Jun 21, 2023
8971a1f
WHIP: Support user specified certificate and key file.
winlinvip Jun 21, 2023
0bd6867
WHIP: Read DTLS certificate file by ffurl.
winlinvip Jul 4, 2023
0b95312
WHIP: Free bio_in if ENOMEM.
winlinvip Jul 8, 2023
63e3a55
WHIP: Refine DTLS, extract DTLS APIs.
winlinvip Aug 24, 2023
172d34b
Sync with FFmpeg master 5ddab49d48
winlinvip Oct 17, 2023
d3561e6
Extract DTLS to dtls.c
winlinvip Oct 17, 2023
3b15ba1
Change some important step log level to info.
winlinvip Oct 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WHIP: Refine code.
  • Loading branch information
winlinvip committed Oct 17, 2023
commit 67f99427dd495b9b9b18e71444f2fa89ed218049
112 changes: 56 additions & 56 deletions libavformat/rtcenc.c
Original file line number Diff line number Diff line change
Expand Up @@ -73,58 +73,58 @@
typedef struct RTCContext {
AVClass *av_class;

/* input audio and video codec parameters */
/* Input audio and video codec parameters */
AVCodecParameters *audio_par;
AVCodecParameters *video_par;

/* the ICE username and pwd fragment generated by the muxer. */
/* The ICE username and pwd fragment generated by the muxer. */
char ice_ufrag_local[9];
char ice_pwd_local[33];
/* the SSRC of the audio and video stream, generated by the muxer. */
/* The SSRC of the audio and video stream, generated by the muxer. */
uint32_t audio_ssrc;
uint32_t video_ssrc;
/* the PT(Payload Type) of stream, generated by the muxer. */
/* The PT(Payload Type) of stream, generated by the muxer. */
uint8_t audio_payload_type;
uint8_t video_payload_type;
/**
* the SDP offer generated by the muxer according to the codec parameters,
* The SDP offer generated by the muxer according to the codec parameters,
* DTLS and ICE information.
* */
char *sdp_offer;

/* the ICE username and pwd from remote server. */
/* The ICE username and pwd from remote server. */
char *ice_ufrag_remote;
char *ice_pwd_remote;
/**
* the ICE candidate protocol, priority, host and port. Note that only
* The ICE candidate protocol, priority, host and port. Note that only
* support one candidate for now. We will choose the first udp candidate.
* We will support multiple candidates in the future.
*/
char *ice_protocol;
char *ice_host;
int ice_port;
/* the SDP answer received from the WebRTC server. */
/* The SDP answer received from the WebRTC server. */
char *sdp_answer;

/* whether the timer should be reset. */
/* Whether the timer should be reset. */
int dtls_should_reset_timer;
/* whether the DTLS is done at least for us. */
/* Whether the DTLS is done at least for us. */
int dtls_done_for_us;
/* the number of packets retransmitted for DTLS. */
/* The number of packets retransmitted for DTLS. */
int dtls_arq_packets;
/**
* the material to build SRTP master key, generated by DTLS, the layout is:
* The material to build SRTP master key, generated by DTLS, the layout is:
* 16B 16B 14B 14B
* client_key | server_key | client_salt | server_salt
*/
uint8_t dtls_srtp_material[DTLS_SRTP_MASTER_KEY_LEN * 2];

/* the SRTP send context, to encrypt outgoing packets. */
/* The SRTP send context, to encrypt outgoing packets. */
struct SRTPContext srtp_send;
/* the SRTP receive context, to decrypt incoming packets. */
/* The SRTP receive context, to decrypt incoming packets. */
struct SRTPContext srtp_recv;

/* the UDP transport is used for delivering ICE, DTLS and SRTP packets. */
/* The UDP transport is used for delivering ICE, DTLS and SRTP packets. */
URLContext *udp_uc;
} RTCContext;

Expand Down Expand Up @@ -388,7 +388,7 @@ static int exchange_sdp(AVFormatContext *s)
int ret;
char buf[MAX_URL_SIZE];
RTCContext *rtc = s->priv_data;
/* the URL context is an HTTP transport layer for the WHIP protocol. */
/* The URL context is an HTTP transport layer for the WHIP protocol. */
URLContext *whip_uc = NULL;

char *tmp = av_mallocz(MAX_SDP_SIZE);
Expand Down Expand Up @@ -419,7 +419,7 @@ static int exchange_sdp(AVFormatContext *s)
while (1) {
ret = ffurl_read(whip_uc, buf, sizeof(buf));
if (ret == AVERROR_EOF) {
/* reset the error because we read all response as answer util EOF. */
/* Reset the error because we read all response as answer util EOF. */
ret = 0;
break;
}
Expand Down Expand Up @@ -537,15 +537,15 @@ static int ice_create_request(AVFormatContext *s, uint8_t *buf, int size_of_buf,
goto end;
}

/* write 20 bytes header */
/* Write 20 bytes header */
avio_wb16(pb, 0x0001); /* STUN binding request */
avio_wb16(pb, 0); /* length */
avio_wb32(pb, 0x2112A442); /* magic cookie */
avio_wb32(pb, av_get_random_seed()); /* transaction ID */
avio_wb32(pb, av_get_random_seed()); /* transaction ID */
avio_wb32(pb, av_get_random_seed()); /* transaction ID */

/* the username is the concatenation of the two ICE ufrag */
/* The username is the concatenation of the two ICE ufrag */
ret = snprintf(username, sizeof(username), "%s:%s", rtc->ice_ufrag_remote, rtc->ice_ufrag_local);
if (ret <= 0 || ret >= sizeof(username)) {
av_log(s, AV_LOG_ERROR, "Failed to build username %s:%s, max=%lu, ret=%d\n",
Expand All @@ -554,13 +554,13 @@ static int ice_create_request(AVFormatContext *s, uint8_t *buf, int size_of_buf,
goto end;
}

/* write the username attribute */
/* Write the username attribute */
avio_wb16(pb, 0x0006); /* attribute type username */
avio_wb16(pb, ret); /* size of username */
avio_write(pb, username, ret); /* bytes of username */
ffio_fill(pb, 0, (4 - (ret % 4)) % 4); /* padding */

/* build and update message integrity */
/* Build and update message integrity */
avio_wb16(pb, 0x0008); /* attribute type message integrity */
avio_wb16(pb, 20); /* size of message integrity */
ffio_fill(pb, 0, 20); /* fill with zero to directly write and skip it */
Expand All @@ -571,14 +571,14 @@ static int ice_create_request(AVFormatContext *s, uint8_t *buf, int size_of_buf,
av_hmac_update(hmac, buf, size - 24);
av_hmac_final(hmac, buf + size - 20, 20);

/* write the fingerprint attribute */
/* Write the fingerprint attribute */
avio_wb16(pb, 0x8028); /* attribute type fingerprint */
avio_wb16(pb, 4); /* size of fingerprint */
ffio_fill(pb, 0, 4); /* fill with zero to directly write and skip it */
size = avio_tell(pb);
buf[2] = (size - 20) >> 8;
buf[3] = (size - 20) & 0xFF;
/* refer to the av_hash_alloc("CRC32"), av_hash_init and av_hash_final */
/* Refer to the av_hash_alloc("CRC32"), av_hash_init and av_hash_final */
crc32 = av_crc(av_crc_get_table(AV_CRC_32_IEEE_LE), 0xFFFFFFFF, buf, size - 8) ^ 0xFFFFFFFF;
avio_skip(pb, -4);
avio_wb32(pb, crc32 ^ 0x5354554E); /* xor with "STUN" */
Expand Down Expand Up @@ -620,33 +620,33 @@ static int ice_handshake(AVFormatContext *s)
goto end;
}

/* make the socket non-blocking, set to READ and WRITE mode after connected */
/* Make the socket non-blocking, set to READ and WRITE mode after connected */
ff_socket_nonblock(ffurl_get_file_handle(rtc->udp_uc), 1);
rtc->udp_uc->flags |= AVIO_FLAG_READ | AVIO_FLAG_NONBLOCK;

/* build the STUN binding request. */
/* Build the STUN binding request. */
ret = ice_create_request(s, buf, sizeof(buf), &size);
if (ret < 0) {
av_log(s, AV_LOG_ERROR, "Failed to create STUN binding request, size=%d\n", size);
goto end;
}

/* fast retransmit the STUN binding request. */
/* Fast retransmit the STUN binding request. */
do {
ret = ffurl_write(rtc->udp_uc, buf, size);
if (ret < 0) {
av_log(s, AV_LOG_ERROR, "Failed to send STUN binding request, size=%d\n", size);
goto end;
}

/* if max retries is 6 and start timeout is 21ms, the total timeout
/* If max retries is 6 and start timeout is 21ms, the total timeout
* is about 21 + 42 + 84 + 168 + 336 + 672 = 1263ms. */
if (fast_retries) {
av_usleep(timeout * 1000);
timeout *= 2;
}

/* read the STUN binding response. */
/* Read the STUN binding response. */
ret = ffurl_read(rtc->udp_uc, buf, sizeof(buf));
if (ret < 0) {
if (ret == AVERROR(EAGAIN) && fast_retries) {
Expand Down Expand Up @@ -730,10 +730,10 @@ static unsigned int openssl_dtls_timer_cb(SSL *dtls, unsigned int previous_us)
AVFormatContext *s = (AVFormatContext*)SSL_get_ex_data(dtls, 0);
RTCContext *rtc = s->priv_data;

/* double the timeout, note that it may be 0. */
/* Double the timeout, note that it may be 0. */
unsigned int timeout_us = previous_us * 2;

/* if previous_us is 0, for example, the HelloVerifyRequest, we should respond it ASAP.
/* If previous_us is 0, for example, the HelloVerifyRequest, we should respond it ASAP.
* when got ServerHello, we should reset the timer. */
if (!previous_us || rtc->dtls_should_reset_timer) {
timeout_us = 50 * 1000; /* in us */
Expand All @@ -755,7 +755,7 @@ static void openssl_state_trace(AVFormatContext *s, uint8_t *data, int length, i
uint8_t handshake_type = 0;
RTCContext *rtc = s->priv_data;

/* change_cipher_spec(20), alert(21), handshake(22), application_data(23) */
/* Change_cipher_spec(20), alert(21), handshake(22), application_data(23) */
if (length >= 1) {
content_type = (uint8_t)data[0];
}
Expand Down Expand Up @@ -784,7 +784,7 @@ static void openssl_state_trace(AVFormatContext *s, uint8_t *data, int length, i
*/
static int openssl_verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
{
/* always OK, we don't check the certificate of client, because we allow client self-sign certificate. */
/* Always OK, we don't check the certificate of client, because we allow client self-sign certificate. */
return 1;
}

Expand All @@ -797,7 +797,7 @@ static av_cold int openssl_init_dtls(AVFormatContext *s, SSL *dtls, SSL_CTX *dtl
{
int ret;

/* should use the curves in ClientHello.supported_groups, for example:
/* Should use the curves in ClientHello.supported_groups, for example:
* Supported Group: x25519 (0x001d)
* Supported Group: secp256r1 (0x0017)
* Supported Group: secp384r1 (0x0018)
Expand All @@ -821,42 +821,42 @@ static av_cold int openssl_init_dtls(AVFormatContext *s, SSL *dtls, SSL_CTX *dtl
goto end;
}

/* for ECDSA, we could set the curves list. */
/* For ECDSA, we could set the curves list. */
SSL_CTX_set1_curves_list(dtls_ctx, "P-521:P-384:P-256");

/* we use "ALL", while you can use "DEFAULT" means "ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2" */
/* We use "ALL", while you can use "DEFAULT" means "ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2" */
if (SSL_CTX_set_cipher_list(dtls_ctx, DTLS_CIPHER_SUTES) != 1) {
av_log(s, AV_LOG_ERROR, "DTLS: SSL_CTX_set_cipher_list failed\n");
ret = AVERROR(EINVAL);
goto end;
}
/* server will send Certificate Request. */
/* Server will send Certificate Request. */
SSL_CTX_set_verify(dtls_ctx, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, openssl_verify_callback);
/* the depth count is "level 0:peer certificate", "level 1: CA certificate",
/* The depth count is "level 0:peer certificate", "level 1: CA certificate",
* "level 2: higher level CA certificate", and so on. */
SSL_CTX_set_verify_depth(dtls_ctx, 4);
/* whether we should read as many input bytes as possible (for non-blocking reads) or not. */
/* Whether we should read as many input bytes as possible (for non-blocking reads) or not. */
SSL_CTX_set_read_ahead(dtls_ctx, 1);
/* only support SRTP_AES128_CM_SHA1_80, please read ssl/d1_srtp.c */
/* Only support SRTP_AES128_CM_SHA1_80, please read ssl/d1_srtp.c */
ret = SSL_CTX_set_tlsext_use_srtp(dtls_ctx, "SRTP_AES128_CM_SHA1_80");
if (ret) {
av_log(s, AV_LOG_ERROR, "DTLS: SSL_CTX_set_tlsext_use_srtp failed, ret=%d\n", ret);
ret = AVERROR(EINVAL);
goto end;
}

/* setup the callback for logging. */
/* Setup the callback for logging. */
SSL_set_ex_data(dtls, 0, s);
SSL_set_info_callback(dtls, openssl_on_info);

/* set dtls fragment size */
/* Set dtls fragment size */
SSL_set_options(dtls, SSL_OP_NO_QUERY_MTU);
SSL_set_mtu(dtls, DTLS_FRAGMENT_MAX_SIZE);

/* set the callback for ARQ timer. */
/* Set the callback for ARQ timer. */
DTLS_set_timer_cb(dtls, openssl_dtls_timer_cb);

/* setup DTLS as active, which is client role. */
/* Setup DTLS as active, which is client role. */
SSL_set_connect_state(dtls);
SSL_set_max_send_fragment(dtls, DTLS_FRAGMENT_MAX_SIZE);

Expand All @@ -872,23 +872,23 @@ static int openssl_drive_context(AVFormatContext *s, SSL *dtls, BIO *bio_in, BIO
char buf[MAX_UDP_SIZE];
RTCContext *rtc = s->priv_data;

/* drive the SSL context by state change, arq or response messages. */
/* Drive the SSL context by state change, arq or response messages. */
r0 = SSL_do_handshake(dtls);
r1 = SSL_get_error(dtls, r0);

/* handshake successfully done */
/* Handshake successfully done */
if (r0 == 1) {
rtc->dtls_done_for_us = 1;
return 0;
}

/* handshake failed with fatal error */
/* Handshake failed with fatal error */
if (r0 < 0 && r1 != SSL_ERROR_WANT_READ) {
av_log(s, AV_LOG_ERROR, "DTLS: Start handshake failed, loop=%d, r0=%d, r1=%d\n", loop, r0, r1);
return AVERROR(EIO);
}

/* fast retransmit the request util got response. */
/* Fast retransmit the request util got response. */
for (i = 0; i < UDP_FAST_RETRIES && !res_size; i++) {
req_size = BIO_get_mem_data(bio_out, (char**)&data);
openssl_state_trace(s, data, req_size, 0, r0, r1);
Expand All @@ -904,14 +904,14 @@ static int openssl_drive_context(AVFormatContext *s, SSL *dtls, BIO *bio_in, BIO

for (j = 0; j < UDP_FAST_RETRIES && !res_size; j++) {
ret = ffurl_read(rtc->udp_uc, buf, sizeof(buf));
/* got response successfully. */
/* Got response successfully. */
if (ret > 0) {
res_size = ret;
rtc->dtls_should_reset_timer = 1;
break;
}

/* fatal error or timeout. */
/* Fatal error or timeout. */
if (ret != AVERROR(EAGAIN)) {
av_log(s, AV_LOG_ERROR, "DTLS: Read response failed, loop=%d, content=%d, handshake=%d\n",
loop, req_ct, req_ht);
Expand Down Expand Up @@ -939,7 +939,7 @@ static int openssl_drive_context(AVFormatContext *s, SSL *dtls, BIO *bio_in, BIO
}
}

/* trace the response packet, feed to SSL. */
/* Trace the response packet, feed to SSL. */
BIO_reset(bio_in);
openssl_state_trace(s, buf, res_size, 1, r0, SSL_ERROR_NONE);
res_ct = res_size > 0 ? buf[0]: 0;
Expand Down Expand Up @@ -969,7 +969,7 @@ static int openssl_dtls_handshake(AVFormatContext *s)
const char* dst = "EXTRACTOR-dtls_srtp";
RTCContext *rtc = s->priv_data;

/* create and initialize SSL context. */
/* Create and initialize SSL context. */
dtls_pkey = EVP_PKEY_new();
eckey = EC_KEY_new();

Expand Down Expand Up @@ -999,7 +999,7 @@ static int openssl_dtls_handshake(AVFormatContext *s)
goto end;
}

/* export SRTP master key after DTLS done */
/* Export SRTP master key after DTLS done */
ret = SSL_export_keying_material(dtls, rtc->dtls_srtp_material, sizeof(rtc->dtls_srtp_material),
dst, strlen(dst), NULL, 0, 0);
if (!ret) {
Expand Down Expand Up @@ -1035,15 +1035,15 @@ static int setup_srtp(AVFormatContext *s)
const char* suite = "AES_CM_128_HMAC_SHA1_80";
RTCContext *rtc = s->priv_data;

/* as DTLS client, the send key is client master key plus salt. */
/* As DTLS client, the send key is client master key plus salt. */
memcpy(send_key, rtc->dtls_srtp_material, 16);
memcpy(send_key + 16, rtc->dtls_srtp_material + 32, 14);

/* as DTLS client, the recv key is server master key plus salt. */
/* As DTLS client, the recv key is server master key plus salt. */
memcpy(recv_key, rtc->dtls_srtp_material + 16, 16);
memcpy(recv_key + 16, rtc->dtls_srtp_material + 46, 14);

/* setup SRTP context for outgoing packets */
/* Setup SRTP context for outgoing packets */
if (!av_base64_encode(buf, sizeof(buf), send_key, sizeof(send_key))) {
av_log(s, AV_LOG_ERROR, "Failed to encode send key\n");
ret = AVERROR(EIO);
Expand All @@ -1056,7 +1056,7 @@ static int setup_srtp(AVFormatContext *s)
goto end;
}

/* setup SRTP context for incoming packets */
/* Setup SRTP context for incoming packets */
if (!av_base64_encode(buf, sizeof(buf), recv_key, sizeof(recv_key))) {
av_log(s, AV_LOG_ERROR, "Failed to encode recv key\n");
ret = AVERROR(EIO);
Expand Down