[Feature] Support io_uring-based TLS I/O (OpenSSL memory BIO)

[MrGuin]

Background

We want to add TLS support without blocking socket I/O. The plan is to:

• Use io_uring for non-blocking recv/send on sockets.
• Use OpenSSL for TLS handshake + encryption/decryption.
• Connect them via memory BIO (BIO_s_mem) so OpenSSL never reads/writes the socket directly.

Goals

• Implement TLS for connections while keeping the networking stack fully io_uring-driven.
• Support non-blocking:
  • handshake
  • application reads (decrypt)
  • application writes (encrypt)
  • graceful shutdown (TLS close_notify)

Proposed Design

• Per-connection state
  • SSL* ssl
  • BIO* rbio (input: ciphertext from socket)
  • BIO* wbio (output: ciphertext to socket)
  • recv_buf for ciphertext
  • send_buf for unencrypted data

[MrGuin]

Goal

Keep the existing fd-based handshake, but once _ssl_state == SSL_CONNECTED, switch the SSL object to use two BIO_s_mem instances. All subsequent TLS encryption/decryption happens against these memory BIOs, while actual socket I/O is handled by io_uring.

1. State & Structure Changes

• Extend Socket with a TlsRingContext struct that holds:
  • BIO* mem_rbio and BIO* mem_wbio (the new BIO_s_mem objects bound to _ssl_session after the handshake).
  • butil::IOBuf pending_cipher_out (or reuse RegisteredRingBuffer) storing ciphertext that still needs to be submitted via io_uring.
  • Flags like bool want_write_pending, plus condition variables to synchronize SSL_write that returned SSL_ERROR_WANT_WRITE.
• Hook this context into /connections output so operators can see whether a socket is “TLS over io_uring”.

2. Lifecycle & Iteration Plan

Step A – Handshake untouched

• Leave Socket::SSLHandshake exactly as today (AddBIOBuffer + fd I/O).
• After handshake succeeds (_ssl_state becomes SSL_CONNECTED), read/write BIOs might still contain residual data; call BIO_flush on the write BIO
  and drain any pending bytes from the read BIO via SSL_read (append them to _read_buf to preserve semantics).

Step B – Rebind to BIO_s_mem

• Allocate two BIO* via BIO_new(BIO_s_mem()).
• Lock _ssl_session_mutex, call SSL_set_bio(_ssl_session, mem_rbio, mem_wbio) to swap the SSL object to the memory BIOs. Store both pointers inside
  TlsRingContext so other code can call BIO_read/BIO_write.
• The fd BIOs created during handshake can now be BIO_free.

Step C – Modify the write path (DoWrite/KeepWrite/Ring callback)

1. In Socket::DoWrite, when FLAGS_use_io_uring && ssl_state()==SSL_CONNECTED, replace the cut_multiple_into_SSL_channel branch with:
   • SSL_write plaintext while holding _ssl_session_mutex.
   • After each SSL_write, loop on BIO_ctrl_pending(mem_wbio) and BIO_read(mem_wbio, tmp) to drain newly produced ciphertext into
     pending_cipher_out.
   • If SSL_write returns SSL_ERROR_WANT_WRITE, set want_write_pending=true and break; the flag tells RingNonFixedWriteCb to resume writing once
     io_uring flushes enough ciphertext.
2. Implement FlushTlsCiphertext():
   • Cut pending_cipher_out into iovecs (or reuse ring buffers) and submit to io_uring via TaskGroup::SocketWaitingNonFixedWrite.
   • If the write is synchronous (stream case), wait for completion via SocketWaitingNonFixedWrite.
3. Update RingNonFixedWriteCb:
   • Pop consumed bytes from pending_cipher_out.
   • If want_write_pending is true and pending_cipher_out is now small enough, clear the flag and wake KeepWrite (so it calls SSL_write again).
   • Maintain the existing logic of returning successful WriteRequests once plaintext buffers are empty.

Step D – Modify the read path (CopyDataRead/DoRead)

• In CopyDataRead() and Socket::DoRead (io_uring branch), instead of appending ciphertext directly to _read_buf, do:
  1. For each ring buffer chunk, call BIO_write(mem_rbio, chunk_data, chunk_len) while holding _ssl_session_mutex.
  2. Loop SSL_read to retrieve plaintext; append it to _read_buf (the same as IOPortal::append_from_SSL_channel used to do).
  3. If SSL_read returns SSL_ERROR_WANT_READ, simply return the number of bytes already produced and wait for more ciphertext from io_uring.
  4. Handle SSL_ERROR_ZERO_RETURN / fatal errors just like the current code.

Step E – Synchronization helpers

• Provide helper functions on TlsRingContext:
  • int DrainCiphertextToRing(bool synchronous) – used by both DoWrite and RingNonFixedWriteCb.
  • ssize_t FeedCiphertextFromRing(const char* data, size_t len) – called inside CopyDataRead.
  • ssize_t ConsumePlaintext(butil::IOPortal* out, size_t hint) – wraps the SSL_read loop for readability.

Step F – Iterative rollout

1. Phase 1: Add TlsRingContext, the post-handshake rebind, and stub helper methods (no io_uring changes yet). Keep gating under a new flag (e.g.,
   FLAGS_tls_ring_mode) so regression risk is minimal.
2. Phase 2: Switch write path to use the new helpers while keeping read path untouched. Validate with unit tests / integration tests.
3. Phase 3: Switch read path, add statistics/metrics.
4. Phase 4: Cleanup legacy assumptions (CHECK(!use_mixed_data_list)), document the new behavior.

Key Code Areas

• src/brpc/socket.cpp
  • Socket::SSLHandshake (post-success hook for rebind).
  • Socket::DoWrite, Socket::KeepWrite, Socket::RingNonFixedWriteCb.
  • Socket::DoRead, Socket::CopyDataRead.
• src/brpc/socket.h
  • Add TlsRingContext into the Socket class, plus helper declarations.
• src/butil/iobuf.cpp
  • No major changes; reuse existing buffer APIs.

This plan lets TLS encryption/decryption live entirely in memory via BIO_s_mem, while io_uring handles the actual socket I/O, without rewriting the
initial handshake.

[MrGuin]

Socket::DoWrite procedure coding almost done, start testing and debugging.

[MrGuin]

Socket::DoWrite procedure coding almost done, start testing and debugging.

DoWrite seems fine. I've done some simple test through echo_client and echo_server both enabling SSL while the client's write uses new code path. The client's SSL-encryted data is successfully received and processed by the server.

[MrGuin]

Progress is currently blocked on implementing the SSL handshake with io_uring; it needs further research and careful thought.

[MrGuin]

In most cases, the read/write flow works end-to-end, but there are still occasional issues. I’m currently debugging them.

[MrGuin]

In most cases, the read/write flow works end-to-end, but there are still occasional issues. I’m currently debugging them.

The cause of the phenomenon is that a epoll_wait bthread is started in SSLHandShake, which blocks the brpc worker.

[MrGuin]

Known bugs fixed and Redis protocol supported. Benchmarking.

[MrGuin]

Epoll based ssl error code handling fixed.

[MrGuin]

Something is wrong with the connection establishing. ssl_do_handshake might return random errors. Trying to figure out why.

[MrGuin]

Fixed a bug that data is not flushed during handshake, but there are still unknown bugs causing handshake error or data cannot be received.
Lots of logging and debugging has been done and
I’m going to put this on hold for now.