io-context overwhelmed by small 16KB TLS reads limits high-throughput scenarios

[hofst]

We use boost beast to read from S3 in high-throughput scenarios: files with hundreds of GBs on machines with 192 cores and 100Gb/s network interfaces to S3 from ec2.
The io-context is limited to ~500k operations per second and saturates at ~8 threads on our machines. More threads will just burn through CPU cycles fighting for locks.
500k operations per second should be plenty to achieve 8+GB/s throughput but it isn't: Every read_some operation only reads a single TLS fragment (max size is 16KB) even when all buffers (socket and user-space buffers) are decently sized. This leads to a limit of 3GB/sec while still beeing inefficient with the io-context busy with tiny read operations.
I tried a hacky boost modification to read more than one TLS fragment per read_some operation which promptly increased throughput by 3x and is close to the theoretical maximum of the machine. All while requiring fewer operations (and CPU utilization) in the io-context.

I don't see any way to control this without modifying boost itself - am I missing something or is this scenario currently not covered?

[vinniefalco]

which version of Boost are you using?

[hofst]

@vinniefalco boost 1.85 - but this has been troubling us for at least 3 years

[ashtum]

500k operations per second should be plenty to achieve 8+GB/s throughput but it isn't: Every read_some operation only reads a single TLS fragment (max size is 16KB) even when all buffers (socket and user-space buffers) are decently sized. This leads to a limit of 3GB/sec while still beeing inefficient with the io-context busy with tiny read operations.

Using Asio's immediate executor should significantly reduce the scheduling cost, you can use bind_immediate_executor to bind an inline_executor to your handler so it just reads the available data and execute your handler immediately. To avoid stack exhaustion you should avoid binding a immediate executor every N operation (N could be like 16).

Because your load is I/O heavy you might also try running one io_context per core to see if it makes a difference. With that you would be able to pass BOOST_ASIO_CONCURRENCY_HINT_UNSAFE_IO to each io_context and reduce contention.

The tips described in this issue might be useful too: #2947

[vinniefalco]

I'd like to see your patch, do you have a fork of Asio you could share?

[hofst]

@ashtum Thank you, I'll keep that in mind. But my instinct is that I rather remove the root cause (unnecessarily small tasks) than tweak the symptoms (make scheduling more efficient). I think the 500k operations we get from io-context should be plenty to solve the problem once the root cause is addressed (as my benchmarks show).

@vinniefalco I'll check next week if I may share source code. In the meantime, the approach is as described above: read more in read_some in beast and change boost/asio/ssl to allow reading more. The tiny 16KB operations are the problem and once this is increased to, e.g., ~100KB problems simply disappear.

[vinniefalco]

Where does beast do a 16kb read?

[hofst]

@vinniefalco

beast/include/boost/beast/http/impl/read.hpp

Line 225 in 67106eb

s_.async_read_some(*mb, std::move(self));

The buffer is sized larger and is bounded by 64k in beast (it would be great to parametrize this upper limit).
But this call s_.async_read_some(*mb, std::move(self)); only reads a single TLS fragment for SSL streams which is 16KB. So every boost/beast op in the io-stream maps to 16KB at maximum.

I don't want to get in a game of ping-pong where we blame either boost/beast or boost/asio/ssl, but:
Either (1) there is a better way to use the streams in boost/beast to fill the buffers to larger sizes
or (2) boost/asio/ssl needs to add a way to retrieve more bytes per read_some operation.