Repository navigation
Sending and receiving data frames introduces significant overhead #902
Description
What is the issue?
I was looking at the perf of requests & responses sent via h2 on a TCP stream versus the perf of raw messages sent back and forth over a TCP stream,
and found throughput with h2 to be reduced by an average of around 50%.
The following table compares the perf that I observed on the test VM that I used with h2 requests & responses versus raw messages:
OS: linux (kernel version 6.6)
Arch: arm64
TCP_NODELAY: True
Data Payload Size | Message Kind | Avg Throughput | Avg Round-Trip Latency
4kiB | h2 over TCP | 0.97 GB/s | 8.43 µs
4kiB | raw over TCP | 1.84 GB/s | 4.45 µs
16kiB | h2... | 2.68 GB/s | 12.23 µs
... | raw... | 5.31 GB/s | 6.17 µs
64kiB | h2... | 3.54 GB/s | 37.05 µs
... | raw... | 8.43 GB/s | 15.54 µs
256kiB | h2... | 4.64 GB/s | 113.00 µs
... | raw... | 9.43 GB/s | 55.57 µs
1024kiB | h2... | 4.28 GB/s | 490.00 µs
... | raw... | 9.22 GB/s | 227.40 µs
Primary root causes
Currently, the encoder can only hold a single pending DATA frame, and blocks the acceptance of any new frame until the
previous one is fully flushed to the socket and reclaimed.
This means every DATA frame triggers its own write() syscall and TCP segment — even when the TCP send buffer could carry far more.
With TCP_NODELAY enabled, there is no Nagle buffering to coalesce these small writes, so the per-frame overhead is fully realized.
On the receiver side, the recv buffer used by the frame decoder is small by default, and even if its buffer size was increased,
the effective capacity of that buffer would degrade over time and never recover, causing small read() calls.
This is because:
- The buffer is repeatedly split to yield decoded frames.
- If the recv buffer runs out of capacity, BytesMut::reserve is called, & if that buffer is still in use (likely),
the current buffer is abandoned & a new buffer with a maximum capacity of 64kB would be created
due to how BytesMut encodes its original capacity to a max of 64kB & caps new reservations
based on that encoded original capacity.
Together, these things inflate the syscall-to-payload ratio on both ends.
Activity
NedAnd1 commentedon May 5, 2026
I have a fork of h2 which allows perf to be improved by almost 2x:
and I can start by raising a PR for the sender side.
h2-001: track hyperium/h2#902/#903, keep grown buffer for large write…
Coalesce h2 DATA frames into one write (vendored h2 0.4.19) (#5969)
FalkWoldmann commentedon Oct 12, 2026
I ran this through a hyper server benchmark on x86, since the numbers so far are from raw h2 on arm64. Unmodified hyper 1.12 on a
current_threadtokio runtime pinned to one core, HTTP/2 without TLS over loopback, a separate closed-loop load generator. This branch against its merge base, 5 interleaved rounds of 5 s per cell, on an i7-6700K with Linux 7.2.writevper request, base