Repository navigation
HTTP/3: don't block the transport/reactor thread with business logic (ACK-timing budget) #80
Description
Problem
In QUIC/HTTP/3 the entire transport — ACK generation, loss detection, congestion control, RTT estimation, PTO/pacing/idle timers — lives in userspace, driven by our own reactor loop (ngtcp2 has no internal threads). If a per-request coroutine runs a long synchronous / blocking burst on that same reactor thread, nobody reads incoming datagrams or sends ACKs while it runs. This is fundamentally different from TCP, where the kernel ACKs independently of the application thread.
This is not "you can never run handler code on the transport thread". Cooperative, non-blocking execution on the same thread is fine. The rule is: the reactor must never be stalled long enough to delay ACKs / timers.
Why it breaks (mechanisms)
- Delayed ACKs inflate RTT and PTO. We advertise
max_ack_delay(default 25 ms). The peer subtracts our reportedack_delayfrom RTT samples, but capped atmax_ack_delay. Delay an ACK longer (thread stuck in handler) → the excess is attributed to "scheduler latency at the peer", folded into path RTT →smoothed_rttandPTO = smoothed_rtt + 4·rttvar + max_ack_delayinflate → slower loss recovery. We effectively lie about RTT and poison the peer's congestion control. (RFC 9002 §5–6.) - ACK-clocking stall / spurious retransmits. QUIC is ACK-clocked. Not draining the socket and not ACKing promptly → peer's
bytes_in_flightdoesn't drain, cwnd can't grow, or it infers loss → spurious retransmits. - Idle timeout / connection drop. Block long enough and we miss idle deadline, PTO probes,
PATH_CHALLENGEresponses → connection killed. msquic literally documents this as "dropped connections". - Thread-level head-of-line blocking — the irony. QUIC removes HoL blocking between streams. But running request A's handler on the reactor blocks the transport for every connection/stream on that thread — reintroducing HoL at a worse layer. One slow handler stalls everyone's ACKs.
What other implementations do (consensus: transport thread ≠ business-logic thread)
- msquic (Microsoft), most explicit: "MsQuic does not use separate threads for protocol execution and upcalls to the application. Therefore, any significant delays in the callback will delay the protocol." / "Any substantial work required by the application must be performed on threads created by the application." / "Any significant delays will delay other QUIC processing (such as sending acknowledgments), which can cause protocol issues (dropped connections)."
- ngtcp2 (what we use): pure callback state machine, no internal threads; app drives
recv → writev_stream → update_pkt_tx_time → get_expiry2 → timer. Stuck thread ⇒ that connection's PTO/ACK/idle/pacing timers are simply not serviced. - nginx: single-threaded event-driven worker; blocking/synchronous work is offloaded to a thread pool.
- quic-go: transport runs in its own goroutines; each stream is handled in a separate goroutine — the runtime offloads handler work off the receive loop for you.
The deadline budget
Loop-iteration latency must stay well under, in order of strictness:
max_ack_delay(default 25 ms) — keep ACK latency under this or you start inflating peer RTT;- PTO (~
smoothed_rtt + 4·rttvar + max_ack_delay); - idle timeout (seconds, hard kill).
Practical target: single-digit-ms reactor iterations.
Action items (our codebase)
- Audit the H3 request path for any synchronous CPU burst without a yield under the reactor loop (heavy render/serialize, sync DB, large gzip, big
smart_strbuilds). Same class of bug as [no-sync-in-async-loop] and [tls-no-coroutine-in-internals]. - Confirm per-request coroutines yield back to the reactor frequently enough; document the ~ms budget.
- Add a rule to CODING_STANDARDS: transport reactor stays strictly non-blocking; long synchronous CPU work goes to a worker thread/queue.
- (Optional) Instrumentation: reactor-iteration latency histogram + "time since last ACK sent per connection"; a watchdog that logs when an iteration exceeds the budget.
- Cross-link with the inter-thread queue work (companion issue) — the offload target for heavy handlers must itself be lock-free so the handoff doesn't reintroduce a stall.
References
- RFC 9002 — QUIC Loss Detection and Congestion Control: https://quicwg.org/base-drafts/rfc9002.html
- msquic Execution model: https://github.com/microsoft/msquic/blob/main/docs/Execution.md
- ngtcp2 programmers' guide: https://nghttp2.org/ngtcp2/programmers-guide.html
- nginx QUIC/HTTP/3: https://nginx.org/en/docs/quic.html
- quic-go streams: https://quic-go.net/docs/quic/streams/
- Cloudflare — QUIC death-spiral fix (cwnd/ACK interactions): https://blog.cloudflare.com/quic-death-spiral-fix/
- "Taking a Long Look at QUIC" (CACM, userspace cost): https://cacm.acm.org/research/taking-a-long-look-at-quic/
- "QUIC is not Quick Enough over Fast Internet" (receiver application-limited): https://arxiv.org/pdf/2310.09423
Activity
EdmondDantes commentedon Jun 2, 2026
Companion: #81 (lock-free inter-thread queue — the non-blocking offload target for heavy H3 handlers).
feat(offload): CPU-offload thread pool substrate (#80)
feat(compression): malloc-domain off-thread gzip compressor (#80)
feat(http3): reactor-iteration watchdog + non-blocking audit (#80)
docs(http3): drop fabricated offload phasing from reactor-pool plan (#80
feat(reactor-pool): pure-C transport reactor pool substrate (#80)
refactor(reactor-pool): channel-driven reactor loop, drop the livenes…
test(reactor-pool): flag-gated self-test hook + phpt substrate test (#80
fix(reactor-pool): lost-wakeup deadlock + future leak found under ASan (
feat(reactor-pool): flat request_wire marshalling type for D2 (#80)
feat(reactor-pool): worker-side http_request_from_wire materialization (
feat(reactor-pool): bring up the transport reactor pool on the pool p…
feat(reactor-pool): reactor_pool_exec — run C init on a reactor's own…
40 remaining items
refactor(core): rename reactor_pool_test.c -> reactor_pool_test_hooks…
fix(build): point build configs at reactor_pool_test_hooks.c (#80)
style: trim comments to the coding standard (#80)
fix(http3): hq-interop TOCTOU/Win32 hardening + #80 review cleanups (#80
ci(windows): build on windows-2025-vs2026 with vs18 + php-sdk-2.7.1 (#80
ci(windows): drop VS toolset pinning, let phpsdk pick it (mirrors rel…
test(h1): de-flake 012 drain-cooldown — free conn1 slot deterministic…
Merge pull request #90 from true-async/80-http3-dont-block-the-transp…
feat(pool): response trailers cross the reactor/worker split (#80, #4)
feat(pool): streaming reverse path — send()/writeMessage() cross the …
feat(pool): credit-based backpressure for the streaming reverse path (#…
docs: gRPC 100% — grpc-web-text + reactor-pool streaming landed (#4, #80
fix(ci): Autobahn image — build ZTS, pin server checkout to the commi…
chore(release): 0.10.0 - gRPC (#4), reactor-pool streaming (#80), HTT…