Deno 32-stream regression test intermittently returns empty body under CI load

[momics]

Summary

The regression #119 — 32-stream burst × 5 iterations test in the Deno adapter intermittently fails in CI with an empty response body on one stream. The failure is non-deterministic — an immediate rerun of the same job passes. The issue was first observed in CI run 25097645022 on commit 93bb084, which only changed bench.yml (no test or adapter code).

This is not the #122 bug (stale handles / unknown handle errors). #122 is closed and the fix landed. This is a new, subtler race: a response arrives with an empty body instead of the expected 4 KiB payload — likely a non-200 status (408 or 503) that r.text() silently consumes.

Evidence

Failing run (job 73539387105):

regression #119 — 32-stream burst × 5 iterations: ... FAILED (2s)
AssertionError: Values are not equal: iter 2 stream 0: body truncated
  Actual:   ""
  Expected: "xxxx…" (4096 chars)

Passing run 27 minutes earlier (job 73535773455):

regression #119 — 32-stream burst × 5 iterations: ... ok (2s)
ok | 40 passed | 0 failed | 1 ignored (40s)

Environment is identical between both runs:

• Deno 2.7.13, Rust 1.95.0, same setup-deno SHA, same runner image
• Only bench.yml changed between commits — no test, adapter, or core code changes

Node test passes consistently — same run, same commit: 181ms, zero errors.

Impact

CI is unreliable — the E2E job can fail on any push without a code change. Currently no workaround other than re-running the job.

The test previously did not assert r.status, so the failure message ("body truncated") hides the real cause. A diagnostic commit has been prepared to assert status codes.

Remediation

Diagnostic (immediate)

Assert r.status === 200 in both the Deno and Node regression tests. Next time this fails, the error message will reveal whether the empty body comes from a 408 (request timeout — core gave up waiting for Deno FFI dispatch), 503 (queue full — previous serve cycle interfering), or a genuine 200 with no body.

Investigation (after diagnostic data)

The most likely cause is that handle.finished resolves before the Deno adapter's FFI polling loop has fully quiesced. When the next iteration calls server.serve(), leftover async work from the previous cycle may interfere — either by consuming a request slot, or by racing with the new serve's on_request dispatch.

Key question: does the Deno adapter's rawServe resolve finished before its pending set is fully settled, or before the Rust-side ServeQueue is fully torn down?

Relevant files

File	Role
packages/iroh-http-deno/test/smoke.test.ts	Failing test
packages/iroh-http-deno/src/adapter.ts	rawServe polling loop — pending set drain
packages/iroh-http-deno/src/dispatch.rs	serve_start, stop_serve, on_request spawn
packages/iroh-http-deno/src/serve_registry.rs	ServeQueue lifecycle
crates/iroh-http-core/src/server.rs	on_request_fire, drain loop, timeout layer

Acceptance criteria

1. The diagnostic commit (assert r.status === 200) is merged so the next failure reveals the HTTP status code
2. Once the status code is known, the root cause is identified and fixed
3. The regression test passes reliably across 10+ consecutive CI runs

[momics]

Root cause identified and fixed in bd64fa2.

The empty body was a 503 Service Unavailable from Tower's LoadShedLayer. loadShed defaults to true, so when maxConcurrency (1024) is reached under CI contention, new requests are immediately rejected with 503 (empty body) rather than queued. The test's r.text() silently consumed the empty 503 response, surfacing only as "body truncated".

Fix: Both the Deno and Node 32-stream burst tests now pass loadShed: false in serve options, so requests queue instead of being rejected. This is a test-only change — production defaults are unchanged.

The loadShed default was also incorrectly documented as false in both JSDoc and the specification — corrected in e185bd0.

This can be closed.

[momics]

The loadShed: false fix in bd64fa2 was necessary but not sufficient — the 503s persisted in CI.

Actual root cause: stop_serve called serve_registry::remove(handle) before ep.wait_serve_stop().await. Lingering connection-handler tasks (detached tokio::spawn from serve_with_events) could still fire on_request after the registry entry was removed. serve_registry::get(handle) returned None → 503 response, regardless of loadShed setting.

Fix (effd747): stop_serve no longer removes the registry entry. The shutdown-flagged entry stays alive so on_request can find it, check shutdown_rx, and reject promptly with a proper 503 only for requests that arrive after shutdown. The entry is replaced by the next serve_start or cleaned up by close_endpoint.