Skip to content

fetch deadlocks (lost-wakeup) when the same process is both an HTTP server and a fetch client #5779

Description

@proggeramlug

Split out of #5757 (the body-corruption half is fixed by #5778). This is a separate, pre-existing concurrency bug.

Symptom

When one Perry process is both an HTTP server and a fetch client — an in-process server that calls fetch() (against itself or any host) while serving (SSR, webhooks, API gateway, tests) — a fetch request can hang forever: the promise never settles, the process never exits.

import { createServer } from "node:http";
const s = createServer((req, res) => {
  let c: Buffer[] = [];
  req.on("data", (b: Buffer) => c.push(b));
  req.on("end", () => res.end("len=" + Buffer.concat(c).length));
});
s.listen(0, async () => {
  const port = (s.address() as any).port;
  for (let i = 1; i <= 40; i++) {
    const r = await fetch(`http://localhost:${port}/`, { method: "POST", body: "x".repeat(i) });
    await r.text();           // eventually hangs here
  }
  console.log("done"); process.exit(0);
});

Single requests usually succeed; it becomes reliable under repeated/concurrent requests and under machine load.

Diagnosis

  • lldb on the hung process: main thread parked on the event-loop condvar, all tokio workers idle, the I/O reactor parked on kevent — i.e. the reqwest request future is parked and never re-woken. The fetch resolution is never queued, so the main loop's idle-cap re-checks find nothing to drain → permanent.
  • Body-agnostic (string and Buffer both hang) and reproduces on main.
  • The node http/https client (different pump) and cross-process fetch (separate runtimes) are not affected — only the in-process loopback where the reqwest client and the hyper server share the tokio runtime (async_bridge::RUNTIME, reached by the server via perry_ffi).
  • It's a reqwest pooled-keep-alive ↔ in-process-hyper-server lost-wakeup on the shared runtime, exacerbated by load.

What was tried (and why it's not enough)

  • pool_max_idle_per_host(0) (disable idle keep-alive): 100% reliable under low load, but still hangs under load, and costs cross-request connection reuse.
  • A dedicated tokio runtime for fetch (separate from the server's): cuts the rate 100%→25% but a permanent residual remains (a deeper same-process lost-wakeup in the await/Promise.all machinery).

A complete fix needs to address the underlying lost-wakeup (reqwest/hyper on the shared runtime, and/or the main-thread event-loop ↔ tokio handoff), not just timing.

Environment

  • Perry main; macOS arm64.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions