Full cargo-test hangs in issue_4903_listen_callback_deferred on every SHA containing #5831 (180-min timeout; blocks nightly + release tag gates)

[proggeramlug]

Symptom

The full-workspace cargo-test job (nightly schedule, tag pushes, workflow_dispatch) stopped completing on 2026-07-03. It hangs inside crates/perry/tests/issue_4903_listen_callback_deferred.rs until the job's 180-minute timeout kills it.

Evidence

• Pre-flight dispatch on bfa0f258d (run 28653372348): last test-result line at 10:37:05, job killed at 13:04:54 — ~2.5 h with zero further progress. The runner's orphan-process cleanup terminated issue_4903_listen_callback_deferred-82a5329debfca2bf, its compiled main_bin, and a sleep — the test binary was alive and parked at kill time.
• July-3 nightly on b08006923 (run 28643991364, first nightly containing runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831): ran ~3 h without finishing cargo-test before being cancelled.
• July-2 nightly on 5df641c2e (pre-runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831): full cargo-test completed in 66 min (its only failure was the since-fixed fix(runtime): #5763 regression — setPrototypeOf cycle walk must not advance past chain end #5866 setPrototypeOf regression).
• Two other tests in the hung run ballooned to 364 s and 283 s before it stalled entirely — event-loop waits got slower before one finally never woke.
• The currently-running Tests dispatch on edd608a44 will hit the same wall (contains runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831).

Why #5831 is the prime suspect

issue_4903 is an in-process HTTP server + http.get client round-trip — exactly the in-process server+client wakeup path that #5831 ("JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive", supersedes #5779) reworked. The #5779 deadlock family was always load/timing-sensitive, which fits a CI-only repro on 2-core ubuntu runners.

Consistent with timing-sensitivity: the same shape passes on macOS at bfa0f258d (perry-dev build, compile + run of the test's main.ts — clean exit, correct output). Linux/CI is where it parks.

Impact

• Nightly full-workspace cargo-test can't complete → no full-suite signal.
• Release tags are blocked: the tag-triggered Tests run gates release-packages.yml via await-tests; it will time out at 180 min on every SHA ≥ 45e6773.
• Today's release is being cut as v0.5.1215 from f3be32fa7 (newest pre-runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831 SHA) to route around this.

Repro pointers

• CI: dispatch test.yml on any SHA ≥ 45e6773, watch cargo test -p perry --test issue_4903_listen_callback_deferred (both tests in the file are in-process server round-trips; the orphan list shows main_bin + sleep parked).
• The 364 s / 283 s slow tests in run 28653372348's log may be a smaller, faster-to-debug signature of the same wait-driver latency issue.

[proggeramlug]

Diagnostic narrowing (hang-lab on debug/5892-hang-lab, ubuntu-latest):

• Round 1 — standalone repro is CLEAN. Both issue_4903 programs compiled with a perry-dev-profile driver and run concurrently through pipes, 40 iterations on the CI runner: every iteration completed correctly (body ok), zero hangs. Same result locally on macOS (80 serial + 60 concurrent loaded iterations).
• So the hang requires the cargo-test harness context — debug-profile perry driver (different runtime-archive selection), Command::output() pipes with null stdin, libtest parallel threads, tempdir cwd — not just the two programs racing.
• Round 2 running now (run 28664851669): the exact cargo test -p perry --test issue_4903_listen_callback_deferred in a watchdogged loop; on hang it gdb thread-apply-all-bt's every process in the tree (harness, perry, main_bin) plus socket-state dump.

Wait-topology read so far, for whoever picks this up: stdlib_fast_drive's gate only checks EXT_BLOCKING_TASKS_INFLIGHT + the HTTP client in-flight counter — HTTP server tasks on the current-thread runtime are invisible to it. And any blocking syscall/sleep inside a spawned task on the current-thread RUNTIME would freeze run_one_tick's block_on (and the whole main thread) permanently, which matches the parked-forever kill picture better than a lost notify (those self-heal within the 1 s idle cap).

[proggeramlug]

Round 2: hang captured on iteration 1 (exact cargo test -p perry --test issue_4903_listen_callback_deferred, run 28664851669). Evidence from the gdb/proc dump of the hung main_bin:

• Single thread, parked on a futex (wchan: futex_do_wait) — NOT on epoll. If the main loop were inside the runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831 wait-driver tick (RUNTIME.block_on(timeout(budget, notified))), the current-thread runtime would park on the I/O reactor (ep_poll). A futex park means either the event-pump condvar fallback (wait-driver never registered?) or a single-thread Mutex self-deadlock.
• ss shows the server socket still LISTENing (fd 3) and zero established connections — the client's http.get never connected. The spawned client task never ran.
• The harness ran the file's two tests in parallel → two perry compile invocations ran concurrently. Round 1 (which compiled sequentially, then ran the same two programs concurrently 40×) never hung. That leaves concurrent-compile binary corruption on the table as a third hypothesis.
• Frames are unsymbolized (debug-driver static-wrapper link strips) — round 3 discriminates behaviorally instead.

Round 3 running (run 28665408372): /proc/pid/syscall (futex timeout arg: 0x0 = infinite = self-deadlock; non-null = cycling condvar), 6 s strace wake-pattern sample, utime/stime delta, md5 of hung binary vs clean sequential recompile of the same source, and a serial --test-threads=1 phase (no concurrent compiles) vs parallel phase.

[proggeramlug]

Root-cause candidate found by static analysis: un-gated test shims shadowing the real perry-ffi async bridge.

perry_ffi_spawn_blocking_with_reactor has FOUR definitions in the workspace:

• perry-stdlib/src/perry_ffi_async.rs:260 — the real one (calls ensure_pump_registered() → registers the runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831 wait-driver, then spawns on the current-thread RUNTIME)
• perry-ext-http/src/test_async_shims.rs, perry-ext-http-server/src/test_async_shims.rs, perry-ext-net/src/test_async_shims.rs — synchronous test shims, declared mod test_async_shims; with NO #[cfg(test)] gate, so they ship in the production ext archives despite their own comment saying "Test-only; never shipped."

The shim versions: spawn_blocking_with_reactor/spawn_blocking run the closure inline (no runtime context, no pump/driver registration) and perry_ffi_spawn_async is a no-op that silently drops the task. If the linker resolves these #[no_mangle] C symbols from an ext archive instead of libperry_stdlib.a (single-pass archive scan: the reference originates inside the ext archive, and the shim definition sits in the same archive), the entire async dispatch dies: the accept loop never spawns properly, client tasks vanish, ensure_pump_registered never runs → wait-driver never registered → the observed 1 s condvar-fallback cycle → hang. The auto-opt/strip-dedup path (#5000/#5685) localizes duplicate globals in wrapper archives, which is why auto-opt-compiled binaries (hang-lab round 1) are healthy while the cargo-test debug-driver path hangs.

Instrumented round 4 is running (registration-vs-fallback slot prints) to confirm on the real CI binary.

Fix candidate: gate all three modules with #[cfg(test)] so the shims never enter production archives — the healthy auto-opt links already prove the ext→stdlib resolution works when the duplicates are out of the picture.

[proggeramlug]

Status: the #5911 gate alone did not clear the real recipe — the v0.5.1219 full-Tests dispatch (28671742044) still hit the 180-min cargo-test timeout (all other jobs green).

Current unifying hypothesis: test.yml's cargo-test job restores target/ from rust-cache, which still carries stale pre-gate libperry_ext_*.a archives — the job never rebuilds ext staticlibs (and cargo clean -p's the ext crates), so harness-compiled binaries keep linking the old ungated archives regardless of the source fix. Every lab configuration that hung used cache-restored archives; every one that rebuilt them in-run was clean.

Round 7 (run 28679227426) replicates the test.yml recipe exactly (same env, same build set, no ext staticlib builds, serial test threads) and captures PERRY_LINK_MAP from the hanging compile — the map names the archive member that provides perry_ffi_spawn_blocking_with_reactor/spawn_async and inventories which .a files the cache delivered. If stale-cache confirmed, candidate durable fixes: (a) delete target/debug/libperry_ext_*.a at the top of the cargo-test job (cache hygiene), (b) extend strip-dedup's duplicate-global localization to every link path, or (c) have the link line prefer libperry_stdlib.a for the perry_ffi bridge symbols.

[proggeramlug]

Complete root cause — two layers, both now fixed:

1. Source layer (fix(link): #5892 — keep perry_ffi test shims out of production ext archives #5911, merged): un-gated test_async_shims in perry-ext-http/-http-server shipped fake perry_ffi_* symbols in production archives (spawn_async = task-dropping no-op; spawn_blocking_with_reactor = inline, never registering the runtime: JS-loop-driven current-thread async runtime + wait-driver with gated fast-native-drive (supersedes #5779) #5831 wait-driver).
2. Cache layer (42e2401): the auto-opt archive cache (target/perry-auto-<hash>) keys on perry-runtime's source hash only and ignores ext-crate sources — so CI's rust-cache-restored stale dir kept linking the pre-fix optimized archives into test binaries even after fix(link): #5892 — keep perry_ffi test shims out of production ext archives #5911 merged. That's why the v0.5.1219 gate run still hung. Every lab configuration that happened to rotate the hash (any perry-runtime edit — instrumentation, gates) linked fresh archives and came back clean, which was the misleading part all day. Hang-lab round 8 replicated the cargo-test recipe exactly against the gated tree and passed 4/4 (first iteration 560 s = the fresh from-source lib rebuild, then 48 s each).

test.yml's cargo-test job now evicts target/perry-auto-* (and restored ext .a) after cache restore, so every run links archives built from the checked-out tree.

Remaining follow-up for this issue: the durable perry-side fix — include ext-crate sources (everything that lands in the optimized archives) in the auto-opt cache key in optimized_libs.rs, so stale-dir reuse is impossible outside CI too. The known symptom of this key bug predates #5892: 'warm auto-opt cache ignores perry-ext-http edits' during #5778 development.

Final gate runs on 42e2401 in progress; will close this issue once green.