Skip to content

Worker-restart replacement can race a drain-delayed port release on Bun/Windows/macOS (EADDRINUSE for worker-owned listeners) #1813

Description

@kriszyp

Summary

restartWorkers() starts the replacement HTTP worker immediately after posting SHUTDOWN to the
old one whenever the platform can't pre-start a SO_REUSEPORT-sharing replacement
(canPreStartReplacement === false: Windows, macOS, and Bun — see
server/threads/manageThreads.js:453 and the worker.startCopy() call at
server/threads/manageThreads.js:548). That ordering assumes the old worker releases its
worker-owned listeners (e.g. MQTT, which binds exclusively rather than via SO_REUSEPORT) "well
before the replacement finishes booting and binds" (see the comment above canPreStartReplacement).

The shutdown-drain feature added in #1621 (components/shutdownDrain.ts,
server/threads/threadServer.js:194-209) breaks that assumption: a worker with in-flight drain
work now delays closeServers() (which releases its ports) until the drain settles or the
configured ceiling is hit — replication.blobSendDrainTimeout, which defaults to 10 minutes
(DEFAULT_DRAIN_CEILING_MS in components/shutdownDrain.ts). On the affected platforms, the
replacement worker can now start and attempt to bind its own copy of those exclusive listeners
long before the old worker actually frees them.

Evidence

CI run for #1791 (Bun job, "Integration Tests 5/6 (Bun)"), instance log for
integrationTests/components/shutdown-drain-e2e.test.ts:

19:05:38.092Z [main/0] restart_service http_workers requested (2nd restart, old worker tid=3)
19:05:39.079Z [job/4] Starting job (drives the restart)
19:05:39.559Z [http/5] new replacement worker boots (tid=5) — started immediately, well before
              tid=3's drain (configured 5s ceiling in the test) has run
19:05:40.672Z [http/5] [error]: Failed to bind TLS listener for component 'mqtt' to port
              127.0.0.2:8883: address already in use by another process
19:05:40.672Z [http/5] [error]: Failed to bind TCP listener for component 'mqtt' to port
              127.0.0.2:1883: address already in use by another process
19:05:49.243Z [job/4] restart_service job completes (tid=3 finally exits ~11s after shutdown)

tid=3 (the old worker) is still bound to the mqtt ports when tid=5 (its replacement) tries to
claim them, because tid=3's shutdown is mid-drain and hasn't reached closeServers() yet. The
bind failure itself is caught and logged as an "external conflict" (listenOnPortsBun()'s
EADDRINUSE handler in server/threads/threadServer.js around line 545) rather than crashing the
worker, but the net effect is a worker-owned listener (mqtt, in this case) going unbound on the
new worker — the same failure mode the canPreStartReplacement comment set out to avoid, just
triggered by drain-added latency rather than the original "brief gap" it was written for. In the
observed run, the subsequent restart-completion polling in the test also failed to observe the new
worker within a generous 35s window, suggesting downstream effects beyond just the logged bind
conflict (not fully root-caused here).

Suggested directions (not investigated in depth)

  • Don't start the replacement worker until the old worker's closeServers() has actually
    completed (i.e. gate worker.startCopy() on an ITC signal from the old worker, not just on
    having posted SHUTDOWN) when !canPreStartReplacement.
  • Or: have the drain path close the worker's exclusive listeners immediately (before the drain
    runs), independent of the rest of the shutdown sequence, since those don't need to stay open
    during a blob-send drain.

Repro

integrationTests/components/shutdown-drain-e2e.test.ts (added in #1791) reproduces this
reliably on Bun; it currently skips on Bun (see that PR) pending this fix, to avoid blocking CI on
a known, understood gap rather than a flake.

Activity

  1. kriszyp commented on Jul 15, 2026

    @kriszyp
    MemberAuthor

    Same root cause reproduces on Windows, not just Bun — confirmed while investigating a CI failure in the newly-added integrationTests/deploy/deploy-dangling-symlink.test.ts (PR #1791). deploy_component with restart: true drives a real worker restart on the same !canPreStartReplacement path (server/threads/manageThreads.js:453, true for win32 too).

    Windows CI evidence (run 29377543554, commit 5fc130138): the replacement worker (http/2) boots and resolves its full HTTP/mqtt middleware chain in ~1.2s with no bind errors logged — but the restart-readiness poll then observes nothing for the remainder of a 45s deadline (already bumped once from 30s) before timing out. No EADDRINUSE this time (unlike the Bun case), which fits the issue's "not fully root-caused" note — on Windows the bind itself apparently succeeds, but something about request routing/reachability to the new worker still stalls for the old worker's full drain window.

    Skipped that test's suite on win32 (matching the existing win32/Bun skip pattern in shutdown-drain-e2e.test.ts) rather than chasing this inside a QA-anchor-promotion PR — see harper-1791-fix2 dispatch log for the full investigation.

  2. added this to the v5.2 milestone on Aug 7, 2026
  3. added theissue type on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P1

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions