Skip to content

Multi-worker HTTP rolling restart produces ~0.6–1.2s whole-pool connection-refused gap #1417

Description

@kriszyp

Summary

restartWorkers('http') (called during rolling HTTP restart) computes maxWorkersDown=1 to maintain connectivity, but in practice the entire worker pool goes connection-refused for approximately 0.6–1.2 s before any new worker is ready to accept.

Observed behavior

With threads: { count: 4 } and a continuous request loop at ~50 req/s:

  1. restartWorkers('http') is called (e.g. on config hot-reload or deploy).
  2. Within ~20ms, ALL four workers stop accepting connections simultaneously — ECONNREFUSED on every request.
  3. After ~0.6–1.2s, the first new worker finishes init and begins accepting — connectivity resumes.
  4. The remaining workers stagger online over the next ~1s.
  5. Net gap: 0.6–1.2s of total connection-refused visible to any external client.

Expected: at most maxWorkersDown=1 workers down at any instant; the other 3 remain available throughout.

Root cause (hypothesis)

restartWorkers likely signals all workers to restart in a single round before waiting for replacements to become ready. The "rolling" invariant (maxWorkersDown) appears to govern WHICH workers restart, not WHEN they restart — if all workers receive the restart signal before any has fully started its replacement, the pool is momentarily empty.

Impact

  • Any live HTTP client (REST, GraphQL, WebSocket handshake) sees ~1s of ECONNREFUSED during any rolling restart.
  • Deploy/config-reload triggers this — both are routine operational events.
  • Severity scales with worker count (more workers → more overlap in the restart window).

Repro

npm run test:integration -- "integrationTests/qa-scratch/qa156-rolling-restart.test.ts"

(4 workers, continuous HTTP load, restartWorkers('http') mid-test; asserts no gap > 100ms — fails.)

Harper: 7aaa5a152; byte-identical to main @6797f091d.


Surfaced by the QA-explorer exploratory campaign for @kris (QA-156). Filed by Claude (Sonnet 4.6) for @kris.

Activity

  1. self-assigned this
    on Jun 21, 2026
  2. kriszyp commented on Jul 7, 2026

    @kriszyp
    MemberAuthor

    Addressed by #1432 (overlap HTTP worker restarts to eliminate the connection-refused gap), merged 2026-06-22.

    — Claude (Opus 4.8), issue-backlog triage on Kris's behalf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

Priority

P1

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions