Summary
restartWorkers('http') (called during rolling HTTP restart) computes maxWorkersDown=1 to maintain connectivity, but in practice the entire worker pool goes connection-refused for approximately 0.6–1.2 s before any new worker is ready to accept.
Observed behavior
With threads: { count: 4 } and a continuous request loop at ~50 req/s:
restartWorkers('http') is called (e.g. on config hot-reload or deploy).
- Within ~20ms, ALL four workers stop accepting connections simultaneously —
ECONNREFUSED on every request.
- After ~0.6–1.2s, the first new worker finishes init and begins accepting — connectivity resumes.
- The remaining workers stagger online over the next ~1s.
- Net gap: 0.6–1.2s of total connection-refused visible to any external client.
Expected: at most maxWorkersDown=1 workers down at any instant; the other 3 remain available throughout.
Root cause (hypothesis)
restartWorkers likely signals all workers to restart in a single round before waiting for replacements to become ready. The "rolling" invariant (maxWorkersDown) appears to govern WHICH workers restart, not WHEN they restart — if all workers receive the restart signal before any has fully started its replacement, the pool is momentarily empty.
Impact
- Any live HTTP client (REST, GraphQL, WebSocket handshake) sees ~1s of
ECONNREFUSED during any rolling restart.
- Deploy/config-reload triggers this — both are routine operational events.
- Severity scales with worker count (more workers → more overlap in the restart window).
Repro
npm run test:integration -- "integrationTests/qa-scratch/qa156-rolling-restart.test.ts"
(4 workers, continuous HTTP load, restartWorkers('http') mid-test; asserts no gap > 100ms — fails.)
Harper: 7aaa5a152; byte-identical to main @6797f091d.
Surfaced by the QA-explorer exploratory campaign for @kris (QA-156). Filed by Claude (Sonnet 4.6) for @kris.
Summary
restartWorkers('http')(called during rolling HTTP restart) computesmaxWorkersDown=1to maintain connectivity, but in practice the entire worker pool goes connection-refused for approximately 0.6–1.2 s before any new worker is ready to accept.Observed behavior
With
threads: { count: 4 }and a continuous request loop at ~50 req/s:restartWorkers('http')is called (e.g. on config hot-reload or deploy).ECONNREFUSEDon every request.Expected: at most
maxWorkersDown=1workers down at any instant; the other 3 remain available throughout.Root cause (hypothesis)
restartWorkerslikely signals all workers to restart in a single round before waiting for replacements to become ready. The "rolling" invariant (maxWorkersDown) appears to govern WHICH workers restart, not WHEN they restart — if all workers receive the restart signal before any has fully started its replacement, the pool is momentarily empty.Impact
ECONNREFUSEDduring any rolling restart.Repro
(4 workers, continuous HTTP load,
restartWorkers('http')mid-test; asserts no gap > 100ms — fails.)Harper:
7aaa5a152; byte-identical tomain@6797f091d.Surfaced by the QA-explorer exploratory campaign for @kris (QA-156). Filed by Claude (Sonnet 4.6) for @kris.