feat(threads): graceful drain hook for in-flight work before worker shutdown

[kriszyp]

Summary

Adds a generic per-worker shutdown-drain registry (components/shutdownDrain.ts, mirroring scopeShutdown.ts) that the worker shutdown path awaits before closeServers(). It's the core half of fix (2) of the deploy-reload blob-divergence root cause: harper-pro registers a drain that lets an in-flight replication blob send finish instead of being torn down mid-stream by a rolling http_workers restart (which leaves the peer's copy diverged). Paired harper-pro PR bumps core to this commit.

Core stays deliberately generic — it knows nothing about blobs. A component registers a { hasWork, drain } hook; runShutdownDrains(deadline) awaits them (failure-isolated, never rejects, hard-bounded by the deadline).

What changed

• components/shutdownDrain.ts (new): registerShutdownDrain / shutdownDrainsHaveWork / runShutdownDrains, plus getShutdownDrainCeilingMs() (coerced config read) and the pure boundedTerminateDelay() used to clamp the terminate timer.
• server/threads/threadServer.js: on SHUTDOWN, run registered drains before closeServers(). Extend the termination backstops only when a drain reports progressing work, then restore the normal short timeout once draining completes.
• server/threads/manageThreads.js: extendShutdownDeadline / restoreShutdownDeadline (worker) push out / restore the self-exit backstop and post EXTEND_SHUTDOWN_DEADLINE so the main thread re-arms its worker.terminate() timer to match — clamped at the ceiling.
• utility/hdbTerms.ts: new replication_blobSendDrainTimeout config (default 600000 = 10 min; 0 disables).

Where to look / lower-confidence areas

• The ITC deadline flow across three handlers (manageThreads main + worker, threadServer). The self-exit extension is order-independent via selfExitDrainDeadline (the two worker SHUTDOWN listeners race). The main-thread terminate timer is armed synchronously in the same tick as the SHUTDOWN post, so the worker's async EXTEND reply can only arrive after it exists.
• Hang-recovery is preserved: backstops extend only while a drain is progressing, and are restored to the normal timeout once the drain returns — so a worker that hangs for an unrelated reason (or after draining) is still force-killed on the normal threadTerminationTimeout window. The main thread caps any worker-supplied deadline at the ceiling.

Review notes (open items, per the cross-model review)

• Deploy-rollout latency: restartWorkers' await Promise.race(waitingToFinish) throttle now waits on a draining worker's exit, so a large progressing send can delay the next worker's restart up to the ceiling. Bounded (force-terminate still fires at ceiling + 2×threadTerminationTimeout; capacity is preserved by the Multi-worker HTTP rolling restart produces ~0.6–1.2s whole-pool connection-refused gap #1417 pre-started replacement) and deliberate — surfaced so operators know a deploy can wait on in-flight sends. A follow-up could stop counting a draining worker against maxWorkersDown.
• Correctness of any unfinished send is already guaranteed by fix (1) on the receiver (harper-pro#527) — this drain is a bounded optimization, not a correctness requirement.

Unit tests cover the registry, config coercion, and the boundedTerminateDelay clamp/guard arithmetic. Cross-model reviewed (Codex + Harper-domain adjudication; the Gemini/agy leg hung out and was skipped).

Generated by Claude (Opus 4.8).

[kriszyp]

📖 Docs: HarperFast/documentation#569 documents the new replication.blobSendDrainTimeout option. (comment by Claude/KrAIs)

[gemini-code-assist]

Code Review

This pull request introduces a graceful shutdown drain mechanism to allow workers to complete in-flight work before exiting. It adds a new shutdownDrain component, updates thread management to support extending shutdown deadlines, and integrates these drains into the worker shutdown sequence. The feedback focuses on robustly handling empty, null, or extremely large values for the drain timeout configuration to prevent silent disabling or setTimeout overflows, clearing lingering timers in runShutdownDrains to avoid open handles in tests, and adding corresponding unit tests.

[claude]

Reviewed; no blockers found.

[kriszyp]

CI note (pre-existing failures, not from this PR):

• Unit Test (subscriptionReplay.test.js) — failing on main too (see main run 28818847554: same Subscription replay / "expected at least 1 event for id=1, got 0" at subscriptionReplay.test.js:1048). This PR's own shutdownDrain suite passes.
• Integration Tests 3/6 — the failing cases are device-type schemas visible in describe_all / Per-device-type LMDB database sharding (schema-propagation/setup timing); main's Integration Tests are green, so this reads as a flaky shard. No causal path from a shutdown-timer change.

This PR's diff is 5 files (shutdown-drain hook + timers + config); neither failing suite touches that path.

(comment generated by Claude/KrAIs)

[github-actions]

Patch cherry-pick: conflict

Cherry-pick onto v5.1 produced conflicts on commit(s): 9018760e480a61e54d280eb07e93fa26c96c9a0b

The conflict markers are committed on branch cherry-pick/v5.1/pr-1621.
A pull request has been opened to land this patch: #1728

[Devin-Holland]

Adds a per-worker graceful drain-before-shutdown hook registry so in-flight work (notably a replication blob send streaming to a peer) can reach a safe point before a worker restart, rather than being torn down mid-stream. Careful and fail-safe.

• runShutdownDrains never rejects (a throwing or hanging drain is logged and abandoned), is bounded by an absolute deadline via an unref'd timer that is always cleared, and no-ops with zero drains — so it can't wedge the shutdown sequence that follows.
• The backstop extension is gated on shutdownDrainsHaveWork(), so a worker hung for an unrelated reason is still force-killed on the normal short timeout. Worker-requested deadlines are clamped to the configured ceiling and to a finite value via the pure, unit-tested boundedTerminateDelay, so a rogue/buggy message can't defer the force-kill unboundedly.
• Ordering: the SHUTDOWN handler and the drain-extension race across listeners, and armSelfExit honors selfExitDrainDeadline regardless of which lands first. Worker self-exits (1x timeout headroom) before the main thread force-terminates (2x) — correct backstop order.
• Config coercion handles the string-from-YAML case and the blank-not-zero trap (a blank value must not read as Number('')===0 and silently disable draining), clamps oversized values below the 2^31 setTimeout-overflow cliff, and lets an explicit 0 disable draining. All covered by tests.

One thing worth putting on the radar: because the draining worker keeps its listening sockets up for the drain window (before closeServers), sustained normal traffic that keeps opening connections to it could keep it "busy" and push each restart closer to the ceiling — which would make routine maintenance / rolling restarts drag under load. Whether that actually happens hinges on the harper-pro blob-send drain hook's semantics:

• If the hook snapshots the set of active sends when the drain begins and only waits on those, new connections arriving mid-drain are irrelevant — it drains what was already streaming and resolves. Non-issue.
• If it treats any active send as work, a busy peer could hold a draining worker near the cap (never beyond it — the deadline is fixed at SHUTDOWN receipt and the force-terminate backstops still fire; SO_REUSEPORT also spreads new connections across the already-accepting replacement).

Worth confirming which semantics the hook uses, so operators know whether reboot time under load is bounded by work-in-flight or by ongoing traffic. Not a blocker.

Approving.