Symptom
Recurring Linux Integration Tests failure on main — currently the top non-Windows CI failure (red in 3 of the last 5 runs):
Error: Worker (index 1) exited with code 0 before reporting ready
at Worker.onExit (server/threads/socketRouter.ts:100:11)
immediately followed in sibling workers by a Database not open cascade, after which every in-flight test in the shard is cancelled.
Occurrences (same shape, different shards, suites, and Node versions — 22 / 24 / 26.5):
The affected suites are all multi-worker (threads.count: 4), RocksDB-engine, high-concurrency startup tests.
Root cause
A Harper worker thread that has not yet posted child_started owns no ref'd libuv handle:
manageThreads.addPort() explicitly unref()s the worker's parentPort (and sibling ports); threadServer.js only .ref()s it after loadRootComponents() resolves.- Component file watchers are chokidar
persistent: false; the resource-report and monitoring timers are unref()d.
So during loadRootComponents(), any await whose completion arrives through a source that does not hold the event loop lets the loop drain — and a worker whose loop drains exits cleanly with code 0, before the ready handshake. createWorkerReadyPromise then rejects, startHTTPThreads' Promise.all(workersReady) rejects, and bin/run.ts's catch calls process.exit(1) — the whole node aborts during boot.
The natural trigger for exactly this window is rocksdb-js: it delivers cross-thread lock-release wakes (DBDescriptor::lockEnqueueCallback, used by Store.tryLock(key, onUnlocked)) and parked-commit retry wakes (coordinated-retry RETRY_NOW in transaction.cpp) through threadsafe functions that are napi_unref_threadsafe_function'd at creation. A pre-ready worker parked on a peer worker's lock — e.g. the cross-thread store.tryLock(componentDirectory) in componentLoader.ts's symlinkHarperModule(), whose non-acquired branch additionally clears the only ref'd timer before waiting — contributes nothing to its own loop's liveness and can exit before the wake arrives. Four workers concurrently loading the same components at first boot make that contention structural; whether the loser happens to have other in-flight ref'd work at the drain instant is the flake coin-flip.
Post-ready workers park on the same locks constantly but survive, because the ref'd parentPort (and bound listeners) hold the loop — which is why this only ever surfaces as "exited … before reporting ready".
The mechanism reproduces standalone: a worker with an unref'd parentPort awaiting a completion behind an unref'd handle exits code 0 at the drain instant. The full-instance drain did not reproduce locally on a 20-core dev box (constant startup ITC traffic keeps reviving the loop there); it reproduces on loaded 4-vCPU CI runners.
Cause vs. effect
The Database not open cascade in sibling workers is an effect, not a second defect: main's process.exit(1) tears down stores while sibling workers are still mid component-load (the stack in the log: processGraphQLSchema → table → openAuditStore → RocksTransactionLogStore).
Fix
threadServer.js: hold parentPort.ref() for the entire pre-ready window (SHUTDOWN still unrefs for graceful exit) — a worker must report ready, throw, or be terminated; never silently drain.componentLoader.ts symlinkHarperModule(): the lock waiter keeps a ref'd bounded timer instead of clearing it (also fixes an unbounded wait if the lock holder dies, and removes the winner's stale 10s timer that could unlock() a lock it no longer owned).
A companion rocksdb-js issue covers the unref'd threadsafe-function wakes themselves (pending cross-thread lock/commit-retry wakes should keep the waiting thread's loop alive).
Related: #1827 (startup hang with no output) may share the symlinkHarperModule unbounded-lock-wait surface fixed here.
Symptom
Recurring Linux Integration Tests failure on
main— currently the top non-Windows CI failure (red in 3 of the last 5 runs):immediately followed in sibling workers by a
Database not opencascade, after which every in-flight test in the shard is cancelled.Occurrences (same shape, different shards, suites, and Node versions — 22 / 24 / 26.5):
The affected suites are all multi-worker (
threads.count: 4), RocksDB-engine, high-concurrency startup tests.Root cause
A Harper worker thread that has not yet posted
child_startedowns no ref'd libuv handle:manageThreads.addPort()explicitlyunref()s the worker'sparentPort(and sibling ports);threadServer.jsonly.ref()s it afterloadRootComponents()resolves.persistent: false; the resource-report and monitoring timers areunref()d.So during
loadRootComponents(), anyawaitwhose completion arrives through a source that does not hold the event loop lets the loop drain — and a worker whose loop drains exits cleanly with code 0, before the ready handshake.createWorkerReadyPromisethen rejects,startHTTPThreads'Promise.all(workersReady)rejects, andbin/run.ts's catch callsprocess.exit(1)— the whole node aborts during boot.The natural trigger for exactly this window is rocksdb-js: it delivers cross-thread lock-release wakes (
DBDescriptor::lockEnqueueCallback, used byStore.tryLock(key, onUnlocked)) and parked-commit retry wakes (coordinated-retryRETRY_NOWintransaction.cpp) through threadsafe functions that arenapi_unref_threadsafe_function'd at creation. A pre-ready worker parked on a peer worker's lock — e.g. the cross-threadstore.tryLock(componentDirectory)incomponentLoader.ts'ssymlinkHarperModule(), whose non-acquired branch additionally clears the only ref'd timer before waiting — contributes nothing to its own loop's liveness and can exit before the wake arrives. Four workers concurrently loading the same components at first boot make that contention structural; whether the loser happens to have other in-flight ref'd work at the drain instant is the flake coin-flip.Post-ready workers park on the same locks constantly but survive, because the ref'd
parentPort(and bound listeners) hold the loop — which is why this only ever surfaces as "exited … before reporting ready".The mechanism reproduces standalone: a worker with an unref'd
parentPortawaiting a completion behind an unref'd handle exits code 0 at the drain instant. The full-instance drain did not reproduce locally on a 20-core dev box (constant startup ITC traffic keeps reviving the loop there); it reproduces on loaded 4-vCPU CI runners.Cause vs. effect
The
Database not opencascade in sibling workers is an effect, not a second defect: main'sprocess.exit(1)tears down stores while sibling workers are still mid component-load (the stack in the log:processGraphQLSchema → table → openAuditStore → RocksTransactionLogStore).Fix
threadServer.js: holdparentPort.ref()for the entire pre-ready window (SHUTDOWN still unrefs for graceful exit) — a worker must report ready, throw, or be terminated; never silently drain.componentLoader.tssymlinkHarperModule(): the lock waiter keeps a ref'd bounded timer instead of clearing it (also fixes an unbounded wait if the lock holder dies, and removes the winner's stale 10s timer that couldunlock()a lock it no longer owned).A companion rocksdb-js issue covers the unref'd threadsafe-function wakes themselves (pending cross-thread lock/commit-retry wakes should keep the waiting thread's loop alive).
Related: #1827 (startup hang with no output) may share the
symlinkHarperModuleunbounded-lock-wait surface fixed here.