Summary
A Node worker thread shutting down segfaulted inside the RocksDB N-API addon's finalizer, killing PID 1 and taking the container down. It happened when the replication peer was stopped — i.e. during a step that a rolling upgrade performs on purpose, on every node.
harper-pro 5.2.3, @harperfast/rocksdb-js 2.7.1, RocksDB storage (post LMDB migration), two-node replicating pair, Linux x64 glibc.
Stack
PID 1 received SIGSEGV for address: 0x2d9000000e1
segfault-handler.node(+0x2e87)
libc.so.6(+0x3fdf0)
node(napi_get_reference_value+0x35)
@harperfast/rocksdb-js-linux-x64-glibc/rocksdb-js.node(+0x398b98)
node(node_napi_env__::CallFinalizer<true>(void (*)(napi_env__*, void*, void*), void*, void*)+0x6a)
node(+0x8c2f34)
node(node::Environment::RunCleanup()+0x9f)
node(node::FreeEnvironment(node::Environment*)+0x79)
node(node::worker::Worker::Run()+0xd0c)
node(+0xaa1e76)
libc.so.6(+0x92b7b)
libc.so.6(__clone+0x40)
Read bottom-up: a worker thread is exiting → FreeEnvironment → RunCleanup → napi finalizers → the addon calls napi_get_reference_value on a reference whose environment is already being torn down → fault on a bogus address. No JavaScript frames are involved.
Trigger — correlates with the peer going away
Log immediately before, hostnames genericised:
16:39:44 [warn] Reconciling replication subscriptions for nodes wedged disconnected on a live worker: ['wss://node-b:9933']
16:39:44 [warn] Reconciling 3 wedged subscription(s) for wss://node-b:9933 (staggered over 150ms)
16:39:45 [error][replication] Error in connection to wss://node-b:9933 due to WebSocket was closed before the connection was established (x3, one per http worker)
16:40:06 PID 1 received SIGSEGV
Node B (the replication peer) had just been stopped for its own upgrade. Worker threads were being recycled as part of the wedged-subscription reconciliation, and one segfaulted on the way out. The process died; the container had to be started again (RestartCount=0, so no restart policy fired).
Why this matters operationally
Stopping one node of a replicating pair is a required step of a rolling version upgrade. If this reproduces, upgrading node B can take down node A — the opposite of what a rolling upgrade is for. It fired once in one attempt here, so treat it as plausible-but-unconfirmed rather than deterministic.
Version signal (suggestive, not conclusive)
| node |
harper-pro |
rocksdb-js |
uptime on RocksDB |
SIGSEGV |
| single-node dev |
5.2.0 |
2.6.1 |
8 days |
0 |
| node A of a pair |
5.2.3 |
2.7.1 |
minutes |
1 |
Two variables differ, so this is not attribution: the dev node is standalone and therefore never experiences a replication peer disconnecting, which is the observed trigger. Noting it because harper-pro@main pins @harperfast/rocksdb-js: 2.7.1, so this is the version production upgrades will land on.
Questions
- Is a finalizer-time
napi_get_reference_value safe during Environment::RunCleanup()? If the addon keeps a napi_ref that can outlive env teardown, the reference should presumably be cleared on env-teardown notification (napi_add_env_cleanup_hook) rather than dereferenced from the finalizer.
- Is anything known to have changed here between rocksdb-js 2.6.1 and 2.7.1?
- Is worker recycling during replication-subscription reconciliation expected? If workers are torn down whenever a peer wedges, this finalizer path will be exercised routinely on any multi-node deployment, not just during upgrades.
Environment notes
RocksDB came back cleanly on restart and the node has been stable since, so no data loss is apparent (WAL recovery presumably handled it) — but it was a hard crash, so worth confirming that is guaranteed rather than lucky.
Summary
A Node worker thread shutting down segfaulted inside the RocksDB N-API addon's finalizer, killing PID 1 and taking the container down. It happened when the replication peer was stopped — i.e. during a step that a rolling upgrade performs on purpose, on every node.
harper-pro 5.2.3,
@harperfast/rocksdb-js2.7.1, RocksDB storage (post LMDB migration), two-node replicating pair, Linux x64 glibc.Stack
Read bottom-up: a worker thread is exiting →
FreeEnvironment→RunCleanup→ napi finalizers → the addon callsnapi_get_reference_valueon a reference whose environment is already being torn down → fault on a bogus address. No JavaScript frames are involved.Trigger — correlates with the peer going away
Log immediately before, hostnames genericised:
Node B (the replication peer) had just been stopped for its own upgrade. Worker threads were being recycled as part of the wedged-subscription reconciliation, and one segfaulted on the way out. The process died; the container had to be started again (
RestartCount=0, so no restart policy fired).Why this matters operationally
Stopping one node of a replicating pair is a required step of a rolling version upgrade. If this reproduces, upgrading node B can take down node A — the opposite of what a rolling upgrade is for. It fired once in one attempt here, so treat it as plausible-but-unconfirmed rather than deterministic.
Version signal (suggestive, not conclusive)
Two variables differ, so this is not attribution: the dev node is standalone and therefore never experiences a replication peer disconnecting, which is the observed trigger. Noting it because
harper-pro@mainpins@harperfast/rocksdb-js: 2.7.1, so this is the version production upgrades will land on.Questions
napi_get_reference_valuesafe duringEnvironment::RunCleanup()? If the addon keeps anapi_refthat can outlive env teardown, the reference should presumably be cleared on env-teardown notification (napi_add_env_cleanup_hook) rather than dereferenced from the finalizer.Environment notes
RocksDB came back cleanly on restart and the node has been stable since, so no data loss is apparent (WAL recovery presumably handled it) — but it was a hard crash, so worth confirming that is guaranteed rather than lucky.