Repository navigation
Conversation
Add cleanup hooks to make sure that addons making use of this legacy API can do so safely in Worker threads or embedder-controlled Node.js instances. Refs: nodejs#63575 Fixes: nodejs#63540 Signed-off-by: Anna Henningsen <anna@addaleax.net>
This content is taken from nodejs#63575. The original commit message is preserved below: --- worker: fix premature addon unload Weak callbacks could run after addons were unloaded, leading to a crash. Signed-off-by: Mohamed Akram <mohd.akram@outlook.com>
|
Review requested:
|
| if (!isMainThread) { | ||
| const addon = require(`./build/${common.buildType}/binding`); | ||
|
|
||
| // Create some garbage |
There was a problem hiding this comment.
Is this necessary to test with the CleanupHook? I don't think it is necessary to trigger GC here, right?
There was a problem hiding this comment.
You're right, GC would prevent the test case from working, if anything – done in fcb0a4d
There was a problem hiding this comment.
@legendecas @addaleax It's not meant to trigger a GC immediately. The purpose of the loop is to make sure the WeakCallback runs, which it does after the addon is unloaded, and therefore causes a segfault absent a fix. Without that loop you won't be testing the visible effects of the clean up hook (i.e. that it removes the WeakCallback). The loop iterations were chosen such that this happens consistently.
There was a problem hiding this comment.
@mohd-akram If the weak callback runs here, won't that mean that the object is destroyed before the Environment and therefore the issue in #63540 will actually not occur? I can reproduce a fairly consistent crash with current Node.js versions (i.e. before this PR) when the loop is commented out and no consistent crash when it is active.
There was a problem hiding this comment.
If the weak callback runs here, won't that mean that the object is destroyed before the Environment and therefore the issue in #63540 will actually not occur?
On my system, it doesn't run there, it runs after.
I can reproduce a fairly consistent crash with current Node.js versions (i.e. before this PR) when the loop is commented out and no consistent crash when it is active.
Hmm, that's surprising. Definitely the opposite happening for me:
$ cat test.js
'use strict';
const common = require('../../common');
const { isMainThread, Worker } = require('worker_threads');
if (!isMainThread) {
const addon = require(`./build/${common.buildType}/binding`);
// Create some garbage
const arr = [];
for (let i = 0; i < 1e5; i++) arr.push(`${i}`.repeat(100));
new addon.MyObject(10);
// Should not segfault
throw new Error('exit');
} else {
const worker = new Worker(__filename);
worker.on('error', () => {});
}
$ node --version
v26.2.0
$ node test.js
zsh: segmentation fault node test.js
What OS are you on?
There was a problem hiding this comment.
It's arm64 macOS.
Testing out a few different combinations: Node.js 26.0.0 has this crash only without the extra array, Node.js 26.1.0 and 26.2.0 do not have it at all, and Node.js @ 40dc5a1 (base of this PR) has it only with the extra array.
I could probably dig deeper into the exact reasons, but regardless of what the outcome of that would be, I don't really see a good reason not to just include both versions as tests, so I've done that in 0cede86
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #63642 +/- ##
==========================================
+ Coverage 90.30% 90.32% +0.01%
==========================================
Files 730 732 +2
Lines 234802 236435 +1633
Branches 43957 44531 +574
==========================================
+ Hits 212041 213560 +1519
- Misses 14485 14596 +111
- Partials 8276 8279 +3 🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
|
Landed in 3518f93...be7ea27 |
This content is taken from #63575. The original commit message is preserved below: --- worker: fix premature addon unload Weak callbacks could run after addons were unloaded, leading to a crash. Signed-off-by: Mohamed Akram <mohd.akram@outlook.com> PR-URL: #63642 Fixes: #63540 Refs: #63575 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
|
https://github.com/nodejs/node/actions/runs/27140598797/job/80104749457
|
|
@addaleax Will this commit be backported to Node 24? #63540 (comment) mentioned that the crash also happens on Node 24. Thank you! |
8월 15일부터 CI가 7연속 빨간불이었다. Node 22는 늘 통과했고 24만 죽었다. 8월 12일 마지막 성공 런의 러너는 24.18.0이었고 첫 실패부터 24.19.0이다. 그 사이 우리 커밋은 룰북과 스킬 문서뿐이라 네이티브 쪽을 건드리지 않았다. 24.19.0이 node::ObjectWrap에 정리 훅을 붙이면서(nodejs/node#63642) better-sqlite3 11.x의 Database 소멸자가 RemoveEnvironmentCleanupHook의 CHECK_NOT_NULL(env)에서 abort한다. 스택이 정확히 그 경로다. Node 쪽 수정(nodejs/node#65196)은 아직 미머지고 24.19.0 뒤로 24.x 릴리즈도 없어서 기다릴 수가 없다. better-sqlite3 13.x가 N-API로 옮겨가며 이 경로를 없앴지만 darwin-arm64에서 Node 20/22가 깨지는 회귀(#1514)가 열려 있어 검증 없이는 못 올린다. 우선 러너를 묶어 불을 끄고 업그레이드는 따로 본다. 커버리지 업로드 조건도 함께 고친다 — 24로 비교하던 자리라 핀을 넣으면 영영 안 걸린다. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rt policy Node.js 24.19.0 added cleanup hooks to node::ObjectWrap (nodejs/node#63642), which aborts NAN-style addons like better-sqlite3 with 'Assertion failed: (env) != nullptr' during GC. This killed karakeep-workers on 2026-08-13 and karakeep-web on 2026-08-15, leaving the vhost 502ing since the module sets no Restart= policy. Build/run karakeep on nodejs_22 until a fixed Node 24.x reaches nixos-unstable (tracked in TODO.md), and restart both units on failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rewrite lifts the pin) (#56) better-sqlite3 13's ground-up N-API rewrite no longer subclasses node::ObjectWrap, which was the sole trigger for the 24.19 teardown abort ("RemoveEnvironmentCleanupHook: Assertion failed: (env) != nullptr", nodejs/node#63642). With sqlite 13 (merged in #52) the exact 24.18.1 pin is no longer needed, so the runtime stage follows node:24-alpine within the 24 major. Frontend-build stage left as-is. Staging experiment: confirm clean graceful shutdown (exit 0, no abort) on current 24.x. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The lifecycle boot report finally caught the killer in the act: # next-server (v16.2.12)[1]: void node::RemoveEnvironmentCleanupHook(...) # Assertion failed: (env) != nullptr That is a native abort, below JavaScript — no handler can catch it, which is why the report filed it as an unclean exit at a healthy 97 MB RSS. Node 24.19.0 (2026-08-03) added cleanup hooks to node::ObjectWrap (nodejs/node#63642), and NAN-style native addons — better-sqlite3 among them, current v12 included, so upgrading the addon is no way out — now abort the process at random when a hook is removed after its Environment is already gone. Timing- and GC-dependent: a process can run for hours and then die minutes after the next boot, which is exactly the shape of this incident (17 h up, then a crash loop within seconds of each start). The Dockerfile pulled the FLOATING node:24-alpine tag, so every CI image built since 2026-08-03 silently carried the regression — the "random 502, especially when I come back after a while" predates #8 and matches this window. The event-loop starvation #8 fixed and the swallowed SIGTERM #12 fixed were real, but this is the crash that kept the story going. Pin all stages (and Dockerfile.dev) to node:24.18.0-alpine, the last release before the regression, verified present on Docker Hub; no fixed 24.19.x exists at the time of writing. The comment in the Dockerfile says what to check (nodejs/node#63642) before ever bumping past it, and the README's troubleshooting section gains the third failure mode: an abort from below JavaScript, recognisable by the native stack trace sitting in docker logs just above the next boot's startup lines. Co-Authored-By: Claude <noreply@anthropic.com>
The remote runner was crash-looping under systemd: Node aborted with "Assertion failed: (env) != nullptr" (src/api/hooks.cc:142) from ~Statement -> ~ObjectWrap, restarting on failure and dying again. Node 24.19.0 (nodejs/node#63642) added cleanup hooks to node::ObjectWrap. That class is header-only, so the change is compiled into every addon built against those headers. better-sqlite3's Statement derives from it, and its destructor now calls RemoveEnvironmentCleanupHook(Isolate::GetCurrent(), ...). When V8 collects a dead statement during a GC driven by a platform task there is no entered v8::Context, so Environment::GetCurrent() returns null and Node aborts the whole process. Pin the runner to a Node whose headers predate that change, and make the pin impossible to half-apply: - .nvmrc is the single source of truth (24.18.1, the newest release without the regression). The machine's default Node is left alone. - build-remote-native.mjs re-executes itself under the pinned Node, so the addons get the same ABI no matter which Node invoked the build, and fails loudly if it ever compiles against affected headers. - run-remote-runner.sh resolves the same version at boot and refuses to start on a mismatch, so ExecStart never names a version and cannot drift from the one that did the build. This also covers the macOS plist, which reached the runner through `pnpm remote:runner` and so used whatever node was on PATH. Note that `pnpm install` and `pnpm test` still rebuild these addons with the default Node; run `pnpm remote:build` again before restarting the runner. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…IGABRT better-sqlite3 Node 24.19.0 (nodejs/node#63642) made ~ObjectWrap() call RemoveEnvironmentCleanupHook via Environment::GetCurrent(isolate), which CHECK-aborts when a GC finalizes a dead better-sqlite3 Statement outside JS execution (task-based GC — no entered context). The floating node:24-slim base pulled 24.19.0 since Aug 3, crashing ferry-cp with SIGABRT on agent runs and in ~8s boot loops. Not a worker-thread issue. Repro: better-sqlite3@11.10.0 + dead statements + async major GC aborts with the exact prod stack on node:24-slim, survives on 24.18.1-slim. Unpin only after upstream ships a fix and the repro passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rkers - pin every workflow's Node to 24.18.0 through a single NODE_VERSION constant, replacing the floating '24' that silently moved to 24.19.0 - Node 24.19.0 shipped nodejs/node#63642, which makes better-sqlite3's Statement destructor call RemoveEnvironmentCleanupHook after the environment is gone; the resulting assertion killed vitest workers with SIGABRT, dropped whole test files from the run, and failed the coverage gate with no code change to explain it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rkers - pin every workflow's Node to 24.18.0 through a single NODE_VERSION constant, replacing the floating '24' that silently moved to 24.19.0 - Node 24.19.0 shipped nodejs/node#63642, which makes better-sqlite3's Statement destructor call RemoveEnvironmentCleanupHook after the environment is gone; the resulting assertion killed vitest workers with SIGABRT, dropped whole test files from the run, and failed the coverage gate with no code change to explain it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Production was returning HTTP 500 on roughly a third to a half of all requests, with no deploy on our side. Vercel logs showed: Node.js process exited with signal: 6 (SIGABRT) (core dumped) # node::RemoveEnvironmentCleanupHook(...) at ../src/api/hooks.cc:142 # Assertion failed: (env) != nullptr ... Statement::~Statement() ... better-sqlite3 Cause is upstream and has nothing to do with this codebase. Node 24.19.0 (2026-08-03) added cleanup hooks to node::ObjectWrap (nodejs/node#63642) without the change that makes RemoveEnvironmentCleanupHook safe to call during garbage collection — that landed only in Node 26 (nodejs/node#63985) and has not been backported to 24.x. better-sqlite3's Statement extends node::ObjectWrap, so when V8 collects a statement there is no entered context, Environment::GetCurrent() returns nullptr, and the assertion aborts the whole process, killing every in-flight request on that instance. Node 22 and 20 predate the ObjectWrap change and are unaffected; the same trace is reported downstream in nexu-io/open-design#6462, where reverting from 24.19.0 to 24.18.0 fixed it. The host exposes only major versions and applies minors automatically, so 24.18.x cannot be pinned and 24.x will keep resolving to the broken build. The only available defence is pinning the major to 22.x. Also: - /api/health now reports process.version, so a runtime regression is diagnosable from outside without reading platform logs. - A test asserts the pin, so "upgrade to the latest Node" cannot silently reintroduce this before the upstream backport ships. Not done deliberately: bumping better-sqlite3 (every version extends ObjectWrap; #6462 crashed on 12.10.0), closing the DB on exit (the abort is at GC time, not teardown), and disabling Fluid compute (no evidence it addresses the assertion). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWrmd97evyLqwXLGqM3xae
Node 24.19.0 (nodejs/node#63642) breaks NAN-style native addons like better-sqlite3 with a SIGABRT during GC, which fired on almost every crawl and crash-looped karakeep-workers. Build karakeep against nodejs_22 (LTS, unaffected) until upstream ships a fix, and add Restart=on-failure to both services as a general safety net.
Node 24.19.0 added self-removing cleanup hooks to node::ObjectWrap (nodejs/node#63642). better-sqlite3 v11 uses the NAN-style ObjectWrap, whose statement destructor calls RemoveEnvironmentCleanupHook under GC with no entered context, so the process aborts: "Assertion failed: (env) != nullptr" at hooks.cc:142, SIGABRT. It fires on garbage collection, not on any request, which is why the bench crashes looked intermittent and unrelatable to what the operator was doing. The abort is not fail-safe - a powered loco keeps running until systemd restarts the unit ~5 s later. v13's node-addon-api rewrite removes the path entirely: the prebuilt linux-x64.node imports zero cleanup-hook symbols (60 napi symbols, none of AddEnvironmentCleanupHook/RemoveEnvironmentCleanupHook), confirmed against the binary on the bench. v13 also ships prebuilds that load on any Node >=22, so it no longer compiles on the box. Verified end to end: full suite green (1427 backend + 273 frontend), lint and tsc clean, drizzle-kit migrate applies all 15 migrations, the SqliteError codes the auth guard reads (SQLITE_CONSTRAINT_UNIQUE, SQLITE_CONSTRAINT_TRIGGER) survive, VACUUM INTO / readonly / fileMustExist still work, and the live Westgate Hollow DB passes integrity_check and foreign_key_check opened read-only under v13 on the bench. Bumps @fastify/static 8.3.0 -> 10.1.3 in the same PR, clearing a high advisory (route-guard bypass via encoded path separators / non-canonical paths) that lands on the deployment.md D3 SPA auth exemption. v10's only breaking change is setHeaders(reply); we don't use setHeaders. staticFrontend.test.ts covers the exemption path, traversal negative included, and stays green. engines.node raised 20 -> 22 (v13's floor). Docs moved with the code: deployment.md gains the pin rationale, current-state.md the long form, bootstrap.sh the corrected native-module note. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Every CI run since #114 merged has failed, always the same way: node::RemoveEnvironmentCleanupHook(v8::Isolate*, CleanupHook, void*) at ../src/api/hooks.cc:142 Assertion failed: (env) != nullptr Statement::~Statement() [core/node_modules/better-sqlite3/build/Release/better_sqlite3.node] Nothing asserts. Every test in the file passes and then the process aborts, which marks the file failed and takes the run with it. #114 is not at fault beyond being the first test to load the session store, and neither is the test: the abort is raised by the library's own destructor. Node 24.19.0, released 2026-08-03, added cleanup hooks to node::ObjectWrap (nodejs/node#63642). NAN-style native addons register through that path, so their destructors now call RemoveEnvironmentCleanupHook after the environment is gone, and that is an assert rather than a throw. better-sqlite3 11.10.0 is NAN-based. 13.0.0 is the first release built on N-API, which never touches node::ObjectWrap, so the assertion cannot fire. Measured, one variable at a time, each with a fresh npm ci: better-sqlite3 11.10.0 + node 24.19.0 abort, 5 runs out of 5 better-sqlite3 11.10.0 + node 24.18.0 clean better-sqlite3 13.0.3 + node 24.19.0 88 pass, 0 fail better-sqlite3 13.0.3 + node 24.18.0 88 pass, 0 fail Pinning the workflow to 24.18.x would also have turned CI green and was the wrong fix. The core loads better-sqlite3 at import and runs under pm2 on the host, so the exposure is not limited to CI: whenever the host moves to 24.19.0, the core starts aborting at teardown. A pin would have hidden that behind a green badge. Upgrade path checked rather than assumed. v12.0.0 dropped EOL node 18 and old Electron, and this repo already requires node >=24. v13.0.0's breaking change is packaging only: prebuild-install is gone and prebuilt binaries ship with the package. Neither changes the API, and the repo's whole usage is prepare, exec, close, pragma and transaction. Regenerating the lock hoists better-sqlite3 to one root entry where there were three per-workspace entries. Verified that core, connectors and insights/collector all still resolve it, on both node lines, from the committed package.json and lock via npm ci. Worth knowing when reproducing: on node 24.18.0 there is nothing to see. A prebuilt binary compiled against 24.18 headers also stays clean when run under 24.19.0, so a partial upgrade hides it too. The abort needs a fresh build against 24.19.0 headers.
In two commits to keep attribution to @mohd-akram's PR intact. If somebody feels strongly that they should be a single commit, I can turn them into one with a
Co-authored-bytag.src: add cleanup hooks to
node::ObjectWrapAdd cleanup hooks to make sure that addons making use
of this legacy API can do so safely in Worker threads
or embedder-controlled Node.js instances.
Refs: #63575
Fixes: #63540
test: add regression test for using
ObjectWrapin workerThis content is taken from #63575.
The original commit message is preserved below: