Summary
Worker threads never close their V8 inspector, so after any worker generation is replaced the
inspector ports stay bound and the new workers fail to open theirs. The failure is caught and logged
at trace, so in production it is completely invisible — worker-thread debuggability is silently
and permanently lost until a full process restart.
Cause
core/server/threads/threadServer.js registers the close handler only for the main thread:
if (isMainThread) {
port = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_PORT) ?? 9229;
const closeInspector = () => {
try { require('inspector').close(); } catch (error) { ... }
};
for (const signal of ['SIGINT', 'SIGTERM', 'SIGQUIT', 'exit']) {
process.on(signal, closeInspector);
}
} else {
// workers: no closeInspector registered anywhere
const startingPort = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT);
if (startingPort && getWorkerIndex() >= 0) port = startingPort + getWorkerIndex();
}
try {
require('inspector').open(port, host, waitForDebugger);
} catch (error) {
harperLogger.trace(`Could not start debugging on port ${port} ...`); // <-- trace
}
Worker N binds THREADS_DEBUG_STARTINGPORT + N. When that worker is replaced (config reload, worker
restart, worker crash-and-respawn) it exits without closing the inspector, the port stays bound, and
the replacement worker's inspector.open() throws address already in use.
Impact
Observed 96 occurrences on a single production node. Every worker that had been replaced was
undebuggable, with no error, warning, or metric to indicate it — the only way to find out is to try
to attach and get nothing, or to grep at trace level.
This matters specifically because the main thread is often not where the interesting state lives. We
just spent hours on a snapshot-retention investigation (harper-pro#662) where the holder could have
been on any of 17 threads; we could only enumerate all of them after a full process restart
reopened every inspector.
Suggested fix
Register closeInspector for workers too. Worker threads don't receive process signals the same
way, so the natural hooks are the worker's own exit path — process.on('exit') inside the worker, or
whatever teardown already runs when a worker generation is retired.
Also worth raising the log level: a failure to open the inspector port is a diagnostic capability
silently disappearing, which is at least warn, not trace.
Detection
Host-side port probes lie, because docker-proxy accepts on the published port regardless of whether
anything inside the container is listening. Check inside the container's netns:
docker logs <container> 2>&1 | grep -c 'address already in use'
# or, listening sockets in the Harper process's netns:
awk 'NR>1 && $4=="0A" {split($2,a,":"); printf "%d\n", strtonum("0x" a[2])}' /proc/<harper-pid>/net/tcp
Environment: harperfast/harper-pro:5.1.26, Node v24.18.1, THREADS_DEBUG_STARTINGPORT=9230,
THREADS_DEBUG_HOST=0.0.0.0, 16 http workers.
Summary
Worker threads never close their V8 inspector, so after any worker generation is replaced the
inspector ports stay bound and the new workers fail to open theirs. The failure is caught and logged
at trace, so in production it is completely invisible — worker-thread debuggability is silently
and permanently lost until a full process restart.
Cause
core/server/threads/threadServer.jsregisters the close handler only for the main thread:Worker N binds
THREADS_DEBUG_STARTINGPORT + N. When that worker is replaced (config reload, workerrestart, worker crash-and-respawn) it exits without closing the inspector, the port stays bound, and
the replacement worker's
inspector.open()throwsaddress already in use.Impact
Observed 96 occurrences on a single production node. Every worker that had been replaced was
undebuggable, with no error, warning, or metric to indicate it — the only way to find out is to try
to attach and get nothing, or to grep at trace level.
This matters specifically because the main thread is often not where the interesting state lives. We
just spent hours on a snapshot-retention investigation (harper-pro#662) where the holder could have
been on any of 17 threads; we could only enumerate all of them after a full process restart
reopened every inspector.
Suggested fix
Register
closeInspectorfor workers too. Worker threads don't receive process signals the sameway, so the natural hooks are the worker's own exit path —
process.on('exit')inside the worker, orwhatever teardown already runs when a worker generation is retired.
Also worth raising the log level: a failure to open the inspector port is a diagnostic capability
silently disappearing, which is at least
warn, nottrace.Detection
Host-side port probes lie, because docker-proxy accepts on the published port regardless of whether
anything inside the container is listening. Check inside the container's netns:
Environment:
harperfast/harper-pro:5.1.26, Node v24.18.1,THREADS_DEBUG_STARTINGPORT=9230,THREADS_DEBUG_HOST=0.0.0.0, 16 http workers.