Skip to content

Worker threads never close their inspector: replaced workers fail to open the debug port with EADDRINUSE, logged only at trace #2108

Description

@harper-joseph

Summary

Worker threads never close their V8 inspector, so after any worker generation is replaced the
inspector ports stay bound and the new workers fail to open theirs. The failure is caught and logged
at trace, so in production it is completely invisible — worker-thread debuggability is silently
and permanently lost until a full process restart.

Cause

core/server/threads/threadServer.js registers the close handler only for the main thread:

if (isMainThread) {
    port = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_PORT) ?? 9229;
    const closeInspector = () => {
        try { require('inspector').close(); } catch (error) { ... }
    };
    for (const signal of ['SIGINT', 'SIGTERM', 'SIGQUIT', 'exit']) {
        process.on(signal, closeInspector);
    }
} else {
    // workers: no closeInspector registered anywhere
    const startingPort = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT);
    if (startingPort && getWorkerIndex() >= 0) port = startingPort + getWorkerIndex();
}

try {
    require('inspector').open(port, host, waitForDebugger);
} catch (error) {
    harperLogger.trace(`Could not start debugging on port ${port} ...`);   // <-- trace
}

Worker N binds THREADS_DEBUG_STARTINGPORT + N. When that worker is replaced (config reload, worker
restart, worker crash-and-respawn) it exits without closing the inspector, the port stays bound, and
the replacement worker's inspector.open() throws address already in use.

Impact

Observed 96 occurrences on a single production node. Every worker that had been replaced was
undebuggable, with no error, warning, or metric to indicate it — the only way to find out is to try
to attach and get nothing, or to grep at trace level.

This matters specifically because the main thread is often not where the interesting state lives. We
just spent hours on a snapshot-retention investigation (harper-pro#662) where the holder could have
been on any of 17 threads; we could only enumerate all of them after a full process restart
reopened every inspector.

Suggested fix

Register closeInspector for workers too. Worker threads don't receive process signals the same
way, so the natural hooks are the worker's own exit path — process.on('exit') inside the worker, or
whatever teardown already runs when a worker generation is retired.

Also worth raising the log level: a failure to open the inspector port is a diagnostic capability
silently disappearing, which is at least warn, not trace.

Detection

Host-side port probes lie, because docker-proxy accepts on the published port regardless of whether
anything inside the container is listening. Check inside the container's netns:

docker logs <container> 2>&1 | grep -c 'address already in use'
# or, listening sockets in the Harper process's netns:
awk 'NR>1 && $4=="0A" {split($2,a,":"); printf "%d\n", strtonum("0x" a[2])}' /proc/<harper-pid>/net/tcp

Environment: harperfast/harper-pro:5.1.26, Node v24.18.1, THREADS_DEBUG_STARTINGPORT=9230,
THREADS_DEBUG_HOST=0.0.0.0, 16 http workers.

Activity

  1. added theissue type on Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions