TLS certificates not auto-reloaded on worker threads (flipping/stale cert after renewal) #586

When TLS certificates are renewed on disk, Harper's worker threads continue serving the old certificate until the process is restarted. touch-ing the cert files does not fix it. Multiple customers have hit this — a customer, a customer ([redacted-customer-host]), and at least one cluster where synthetic monitoring saw responses flipping between the old cert and the new cert because different worker threads hold different cached certs.

Root cause (likely)

loadAndWatch for certificate files is gated on isMainThread in security/keys.js:190:

if (path && isMainThread) {
    loadAndWatch(
        path,
        (certificate) => { ... },
        'certificate'
    );
}

So only the main thread's chokidar watcher fires on cert file change. Each HTTP worker thread loaded its own secure context at startup and is never notified. Devin's synthetics observation — flipping between two different certs over time — is exactly what you'd see if some worker threads have refreshed and others haven't.

There's a databases?.system.hdb_certificate.subscribe(...) call in the same file around line 850 that may be intended as the cross-thread propagation mechanism, but it doesn't appear to drive setSecureContext on each worker's server in a way that replaces the cached context.

Evidence from Jira

• 2026-03-17 (a customer rd-stage): fullchain.pem updated on host but openssl s_client still served the old cert. Touching the files didn't help. Restart fixed it.
• 2026-04-07 (Devin): Datadog synthetic monitor showed responses flipping between the old cert (R12 issuer) and the new cert (R13 issuer) over the same window — proving per-thread divergence.
• 2026-04-12 (a customer): certbot renewed [redacted-customer-host] cert, openssl s_client still served the old one, another renewal didn't help, restart was required. CHANGE-563 tracks the resulting CR.

Customer-incident tickets that link to this

• CORE-3039 — "No Node Found / database_sockets disconnecting" — the CERT_HAS_EXPIRED log on cluster peers is the same stale-cert bug from the replication side. Maurice's investigation confirmed certs on disk were valid but Harper saw them as expired.

Acceptance criteria

• When cert files change on disk, all worker threads pick up the new cert without a Harper restart.
• The on-disk cert and the cert served by every worker thread match within a small window (seconds, not "until restart").
• A regression test (or at minimum a documented manual smoke test) writes a new cert to disk and asserts openssl s_client reflects it on every Harper port.

Likely implementation direction

1. Lift the isMainThread guard from the file watcher, or keep the main-only watcher and have it broadcast a "cert changed" ITC event to all worker threads, each of which then reloads the file and calls server.setSecureContext(secureOptions).
2. Verify the hdb_certificate.subscribe path is actually exercised by the file watcher and reaches each worker. If it is, why isn't setSecureContext (or the SNI default-context replacement) sticking?
3. The commented-out //server.setSecureContext?.(server, secureOptions); at security/keys.js:837 was disabled intentionally per the inline comment — verify whether that intentional disable is part of the problem.

Related recent commits (since ticket was filed)

• 32d10a8ef (2026-03-18) — Make TLS (re)loading explicitly async
• e223653ca (2026-03-18) — filter certs on update to ensure composite config is reflected
• ede574626 (2026-04-21) — Fix replication certificate configuration (SNI selector / cert uses)
• 2b866265e — downgrade TLS SNI log from info to debug

None of these address the main-thread-only watcher gate, which appears to be the still-active root cause.

Tracked in Jira: CORE-3036
Also addresses: CORE-3039 (customer-incident tracking for the same bug from the replication side)

🤖 Filed by Claude on behalf of Kris.