Skip to content

A replication worker type: dedicated threads that run Harper's built-in services without application code #3028

Description

@kriszyp

Part of HarperFast/harper-pro#435 (W6, dedicated replication threads; design in this comment). This is the core half; the harper-pro half routes replication onto the pool.

Problem

Replication runs on the HTTP worker threads, so catch-up sending, base copy and egress back-pressure compete with application request handling. Core has no way to start a worker that runs Harper's built-in services without application code: THREAD_TYPES has only http and job (utility/hdbTerms.ts:984-987).

Scope

  • A replication worker type started by the main thread when replication.threads > 0 (default 0, which changes nothing).
  • Generalize the isolated-application slot machinery (server/threads/socketRouter.ts:100-109,216-248, from feat(threads): run an isolated application in a dedicated worker thread #2524) instead of adding a second one-off pool. It already provides indices past the HTTP pool, heap-share accounting (heapShareCount) and scoped restart. Multi-tenant within process: SNI-based instance routing with isolated workers harper-pro#247 (SNI multi-tenant isolation) wants the same capability.
  • Readiness without application code: pool workers open the databases and tables and load trusted built-ins (replication), but no applications. placedOnThisThread (components/componentLoader.ts:178) decides placement per thread type. They signal ready without depending on threadServer.js HTTP startup.
  • Exclusive port ownership: a component on a worker type can own a port so that only those workers bind it. This generalizes the isolated-worker port skip at server/threads/threadServer.js:376. Replication uses it for replication.securePort/port.
  • Restart wiring: restartWorkers('http', …) in bin/restart.ts:67,328 and components/operations.js:972,1631. Restarts also restart the pool, deploys of application code do not, and a rolling restart keeps at least one pool worker up where possible (as with OVERLAPPING_RESTART_TYPES, manageThreads.js:660).
  • Status: server/status/index.ts:137 and any other name === 'http' worker enumeration that should include the pool.
  • Broadcasts already reach every non-job port (manageThreads.js:359); add a test that pool workers receive schema broadcasts.
  • threads.count === 0 (main thread acts as the worker): the pool is unavailable. Log once and run replication where it runs today.

Out of scope

Routing replication onto the pool (harper-pro). Elastic or auto-scaling pools.

Acceptance

  • With replication.threads: 2, two workers named replication start, load no application code, and receive schema broadcasts.
  • A component can bind a port on those workers only.
  • restart and restart_service handle the pool; deploying an application does not restart it.
  • With the default 0, behavior and the thread list are unchanged.

🤖 Claude Opus 5.5 on behalf of Kris.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    Priority

    P1

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions