Replicated component deploys sometimes land on only one node (silent partial deploy) #163

Investigative. harperdb deploy --replicated=true of a local component sometimes lands on only one instance in a cluster, leading to inconsistent state (app visible in the UI on one node but not on a sibling, or vice versa). Reported pattern: more likely when the deploy takes a long time, or when there's an error in the component deploy. No reliable repro steps.

Why this matters

Partial-deploy is silent — the operator sees a successful deploy response from the node they targeted, but sibling nodes don't pick up the component. The drift is only caught later by users hitting an inconsistent endpoint or by manually comparing cluster state.

To investigate

• Trace the replicated-deploy code path: where does the deploy operation become a replication event, and what guarantees does it carry about cross-node acknowledgement?
• Does a deploy failure on a sibling node (e.g. install error, port collision, missing dep) bubble back to the originating deploy call, or is it swallowed?
• For long deploys: is there a timeout window in the deploy-replication path that drops the replication event without surfacing an error?
• Inventory the failure modes: which combinations of {component already deployed, install error, package extraction error, port-in-use} produce silent partial deploys vs. propagated errors?

Acceptance criteria

• A replicated deploy either succeeds on all targeted nodes or surfaces a clear error indicating which nodes didn't apply it.
• A small operator-facing way to confirm "is component X present on every node in the cluster" (component-status / cluster-status surface), if not already available.
• Document the expected semantics of --replicated=true (best-effort with reporting vs. all-or-nothing).

Related context

• Slack thread referenced in Jira: link

Tracked in Jira: CORE-3037
Status: Not Ready — needs repro / investigation.

🤖 Filed by Claude on behalf of Kris.