Skip to content

deploy_component: fail the deploy on peer replication failures (non-zero exit) - #1334

Merged
kriszyp merged 2 commits into
mainfrom
kris/deploy-fail-on-replication-error
Jun 17, 2026
Merged

kriszyp merged 2 commits into
mainfrom
kris/deploy-fail-on-replication-error

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 17, 2026 •

Copy link
Copy Markdown
Member

Closes #1333

Summary

deploy_component records per-node replication status in peer_results but never let a peer failure change the overall outcome — the deploy finished success, returned 2xx (SSE done), and the CLI exited 0 even when the package failed to replicate to the cluster.

This treats any failed peer as an overall deploy failure (non-2xx → non-zero CLI exit) by default, with an ignore_replication_errors: true opt-out for best-effort deploys to partially-available clusters. The component is still deployed and (if requested) restarted on the origin node; the restart runs before the failure is reported. The failure response carries failed_peers + deployment_id on both the SSE error event and the non-SSE structured body, so callers see which nodes failed and why without a second get_deployment round-trip.

Changes

  • components/operations.js — after the replicate (and restart) phase, throw when getFailedPeers() is non-empty and ignore_replication_errors is not set; the existing catch wraps it as a 500, finishes the row failed, and surfaces failed_peers/deployment_id on both transports. Also wrap recorder.finish('failed') so a row-write error can't mask the real deploy error.
  • components/deploymentRecorder.ts — new getFailedPeers() helper; normalizePeerResult now reads the replicator's { status: 'failed', reason, node } shape (failure detail is on reason, not error).
  • components/operationsValidation.js — ignore_replication_errors boolean added to the deploy validator.
  • Unit tests for getFailedPeers, the reason-shaped peer result, and a bare { status: 'failed' }.

Where to look / open questions for the reviewer

  • normalizePeerResult change is the highest-leverage bit: without it the failure message and failed_peers always read "unknown error", because replicateOperation (harper-pro replicator.ts) reports per-peer failures via reason, not error. Worth confirming the shape against the replicator you have in mind.
  • Terminal status taxonomy (deliberately not changed): a successful-origin-but-replication-failed deploy ends with the hdb_deployment row status = 'failed'. Per-peer granularity is preserved in peer_results, but the aggregate status alone can't distinguish "origin failed" from "origin OK, one peer offline". Raised in deploy_component: treat replication failures as an overall deploy failure (non-zero exit code) #1333 as an open question — kept as failed here to match the requested "overall failure" semantics; happy to add a replication_failed/partial status if preferred.
  • Pre-existing validation quirk (not fixed here): validateBySchema discards Joi's coerced value, so a raw-HTTP JSON ignore_replication_errors: "false" (string) would be truthy and suppress errors. Affects all boolean deploy flags (force, install_allow_scripts), not just this one; the CLI is safe (it JSON.parses flag values). Fixing the wrapper is broader than this change.
  • No core integration test for the end-to-end path: OSS core's replicateOperation is a stub that returns no peers, so the multi-node assertion belongs in harper-pro (alongside the existing deploy-tracking peer test). Covered here by unit tests of the detection logic.

Review

Cross-model review (thorough): Harper-domain (deep-review) + Gemini (agy). The Codex CLI was unavailable. Domain-pass findings (the reason/error shape mismatch and the SSE-vs-non-SSE failed_peers asymmetry) and the Gemini finding (a finish() write error could mask the informative deploy error) are all fixed in this PR. Remaining open considerations are the items above.

🤖 Generated with Claude Code (Opus 4.8)

deploy_component recorded per-node replication status in peer_results but
never let a peer failure change the overall outcome: the deploy finished
'success', returned 2xx / an SSE done event, and the CLI exited 0 even when
the package failed to replicate to the cluster.

Treat any failed peer as an overall deploy failure (non-2xx → non-zero CLI
exit) by default, with an ignore_replication_errors opt-out for best-effort
deploys. The component is still deployed and (if requested) restarted on the
origin; restart runs before the failure is reported. The failure response
carries failed_peers + deployment_id on both the SSE error event and the
non-SSE structured body.

Also fix normalizePeerResult to read the replicator's { status: 'failed',
reason, node } shape — failure detail lives on `reason`, not `error`, so
without this the failure message and failed_peers reported "unknown error".

Closes #1333

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gemini-code-assist

This comment has been minimized.

@kriszyp kriszyp self-assigned this Jun 17, 2026
@claude

claude Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

If recorder.finish('failed') rejects (full disk, lock, dropped system
table), the original outErr — carrying phase, install output, and
failed_peers — was swallowed and replaced by the DB write error. Wrap
finish() so outErr always propagates; log the record failure.

Surfaced by the Gemini cross-model review leg.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kriszyp
kriszyp requested review from cb1kenobi and hdbjeff and removed request for cb1kenobi June 17, 2026 01:31
@kriszyp
kriszyp marked this pull request as ready for review June 17, 2026 01:35
@gemini-code-assist

This comment has been minimized.

@kriszyp
kriszyp merged commit 65d2058 into main Jun 17, 2026
48 checks passed
@kriszyp
kriszyp deleted the kris/deploy-fail-on-replication-error branch June 17, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deploy_component: treat replication failures as an overall deploy failure (non-zero exit code)

1 participant