Skip to content

fix(replication): report which peers failed instead of a failed operation - #1755

Draft
dawsontoth wants to merge 1 commit into
stagefrom
fix/replication-failure-copy-origin-succeeded
Draft

dawsontoth wants to merge 1 commit into
stagefrom
fix/replication-failure-copy-origin-succeeded

Conversation

@dawsontoth

@dawsontoth dawsontoth commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

When a replicated operation failed on the node's only peer, or on every peer, Studio's toast said "The operation failed on the single node" or "…on all N nodes". Both claims are false. Harper applies the operation on the node Studio called before it fans out (harper components/operations.js:1396-1402 for set_component_file, and the same order for DDL, add_component, drop_component and set_configuration). harper-pro's replicateOperation fills replicated[] from server.nodes, which lists only the peers (replication/replicator.ts:867-893 at harper-pro a6fde69). rejectReplicationFailures now reports only what the response shows: Failed to replicate to the peer node:, …to all 3 peer nodes:, or …to 1 of 2 peer nodes:, followed by the per-node reasons as before.

Found in the 2026-09-25 daily RUM review: 7 events in 2 sessions over the last 2 days, all file saves on the apps view where the one peer answered Client network socket disconnected before secure TLS connection was established. None appear earlier in the 30-day window.

The request still rejects on any peer failure. Only the message changed.

For the human reviewer

  1. Rejecting on a peer failure is unchanged, and it is the bigger problem. A 200 whose origin write landed still takes each caller's failure path. For example, CreateNewTableModal never runs onSuccess, so describe_all is not invalidated, the modal stays open, and a resubmit fails with "already exists". Delete Table, Add SSH Key and file saves behave the same way. Resolving with a warning toast would change the contract for every caller of the interceptor, so it needs its own design pass. I kept it out of this PR on purpose.
  2. The wording is neutral on purpose. An earlier draft said "Applied on the connected node, but…". The adjudicator pointed out that a success claim next to a UI that treats the write as failed makes the mismatch in (1) worse. "Failed to replicate to N of M peer nodes" is true however (1) is decided.
  3. The peer count relies on replicated[] excluding the called node. I traced this in harper-pro (above) and confirmed that central-manager's /Cluster/{id}/operation proxy passes the body through without touching replicated. The only replicated hit in its src is an unrelated comment. restart_service builds its own replicated entries without status: 'failed' (harper bin/restart.ts:259-305), so it never reaches this branch.

Verification

  • End-to-end route: not reproduced live. A partial failure needs a cluster with a peer that is down. The trigger is from production RUM, and the fix is covered at unit level.
  • src/integrations/api/replication.test.ts: the four rejection cases are updated, one per branch (the only peer, all peers, 1 of 2, 2 of 3). Against the old replication.ts all four fail (fails-on-base check).
  • Full gate under Node 24.21.0: vitest run 365 files, 3,334 passed; tsc -b, oxlint, dprint check all clean. The pre-commit hook also ran the full gate on the final commit.

Complexity: easy

🤖 Generated with Claude Code

Review-Coverage: authored=claude; ran=gemini,cursor-grok; adjudicated=domain; blocked=codex(model-unavailable); declined=cursor-composer,cursor-kimi,cursor-muse; rounds=3 @ d4b55da

Human-Review-Need: 4 (decisions: partial-peer-failure-rejects, origin-excluded-count, peer-node-vocabulary) @ d4b55da

…tion

Harper applies a replicated operation on the node Studio called before
fanning out, and `replicated` lists only the peers. When the lone peer (or
every peer) failed, Studio said "The operation failed on the single node" /
"on all N nodes", although the node it talked to had already applied the
change. Production RUM shows this on file saves whose one peer dropped TLS.

The message now names only what the response establishes: replication to
N of M peers failed. The request still rejects; only the message changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the replication failure error messages in rejectReplicationFailures to clarify that the failures occurred on peer nodes rather than the called node itself, and updates the corresponding unit tests. Feedback suggests simplifying the nested ternary operator and removing a redundant pluralize call in the error message construction for better readability.

Comment on lines +46 to +48
const failedPeers = failures.length === peerCount
? peerCount === 1 ? 'the peer node' : `all ${peerCount} peer nodes`
: `${failures.length} of ${pluralize(peerCount, 'peer node', 'peer nodes')}`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The nested ternary operator here can be simplified for better readability. Additionally, in the else branch (where failures.length !== peerCount), peerCount is guaranteed to be at least 2 (since failures.length >= 1 and failures.length < peerCount). Therefore, the call to pluralize is redundant because the plural form 'peer nodes' will always be used. We can simplify this to a direct string concatenation.

Suggested change
const failedPeers = failures.length === peerCount
? peerCount === 1 ? 'the peer node' : `all ${peerCount} peer nodes`
: `${failures.length} of ${pluralize(peerCount, 'peer node', 'peer nodes')}`;
const failedPeers = failures.length === peerCount
? (peerCount === 1 ? 'the peer node' : 'all ' + peerCount + ' peer nodes')
: failures.length + ' of ' + peerCount + ' peer nodes';

@github-actions

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 65.18% 9456 / 14507
🔵 Statements 65.41% 10112 / 15459
🔵 Functions 57.97% 2413 / 4162
🔵 Branches 60.24% 7161 / 11886
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/integrations/api/replication.ts 100% 100% 100% 100%
Generated in workflow #2008 for commit d4b55da by the Vitest Coverage Report Action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant