Part of #752 (Durable Execution v0.1 epic).
Problem
Harper's current replication is asynchronous, peer-to-peer, last-writer-wins on timestamp with node-name tiebreak — verified in `Table.ts` at `precedesExistingVersion` (around line 4089) and supported by the node-name → node-id mapping in `nodeIdMapping.ts`. There is no Raft, no quorum, no leader election in steady state: `replicateTo` controls residency and an optional per-request commit-confirmation count, not a quorum write.
This is fine for application data, but unacceptable for workflow execution. A failure mode: node A commits a `step-7-completed` checkpoint locally, then crashes before the change propagates. Node B picks up the workflow, doesn't see the checkpoint, re-executes step 7 — and re-charges Stripe. LWW conflict resolution on recovery does not save us; the checkpoint can be lost outright.
Proposed shape
A per-shard Raft group on a dedicated workflow-state subsystem, alongside the existing replication path — not replacing it. Existing data continues to use the current high-throughput peer-to-peer lane; workflow checkpoints and step outputs use a quorum-replicated commit log.
- Sharding key: workflow ID. Each workflow has a strict serial order; cross-workflow operations remain eventually consistent.
- Quorum: standard Raft majority. A workflow step is not acknowledged committed until a quorum has it.
- Co-location: the shard for workflow W lives on the same nodes (or a subset) that hold W's checkpoint data, so the workflow runtime can wake on whichever node holds the leader of W's shard without an extra hop.
- Leader election + shard placement: out-of-band; placement should be observable and rebalanceable as load metrics develop.
Why this is not satisfied by #483
#483 (exclusive record locks) is concurrency control — it prevents two nodes writing the same record at once by broadcasting a lock acquisition. It does not make a locally-committed write durable across node failure. The two are complementary: locking ensures a single executor; consensus ensures the executor's checkpoints are durable before the step is acknowledged.
Open design questions
- Do we use an in-house Raft implementation or pull in a battle-tested library (etcd/raft, openraft, etc.) and bridge to it?
- Is the log persisted using RocksDB column families inside the existing engine, or a separate store?
- Cross-shard transactions (a workflow needing to write to data that lives in a different shard) — saga at the workflow layer is the v0.1 answer, but call it out.
- Snapshot + log-compaction strategy for long-running workflows.
Scope of this issue
v0.1 implementation of the consensus lane with per-workflow-shard Raft groups, integrated with the workflow checkpoint store from the step-memoization issue.
🤖 Filed by Claude on behalf of @kriszyp
Part of #752 (Durable Execution v0.1 epic).
Problem
Harper's current replication is asynchronous, peer-to-peer, last-writer-wins on timestamp with node-name tiebreak — verified in `Table.ts` at `precedesExistingVersion` (around line 4089) and supported by the node-name → node-id mapping in `nodeIdMapping.ts`. There is no Raft, no quorum, no leader election in steady state: `replicateTo` controls residency and an optional per-request commit-confirmation count, not a quorum write.
This is fine for application data, but unacceptable for workflow execution. A failure mode: node A commits a `step-7-completed` checkpoint locally, then crashes before the change propagates. Node B picks up the workflow, doesn't see the checkpoint, re-executes step 7 — and re-charges Stripe. LWW conflict resolution on recovery does not save us; the checkpoint can be lost outright.
Proposed shape
A per-shard Raft group on a dedicated workflow-state subsystem, alongside the existing replication path — not replacing it. Existing data continues to use the current high-throughput peer-to-peer lane; workflow checkpoints and step outputs use a quorum-replicated commit log.
Why this is not satisfied by #483
#483 (exclusive record locks) is concurrency control — it prevents two nodes writing the same record at once by broadcasting a lock acquisition. It does not make a locally-committed write durable across node failure. The two are complementary: locking ensures a single executor; consensus ensures the executor's checkpoints are durable before the step is acknowledged.
Open design questions
Scope of this issue
v0.1 implementation of the consensus lane with per-workflow-shard Raft groups, integrated with the workflow checkpoint store from the step-memoization issue.
🤖 Filed by Claude on behalf of @kriszyp