[Epic] Durable Execution v0.1 #752

Summary

Tracking issue for a Harper-native durable execution engine: a workflow runtime where step results are checkpointed, workflows survive crashes via replay, signals/sleeps survive node restarts, and — Harper's structural differentiator — workflow checkpoints commit atomically with multi-model application data inside a single Harper transaction.

Source proposal: design doc (Durable Execution on Harper) — happy to attach on request, but the salient details are below and broken out into child issues.

Why

Every major agent framework (LangGraph 1.0, LlamaIndex Workflows, Pydantic AI, OpenAI Agents SDK + Temporal) has converged on durable execution as a baseline for production AI agent workloads. Harper's pitch is unique here: because the database engine and application/Resource layer run in the same Node.js process, the workflow checkpoint can ride inside the same LMDB/RocksDB transaction as the user's table writes. Temporal cannot do this. DBOS can do it only when every write is SQL against the same Postgres-compatible database. For multi-model agent data, Harper would be the only credible option.

Architectural position

Two compounding benefits:

1. Atomic commits between workflow state and multi-model application data when data lives in Harper.
2. Lower engine overhead per step regardless of where data lives — every step transition is in-process function calls + a local commit, not several network round trips.

These support both "system of record stays in Harper" and "system of record stays elsewhere; agent working data + workflow state in Harper" deployments.

Scope of v0.1

New-core primitives that cannot be approximated on top of existing Harper primitives:

• Consensus-replicated lane for workflow state #753 — Consensus-replicated lane for workflow state (per-shard Raft)
• Native durable timer service (sharded hierarchical timing wheel) #754 — Native durable timer service (sharded hierarchical timing wheel)
• Step memoization with idempotency keys for workflow replay #755 — Step memoization with idempotency keys
• Determinism contract for workflow code (SES Compartment policy) #756 — Determinism contract for workflow code (SES Compartment policy)
• Workflow developer API surface (Workflow base class, ctx.step, ctx.atomic, ctx.signal, ctx.sleep) #757 — Workflow developer API surface (Workflow base class, ctx.step, ctx.atomic, ctx.signal, ctx.sleep)

Reusable as-is (not separate issues)

The proposal identified primitives already present:

• `transaction(fn)` in resources/transaction.ts for multi-statement atomic writes within a database — direct foundation for `ctx.atomic`.
• `Context` propagation via `AsyncLocalStorage` — the natural carrier for workflow context.
• SES Compartment support in security/jsLoader.ts — substrate for the determinism sandbox.

POC-tier (do inline during prototyping)

Per scope decision, no separate tickets for POC-tier items now — they will be assembled on existing primitives during v0.1 prototyping and replaced with optimized core primitives as benchmarks expose ceilings:

• Commit-log subscription via `RocksTransactionLogStore` `'aftercommit'` hook → needs a workflow-aware fast-path filter at scale
• Point-in-time replay via the existing audit-log + `getRecordAtTime` → needs a dedicated workflow event store with workflow-ID-keyed indexes and its own retention
• Signal delivery via the Resource subscription mechanism → may need a workflow-specific signal primitive for one-shot wake-and-resume routing
• Scheduler as a polling Resource on `(fireAt, workflowId, stepId, payload)` rows → replaced by the durable timer service (Native durable timer service (sharded hierarchical timing wheel) #754)

v0.2 follow-ups

• Cross-database atomicity for workflows that span databases. Multi-database writes inside `transaction(fn)` are chained sequentially with no 2PC (`DatabaseTransaction.this.next`) — partial-write hazard if the workflow checkpoint and user data live in different DBs. v0.1 answer: workflows scoped to a single database. v0.2 needs 2PC or a saga primitive at the workflow layer.

Alignment with related work

• #626 (Built-in Harper Agent Component) — the built-in agent is a natural first internal consumer once the engine exists. Its `schedule_followup` tool (currently `setTimeout(...).unref()` on main thread) should ride the durable timer service (Native durable timer service (sharded hierarchical timing wheel) #754). Its `hdb_agent_session` + `pendingApprovals` are bespoke versions of `ctx.step` checkpointing + `ctx.signal`. Built-in Harper Agent Component harper-pro#676 should ship independently; revisit when v0.1 lands.
• Add ability to exclusively lock a record for performing a safe operation #483 (Exclusive record locks) — complementary, not a substitute. Locking solves "single executor" for a workflow; consensus (Consensus-replicated lane for workflow state #753) solves "checkpoint is durable before we ack the step." The runtime probably wants both.

What this is not solving

• Universal cross-system atomicity. External calls (LLM providers, Stripe, third-party APIs) remain at-least-once with idempotency keys.
• A turnkey Temporal replacement at scale. Different sweet spot: workloads where the atomic-commit-with-data guarantee is decisive.
• Multi-tenant managed workflows. Workflows for untrusted multi-tenant scenarios require hardening SES Compartment isolation or process-level isolation — out of scope for initial release.

Open product questions

• Design partner first vs. spec first (proposal recommends design partner before v0.1 lands).
• Declarative graph DSL in addition to imperative TS workflows?
• User control over sharding granularity?
• Pricing/packaging — Harper feature, separate tier, or Fabric-only?
• Determinism strictness — mechanical SES enforcement vs. convention + warnings?

🤖 Filed by Claude on behalf of @kriszyp