Native Durable Execution Engine

Overview

The Native Durable Execution Engine is a lightweight Java-based system that enables durable, resumable workflow execution using plain Java code. The engine guarantees that each logical step in a workflow is executed at most once, even in the presence of process crashes or restarts.

---

Problem Statement

Long-running workflows often fail due to crashes, restarts, or partial execution. Re-running such workflows from scratch can:

• Re-trigger already executed side effects
• Corrupt system state
• Cause duplicate external actions

The goal of this assignment is to build a durable execution primitive that ensures:

A step that has completed successfully is never executed again.

---

Key Features

• Durable step execution using persistent storage
• At-most-once semantics per workflow step
• Automatic resume on restart
• Support for parallel execution
• Minimal API surface (single step() primitive)
• Native Java implementation (no DSLs, no frameworks)

---

Architecture Overview

Application Code
   │
   ▼
DurableEngine.step(id, fn)
   │
   ▼
SQLite Database (steps table)
   │
   ├── If DONE → return cached output
   └── If NOT DONE → execute & persist result

Each workflow step is identified by a logical step ID combined with a sequence number, ensuring correctness across loops and conditionals.

---

Technology Stack

Component	Technology
Language	Java 21
Concurrency	Virtual Threads + CompletableFuture
Persistence	SQLite
Serialization	Jackson
Build Tool	Maven

---

Project Structure

durable-engine/
├── pom.xml
├── README.md
└── src/main/java
    ├── com/example/durable
    │   ├── DurableEngine.java
    │   └── MainApp.java
    └── examples/onboarding
        └── EmployeeOnboardingWorkflow.java

---

Core Concepts

1. Durable Step Primitive

<T> T step(String id, Callable<T> fn)

• Executes the provided function only if the step has not already completed
• Persists the result upon successful execution
• Returns cached output on subsequent runs

---

2. Step Sequencing

To support loops and repeated step names, the engine internally generates a sequence number:

step-key = stepId + sequenceNumber

This mimics the logical clock used by production workflow engines.

---

3. Crash & Resume Behavior

1. Start workflow
2. Some steps complete and are persisted
3. Process crashes or is killed
4. Application restarts
5. Engine skips completed steps and resumes execution

---

4. Parallel Execution Support

The engine supports parallel steps using standard Java concurrency:

• CompletableFuture
• Executors.newVirtualThreadPerTaskExecutor()

Thread safety is ensured via synchronized access to the durable step method.

---

How to Run

Prerequisites

• Java 21+
• Maven 3.8+
• IntelliJ IDEA (recommended)

Steps

1. Open the project in IntelliJ IDEA
2. Ensure Project SDK is set to Java 21
3. Run MainApp.java
4. Kill the process midway
5. Re-run to observe durable resume behavior

---

Conclusion

This project demonstrates how durable execution semantics can be built natively in Java with minimal infrastructure. While simple, the engine captures the core ideas behind production workflow systems and serves as a strong foundation for further extension.