A fully serverless storage for event sourcing-based systems.
- What is Event Sourcing?
- What is Event Store?
- Getting Started
- Push Subscriptions
- Pull Subscriptions
- Architecture
- Data Model
- Ordering Guarantees
- Testing
- Limitations
Traditionally, software systems operate on state-based data. In other words, business entities and concepts are represented as a snapshot of their current state. E.g.:
| Id | Name | Team |
|---|---|---|
| 1 | Gillian | Administration |
| 2 | Krzysztof | Accounting |
| 3 | Robyn | Frontend |
In the above example, all we know about the data is its current state. But how did it get to the current state? — We don't know. The Event Sourcing pattern answers this and many other questions.
Event Sourcing introduces the dimension of time into the modeling of business entities and their lifecycles. Instead of capturing an entity's current state, an event-sourced system keeps a transactional record of all events that have occurred during an entity's lifecycle. For example:
{ "id": 3, "type": "initialized", "name": "Robyn", "timestamp": "2021-01-05T13:15:30Z" }
{ "id": 3, "type": "assigned", "team": "Frontend", "timestamp": "2021-01-05T16:15:30Z" }
{ "id": 3, "type": "promoted", "position": "team-leader", "timestamp": "2021-01-22T16:15:30Z" }
By modeling and persisting events, we capture exactly what happened during an entity's lifecycle. Events become the system's source of truth. Hence the name: event sourcing.
Not only can we derive the current state by sequentially applying events, but the flexible event-based model also allows projecting different state models optimized for different tasks.
Finally, Event Sourcing is not Event-Driven Architecture (EDA):
EventSourcing is not Event driven architecture. The former is about events inside the app. The latter is about events between (sub)systems
~ @ylorph
An event store is a storage mechanism optimized for event-sourcing-based systems. It should provide the following functionality:
- Append events to a stream (stream = events of a distinct entity).
- Read events from a stream.
- Concurrency management to detect collisions when multiple processes write to the same stream.
- Enumerate events across all streams (e.g., for CQRS projections).
- Push newly committed events to interested subscribers.
All of the above functions are supported by the Elastic Event Store.
-
Install AWS SAM CLI and configure your AWS credentials.
-
Clone the repository:
git clone https://github.com/doitintl/elastic-event-store.git
cd elastic-event-store- Build and deploy a new instance:
sam build
# ... Build Succeeded
sam deploy --guided
# ...
# ApiEndpoint: https://XXXXXXXXXXXX.execute-api.XXXXXXXX.amazonaws.com/Prod/Verify installation:
curl https://XXXXXXXXXXXX.execute-api.XXXXXXXX.amazonaws.com/Prod/version
# { "version": "0.0.1" }EES_URL=https://XXXXXXXXXXXX.execute-api.XXXXXXXX.amazonaws.com/Prodcurl $EES_URL/streams/stream-aaa-111 \
-H 'Content-Type: application/json' \
-X POST \
--data @- <<BODY
{
"metadata": {
"command": "do_something",
"issuedBy": "me"
},
"events": [
{ "type": "init", "data": 1 },
{ "type": "sell", "data": 20 },
{ "type": "buy", "data": 5 }
]
}
BODYThe Elastic Event Store is opinionated about concurrency control: it is mandatory. When committing to an existing stream, you must specify the expected last changeset:
curl "$EES_URL/streams/stream-aaa-111?expected_last_changeset=1" \
-H 'Content-Type: application/json' \
-X POST \
--data @- <<BODY
{
"metadata": {
"command": "do_something_else",
"issuedBy": "me"
},
"events": [
{ "type": "buy", "data": 100 },
{ "type": "buy", "data": 220 },
{ "type": "sell", "data": 15 }
]
}
BODYcurl $EES_URL/streams/stream-aaa-111/changesetscurl $EES_URL/streams/stream-aaa-111/eventscurl $EES_URL/streamsNote: Statistics are updated asynchronously every minute.
The CloudFormation stack includes two SNS FIFO topics:
ees_changesets_XXX_XXX_.fifo— for new changesetsees_events_XXX_XXX_.fifo— for individual events
To enumerate global changesets:
curl "$EES_URL/changesets?checkpoint=0"Use the next_checkpoint value to fetch the next batch. This endpoint is critical for CQRS projections and state rebuilds.
- REST API exposed via API Gateway
- System logic in AWS Lambda
- Events stored in DynamoDB
- DynamoDB Streams trigger Lambdas for global indexing and publishing
- SNS FIFO topics for push subscriptions
- SQS DLQs for failed stream processing
Each partition in the events table represents a stream — i.e., a business entity's event history.
Main DynamoDB schema:
| Column | Type | Description |
|---|---|---|
| stream_id | Partition Key (String) | Stream ID |
| changeset_id | Sort Key (Number) | Commit ID in stream |
| events | JSON (String) | Committed events |
| metadata | JSON (String) | Changeset metadata |
| timestamp | String | Commit timestamp |
| first_event_id | LSI (Number) | First event ID in stream |
| last_event_id | LSI (Number) | Last event ID in stream |
| page | GSI Partition (Number) | For global ordering |
| page_item | GSI Sort (Number) | Index within global page |
- Intra-stream order is preserved and strongly consistent.
- Inter-stream order is not guaranteed but is repeatable — global enumeration always yields the same result.
- Set the
SAM_ARTIFACTS_BUCKETenvironment variable:
export SAM_ARTIFACTS_BUCKET=your-bucket-name- Deploy the test environment:
./deploy-test-env.sh- Run unit tests:
./run-unit-tests.sh- Run unit and integration tests:
./run-all-tests.shBecause DynamoDB is used:
- Maximum item (changeset) size: 400 KB
- Maximum item collection (stream) size: 10 GB
As with all serverless solutions, at high scale, a self-managed deployment may be more cost-effective.