Skip to content

feat(storage): durable Postgres store + shipped migrations (FOR UPDATE SKIP LOCKED) (#38)#62

Merged
williamdewitt merged 1 commit into
mainfrom
feat/postgres-durable-store
Jun 28, 2026
Merged

feat(storage): durable Postgres store + shipped migrations (FOR UPDATE SKIP LOCKED) (#38)#62
williamdewitt merged 1 commit into
mainfrom
feat/postgres-durable-store

Conversation

@williamdewitt

Copy link
Copy Markdown
Owner

Closes #38 — the scale-up production storage tier for the M2 epic (#8). risk:critical (ships EF migrations → migration-safety review; does not auto-merge).

What this delivers

  • One-line opt-in, both modes:
    • Standalone: AddCaliberWebhooks(o => o.UsePostgres("Host=…;Database=…;Username=…;Password=…"))
    • Outbox: existing UseEntityFramework<TContext>() now detects an Npgsql caller context and shares its database.
    • Both route through a single CaliberNpgsqlConfiguration.Apply(...) so every entry point (standalone, outbox, design-time) configures Npgsql identically.
  • No-double-send claim, pushed to one atomic SQL statement per provider in EfMessageStore:
    • Postgres: UPDATE messages … FROM (SELECT id … WHERE due ORDER BY next_attempt_at FOR UPDATE SKIP LOCKED LIMIT @batch) … RETURNING … — N dispatchers claim disjoint batches with no contention.
    • SQLite (unchanged): equivalent UPDATE … RETURNING; the single writer serialises it.
    • Fan-out idempotent on (event_id, endpoint_id) via ON CONFLICT DO NOTHING (both providers).
  • Shipped, reviewed migration (InitialCreate) applied on startup — CaliberWebhooksPostgresMigrator (standalone) / provider-aware CaliberWebhooksSharedSchemaInitializer (outbox: migrate on PG, create-tables-if-missing on SQLite). No EnsureCreated in production.
  • Migrations tracked in a dedicated __caliber_migrations_history table so they never collide with the caller's own migrations when Caliber shares the caller's database (outbox mode).
  • Microsoft.EntityFrameworkCore.Design is design-time only (PrivateAssets=all) — never flows to consumers. Npgsql provider pinned to the EF Core 8 line (8.0.11) to unify with the existing Sqlite 8.0.28 pin under transitive pinning.

Migration-safety review

Generated SQL (dotnet ef migrations script --idempotent) is purely additive and idempotentCREATE TABLE / CREATE INDEX guarded by IF NOT EXISTS … __caliber_migrations_history. No DROP, no destructive ALTER, no data movement. Indexes match the access paths:

  • IX_messages_status_next_attempt_at — the claim predicate + ORDER BY.
  • unique IX_messages_event_id_endpoint_id — backs idempotent fan-out (ON CONFLICT).
  • IX_endpoints_enabled — endpoint lookup.

Verification

  • dotnet build0 warnings, 0 errors (net8.0 + net10.0).
  • dotnet test203 / 203 pass on both targets.
  • dotnet ef migrations script generates clean Postgres DDL.

Scope / follow-ups

🤖 Generated with Claude Code

…E SKIP LOCKED)

The scale-up production tier (#38). `UsePostgres(conn)` (standalone) and the
outbox `UseEntityFramework` path now target Npgsql, sharing one
`CaliberNpgsqlConfiguration` so every entry point configures the provider
identically.

EfMessageStore's claim path is one atomic statement per provider: on Postgres
`UPDATE … FROM (SELECT … FOR UPDATE SKIP LOCKED LIMIT @Batch) … RETURNING`, so
N dispatchers claim disjoint batches with no contention and no double-send; the
SQLite variant keeps the equivalent `UPDATE … RETURNING` (single writer
serialises it). Fan-out stays idempotent via `ON CONFLICT (event_id,
endpoint_id) DO NOTHING`.

Schema is provisioned by a shipped, reviewed migration (InitialCreate),
applied on startup by CaliberWebhooksPostgresMigrator (standalone) and the
provider-aware CaliberWebhooksSharedSchemaInitializer (outbox). Caliber tracks
its migrations in a dedicated `__caliber_migrations_history` table so they
never collide with the caller's own migrations when sharing a database.
Microsoft.EntityFrameworkCore.Design is design-time only (PrivateAssets=all).

The cross-instance no-double-send proof (Testcontainers) is #39, blocked by
this; its test-only deps land there.

Closes #38

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added the risk:critical SSRF/signing/secret/public-API/migrations; review + design (Opus, max) label Jun 28, 2026
@williamdewitt williamdewitt merged commit 38e962b into main Jun 28, 2026
9 checks passed
@williamdewitt williamdewitt deleted the feat/postgres-durable-store branch June 28, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:critical SSRF/signing/secret/public-API/migrations; review + design (Opus, max)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(storage): durable Postgres store + shipped migrations (FOR UPDATE SKIP LOCKED)

1 participant