A production-shaped study of asynchronous delivery in modern PHP: Dev Containers, a Symfony monorepo, FrankenPHP/Caddy, transactional outbox publishing, idempotent APIs, ULIDs, RabbitMQ messaging, and Prometheus/Grafana observability.
Most webhook examples stop at an HTTP client. This project focuses on the harder part: reliably moving an accepted event across database, broker, worker, and an unreliable subscriber while making the resulting system observable and testable.
Client
│ POST /v1/events (Idempotency-Key)
▼
API ── transaction ──► Postgres: event + outbox row
│
▼
Outbox publisher ──► RabbitMQ ──► Worker ──► Subscriber
│
retries exhausted
▼
Redis failure transport
Prometheus ── scrapes /metrics ──► API
│
└───────────────────────────► Grafana dashboard
This is a backend portfolio project about the boundaries where distributed systems usually become interesting: duplicate requests, partial failures, delayed work, message publication, retries, and operational visibility. It deliberately keeps the product surface small so those engineering choices are easy to inspect.
The API and worker are separate Symfony deployables in one monorepo. They share message contracts, persistence entities, and metrics primitives without becoming a single runtime.
POST /v1/events stores the event and an outbox row in the same Postgres
transaction. A separate publisher reads the outbox and sends the delivery message
to RabbitMQ with publisher confirms. This avoids the classic dual-write gap where
a request has been saved but its message was never published.
Clients supply an Idempotency-Key. Replaying the same canonical JSON payload
returns the original event; reusing a key for a different payload is rejected.
Event IDs are ULIDs, which are stable to expose and naturally sortable by creation
time.
The worker retries failed subscriber requests with exponential backoff. Exhausted messages move to the Redis-backed failure transport. A crash after the subscriber accepts a request but before the worker persists success can still cause a repeat, so subscribers must be idempotent too. The project makes that trade-off explicit instead of claiming exactly-once delivery.
The API exposes Prometheus metrics for accepted events, idempotent replays, delivery outcomes, latency, event states, outbox backlog, and failed messages. Grafana provisions a dashboard for those signals automatically. A scripted demo creates steady traffic, replay activity, failures, and recovery so the dashboard shows a useful operational story rather than an empty screen.
| Area | Choice | Why it matters here |
|---|---|---|
| Runtime | PHP 8.4, Symfony 7.4 | Typed modern PHP, Symfony Messenger, and separate API/worker applications |
| HTTP server | FrankenPHP / Caddy | Production-style PHP serving rather than a development-only server |
| Persistence | Postgres 16 + Doctrine ORM | Transactional event and outbox persistence |
| Messaging | RabbitMQ 3 | Durable asynchronous delivery with retry handling |
| Failure transport and shared metrics | Redis 7 | Failed Messenger envelopes plus atomic cross-process metric aggregation |
| Observability | Prometheus + Grafana | Pull-based metric history and a provisioned dashboard |
| Quality | PHPUnit 11, Xdebug, PHPStan, GitHub Actions | Unit/integration coverage, coverage reports, static analysis, and an end-to-end CI path |
| Development environment | Dev Containers + Docker Compose | Repeatable tooling and a complete local dependency stack |
Prerequisites: Docker with Docker Compose. For editor tooling and direct PHP commands, open the repository in VS Code using the included Dev Container.
make up
make migrate
make check
make smoke| Service | Address |
|---|---|
| API | http://localhost:8080 |
| OpenAPI | http://localhost:8080/api/doc |
| RabbitMQ management | http://localhost:15672 (guest / guest) |
| Prometheus | http://localhost:9090 |
| Grafana dashboard | http://localhost:3000/d/webhook |
To populate Grafana with a compact end-to-end story, run:
make demo-dashboardThe demo takes about 90 seconds. In Grafana, select Last 15 minutes to see throughput, idempotent replays, delivery failures, p95 latency, and queue backlogs. Local Grafana is deliberately configured with anonymous admin access; this is a development convenience, not a deployment configuration.
curl -sS -X POST http://localhost:8080/v1/events \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: order-42' \
-d '{"type":"order.created","orderId":"42"}'The first request returns 202 Accepted with an event ID. Repeating it with the
same key and payload returns 200 OK and the same ID. Query that event to follow
its asynchronous status:
curl -sS http://localhost:8080/v1/events/<event-id>The primary event states are accepted, delivered, and failed.
apps/api/ HTTP ingestion, event status, health, metrics, OpenAPI, migrations
apps/worker/ outbox publisher and webhook delivery worker
packages/shared/ entities, message contract, metrics primitives
observability/ Prometheus scrape config and provisioned Grafana dashboard
mock-subscriber/ local receiver with a controllable failure mode
scripts/ happy-path, failure-path, and dashboard-demo workflows
.devcontainer/ reproducible PHP/Composer/editor environment
make check # Composer validation, container linting, schema check, PHPStan, PHPUnit
make smoke # accepted event, idempotent replay, eventual delivery
make smoke-failure # retries exhausted and failure transport path
make coverage # Xdebug-powered PHPUnit HTML coverage reportsGitHub Actions builds the Compose stack, migrates Postgres, runs the quality gate, and executes both end-to-end smoke paths.
- Subscriber registration, event filtering, and fan-out
- HMAC request signing and secret rotation
- Delivery-attempt history and a subscriber-facing audit view
- Outbox retention and operational alert rules