Epic: DurabilityMode.ServerlessWithOutbox — a transactional outbox for serverless functions #5016

Summary

Wolverine's DurabilityMode.Serverless keeps the cold-start story lean, but it has no transactional outbox. Users on serverless platforms (Cloud Run, AWS Lambda, Azure Functions) who want Marten or EF Core transactional middleware and guaranteed delivery of cascaded messages have no supported option today. This epic adds a new DurabilityMode.ServerlessWithOutbox: a real outbox that works without any always-on background process, recovered by an externally triggered "durability pass" (cron, scheduler, job), with latency traded for cost.

This came out of a conversation with a community user running Wolverine + Marten (and a second service on EF Core) on Google Cloud Run with Google Cloud Pub/Sub, on a tight budget. They've volunteered to test it. That scenario is the driving use case, but the design is transport- and provider-agnostic.

The problem today

• DurabilityMode.Serverless drops the outbox without saying so. ServerlessEndpointsMustBeInlinePolicy (src/Wolverine/Configuration/IEndpointPolicy.cs) coerces every endpoint to EndpointMode.Inline. A UseDurableOutbox() endpoint is downgraded with no warning (only NativeAck gets one). Cascaded messages are then sent after the transaction commits with no outbox row behind them. If the commit succeeds and the send fails, or the instance is frozen or killed between the two, the message is lost.
• Background sending can't be relied on. On platforms that throttle CPU once the HTTP response is written, anything left to a background BufferedSender / DurableSendingAgent may never run.
• Nothing runs the durability work in this mode. No DurabilityAgent, no scheduled-job polling (WolverineRuntime.Agents.cs), and recurring schedules are refused at startup.
• Scheduled sends partly break. MessageRouterBase falls back to TransportConstants.DurableLocalUri for scheduled envelopes, which throws UnknownTransportException after Options.Transports.RemoveLocal(). The TODO in WolverineRuntime.HostService.cs already notes this.
• Wolverine.Http's existing HTTP transport receiver can't work here. HttpTransportExecutor.ExecuteBatchAsync() hands envelopes to a local queue (AgentForLocalQueue), and Serverless strips local queues.

The pieces of one recovery pass already exist, though. The RDBMS DurabilityAgent.buildOperationBatch() runs incoming and outgoing recovery plus BumpStaleOutgoingEnvelopesOperation, and RunScheduledMessagesOperation releases due scheduled messages. They are only driven by timers inside a long-running agent.

Proposed design

1. New DurabilityMode.ServerlessWithOutbox

A new mode value, chosen for discoverability over a flag on Serverless:

opts.Durability.Mode = DurabilityMode.ServerlessWithOutbox;

• Same lean runtime as Serverless: no node registration, no node agent controller, no leader election, no local queues, no listeners auto-started, no background durability timers.
• But the message store's outbox stays active: outgoing envelopes are persisted inside the handler's transaction (Marten, EF Core, or the RDBMS providers).
• Existing Serverless behavior doesn't change (no breaking changes). Every switch on DurabilityMode must handle the new value. This widens the test matrix; that's accepted.

2. owner_id = 0 from the start, plus a stale-time guard

In this mode, outgoing (and scheduled) envelope rows are written with owner_id = 0 (any node) from the beginning, since no node record exists to own them.

• Recovery only claims owner-0 rows older than DurabilitySettings.OutboxStaleTime. This avoids a race where a scheduled durability pass resends a row the originating request is still sending.
• OutboxStaleTime is nullable today. ServerlessWithOutbox needs a sensible default (for example 30–60 seconds, to be decided). That default is the recovery latency floor for messages whose inline send failed.
• The stale predicate on owner-0 rows needs a timestamp check that the current recovery queries don't have (today, owner_id = 0 means "immediately claimable"). This must be added per provider without changing Balanced/Solo behavior.

3. MessageContext.SendAllOutgoingMessagesInlineAsync(): a transport-agnostic flush

A new method on MessageContext that, after the transaction commits:

1. Sends every outgoing envelope right now, through an inline sender for its destination endpoint, regardless of how that endpoint is configured (buffered, durable, or inline). It awaits each send so all work finishes inside the request's CPU allocation.
2. Deletes the matching outbox rows for every envelope that was sent successfully, in one batched call per message store (not one round trip per envelope).
3. Leaves failed envelopes in the outbox (still owner_id = 0) for the next durability pass, and logs them.

In ServerlessWithOutbox, Wolverine's transactional middleware and the Wolverine.Http endpoint flush call this instead of the normal FlushOutgoingMessagesAsync() handoff to sending agents. It should also be public, so a hand-written function entry point can call it.

Open question, "DurableInline": should this be expressed as a real sending model, a DurableInline endpoint mode (persist in the transaction, send inline after commit, batch-delete), that we recommend for ServerlessWithOutbox, or that the mode applies quietly to every endpoint? Needs discussion. Batching the "mark sent / delete" calls applies either way.

4. One bounded durability pass, triggered from outside

A single core API that runs one pass of durability work and returns:

var report = await host.RunDurabilityPassAsync(maxDuration: 30.Seconds());
// report: outgoing sent / failed / remaining, scheduled released, incoming recovered, ...

The pass:

• recovers and sends stale owner-0 outgoing rows, batching the deletes;
• releases due scheduled messages: external destinations are sent; local message types are executed inline by the pass, since there are no local queues;
• recovers incoming rows;
• runs expiry cleanups (handled envelopes, dead letters, deduplication claims);
• is limited by both batch size and elapsed time, and reports what is left so the caller can loop;
• takes a non-blocking try-lock on the database (advisory lock or equivalent), so overlapping triggers skip instead of double-sending;
• has an extension point (for example an IDurabilityPassStep) for later additions, such as catching up Marten / Polecat async projections. That is out of scope for v1.

Three ways to trigger it:

• HTTP: app.MapWolverineDurabilityEndpoints() in Wolverine.Http, for Cloud Scheduler, EventBridge Scheduler, or timer-triggered functions. The docs must say to protect it (Cloud Run IAM/OIDC, function keys, and so on).
• CLI: for example dotnet run -- storage flush (or a new verb), for Cloud Run Jobs, scheduled Lambda invocations, and Kubernetes CronJobs.
• Programmatic: call the API directly.

Scheduled-message latency becomes the trigger interval (Cloud Scheduler's minimum is one minute). That's an explicit, documented trade-off.

5. Inbound: a transport-agnostic push receiver in Wolverine.Http

Serverless platforms deliver broker messages by push (Pub/Sub push subscriptions, EventBridge/SNS to HTTP, Event Grid, and so on). Add a receiver to Wolverine.Http that:

• accepts a push request, maps it to an Envelope through a pluggable, transport-supplied mapper (Pub/Sub would reuse its IPubsubEnvelopeMapper), and executes the handler inline;
• calls SendAllOutgoingMessagesInlineAsync() before responding;
• returns 2xx to ack, and non-2xx so the broker retries with its own backoff and dead-letter topic. Wolverine's "move to error queue" writes to the store's dead-letter table and acks;
• optionally uses the durable inbox to dedupe at-least-once redelivery by message id.

Exact API shape and how transports register mappers are TBD in the sub-issues.

6. Warnings

When Serverless (not ServerlessWithOutbox) downgrades a UseDurableOutbox() endpoint to Inline, log a warning saying the outbox guarantee no longer applies and pointing at ServerlessWithOutbox. (Warning vs. error is still to be discussed; leaning toward warning.)

Scope: every supported outbox

This is a slower epic. ServerlessWithOutbox must work against every message store that supports the outbox, each with compliance coverage:

• PostgreSQL (Wolverine.Postgresql)
• Marten (Wolverine.Marten)
• EF Core on PostgreSQL and SQL Server (Wolverine.EntityFrameworkCore)
• SQL Server (Wolverine.SqlServer)
• Polecat (Wolverine.Polecat)
• MySQL (Wolverine.MySql)
• Oracle (Wolverine.Oracle)
• SQLite (Wolverine.Sqlite)
• Fisher (Wolverine.Fisher)
• RavenDB (Wolverine.RavenDb)
• Cosmos DB (Wolverine.CosmosDb)
• Multi-tenanted message databases (pass iterates every tenant database)

Suggested order: core mode + SendAllOutgoingMessagesInlineAsync() + durability pass on PostgreSQL / Marten first (that's what the early tester runs), then fan out per provider, then the push receiver.

Open questions

1. A DurableInline sending model: explicit and recommended, or applied quietly under ServerlessWithOutbox?
2. Warning vs. error when plain Serverless downgrades a durable endpoint.
3. Default OutboxStaleTime for the new mode.
4. Whether any provider needs a schema change for the stale owner-0 predicate (an index on timestamp). If so, it needs an opt-in or migration.
5. The push receiver's mapper registration API.