GH-4662: a durable publish that cannot be persisted fails the caller

[jeremydmiller]

Closes #4662.

⚠️ Behaviour change, but a narrower one than it first looks. See Where this does and does not apply below. Only a publish made outside a handler and outside any transaction can now throw.

PublishAsync/ScheduleAsync from outside a handler and outside any transaction, to a local queue made durable by UseDurableLocalQueues(), while the message store was unreachable, returned without an error. Four attempts were made in ~400 ms, the envelope was discarded with a single line at Information, and the message was never handled — not even after the database came back, because no inbox row was ever written.

Why

The problem is the shape of the retry, not the retry.

RetryBlock.PostAsync runs one attempt inline and posts the rest to its own worker, so the task the caller awaits completes successfully the moment the first attempt fails. By the time the block gives up, the caller has long since returned.

That is the correct shape once an envelope is durable somewhere and something else can recover it — it is exactly what #3770 relies on for a broker delivery, which can be deferred back to the broker. It is the wrong shape for the write that makes an envelope durable, because there is no other copy to fall back on: no broker delivery, no inbox row, nothing for recovery to find. There is no option here that is both silent and safe.

The fix

DurableWriteRetry keeps the same 50/100/250 ms budget, awaits it inline, and lets the last exception reach the caller. A transient blip still costs latency rather than an exception; a real outage now fails the publish.

Both first-write paths get it: DurableLocalQueue.StoreAndForwardAsync (the reported one) and DurableSendingAgent.storeAndForwardAsync (the same shape, which the reporter also observed while writing the #4658 repro). Their RetryBlocks are gone, along with the drain/dispose calls that went with them.

Where this does and does not apply

Traced through MessageBus.PersistOrSendAsync and MessageContext.FlushOutgoingMessagesAsync. There are three cases and only one of them changes:

1. A real outbox is enlisted (EF Core, Marten, DatabaseEnvelopeTransaction) — no impact at all.
Transaction != this, so the publish goes to envelope.PersistAsync(Transaction) and never reaches this code. The post-commit flush branches on ReferenceEquals(this, Transaction), which is false here, so it takes QuickSendAsync rather than StoreAndForwardAsync. This is the path the reporter noted already fails visibly, and it still does, by an entirely separate route.

2. Inside a handler with no explicit outbox — no new exception escapes.
ReadEnvelope sets Transaction = this, so a publish is buffered into _outstanding and cannot throw. The send happens later in FlushOutgoingMessagesAsync, whose loop catches, logs, and records a DiscardedEnvelope. A cascaded message whose store is down therefore does not turn a successful handler into a failed one. What changes is visibility: the discard moves from an Information line buried in a RetryBlock to an Error plus a real tracking event.

3. Outside a handler and outside any transaction — this is the change.
Transaction is null, so the exception reaches the caller.

Release note

A durable publish or schedule made outside a transaction and outside a handler now surfaces a store failure to the caller instead of discarding the message. Applications that call PublishAsync/ScheduleAsync to a durable endpoint from outside a handler should be prepared to handle the exception. Publishing through an outbox, and cascading messages from inside a handler, are unchanged.

Logging that this makes reachable

Three places assumed a durable store failure could never reach them. All three are now corrected:

• the flush loop logged "most likely due to serialization issues" — a fair guess when a store failure was absorbed upstream, now misleading, since an unreachable store is the more common of the two causes. It names both, and names the envelope, which it never did;
• both acknowledgement paths claimed their catch "should never happen". Both are reachable now, and both still swallow on purpose — a lost acknowledgement is not worth failing a message that was handled successfully. The comments now say so rather than the opposite.

This also makes an existing comment in SendingAgent.StoreAndForwardAsync true. It only logs Sent after storeAndForwardAsync returns, and says "a store that throws still reports no send" — but nothing could throw there before, so a discarded envelope was reported as sent.

Testing

durable_publish_failure_reaches_the_caller_4662 runs against a real host whose store is a NullMessageStore subclass with only the inbox made unreachable, so the host boots exactly as it normally does and the only variable is the write. Five facts: publish throws, schedule throws, the full retry budget is still spent first (>350 ms, 4 attempts), a store that recovers inside the budget still succeeds, and a cascaded message is discarded rather than failing its handler — which pins case 2 above, and asserts the cascade really did reach the store and fail there so it cannot pass vacuously.

Reverting only the production files fails all of them.

Also green: CoreTests (3158), PostgresqlTests (615 — the suite that exercises DurableSendingAgent hardest), and dotnet build wolverine.slnx -c Release -f net9.0 at 0 warnings / 0 errors.

Not in scope

#4664 (a completion that cannot be written leaves the envelope owned by a live node) and #4659 (deferred tenant-scoped envelopes redeliver with no delay) are the other two faces of this retry budget, each on its own issue and PR. #4664 in particular needs the opposite treatment — the envelope is already durable there, so the answer is holding on rather than throwing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj

[jeremydmiller]

Released in Wolverine 6.42.0.