Repository navigation
Durable receiver settles unacked deliveries when the inbox database is unavailable (GH-3767) - #3770
Merged
Merged
Conversation
…able (GH-3767) When inbox persistence failed for a broker delivery on an unlatched DurableReceiver, receiveOneAsync rethrew into its RetryBlock, which retries 3 times over ~400ms and then discards. The delivery was left neither acked, nacked, nor deferred -- unacknowledged on a live consumer, which brokers like RabbitMQ never redeliver while the connection stays up. For any inbox outage longer than the retry window, that was permanent message loss under a durable inbox. The latched branch already had the right semantics (persist as owner 0, defer back to the listener). Now the unlatched persist-failure path defers back to the listener too, settling the delivery inline on the first failure instead of racing the listener pause: broker redelivery plus the paused listener's inbox-recovery restart carry the retry. Envelopes without a listener keep the old rethrow/retry behavior since there is nothing to settle with. This is the sibling of GH-3680, which fixed the latched path writing recovery-invisible inbox rows; this closes the remaining unlatched- first-casualties path. Closes GH-3767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ
This was referenced Aug 1, 2026
This was referenced Aug 12, 2026
This was referenced Aug 20, 2026
This was referenced Aug 28, 2026
This was referenced Sep 28, 2026
BlackChepo
pushed a commit
to BlackChepo/wolverine
that referenced
this pull request
Oct 5, 2026
… caller PublishAsync or ScheduleAsync from outside a handler and outside any transaction, to a local queue made durable by UseDurableLocalQueues(), while the message store was unreachable, returned WITHOUT an error. Four attempts were made in about 400 ms, the envelope was discarded with a single line at Information, and the message was never handled -- not even once the database came back, because no inbox row was ever written. A caller could not tell a published message from a lost one. The cause is the shape of the retry, not the retry itself. RetryBlock.PostAsync runs one attempt inline and posts the rest to its own worker, so the task the caller awaits completes successfully the moment the first attempt fails. That is the right shape once an envelope is durable somewhere and something else can recover it -- it is what JasperFxGH-3770 relies on for a broker delivery. It is the wrong shape for the write that MAKES an envelope durable, because there is no other copy to fall back on: no broker delivery, no inbox row, nothing for recovery to find. So DurableWriteRetry keeps the same 50/100/250ms budget, awaits it inline, and lets the last exception reach the caller. A transient blip still costs latency rather than an exception; an outage now fails the publish, which is what the EF Core and database outbox paths already did for the same outage. Both first-write paths get it: DurableLocalQueue.StoreAndForwardAsync and DurableSendingAgent.storeAndForwardAsync. Their RetryBlocks are gone, along with the drain and dispose calls that went with them. This also makes an existing comment true. SendingAgent.StoreAndForwardAsync only logs Sent after storeAndForwardAsync returns, and says so -- "a store that throws still reports no send" -- but nothing could throw there before, so a discarded envelope was reported as sent. This is a behavior change: a durable publish outside a transaction can now throw where it previously returned silently. That is the point, and it needs a release note. Reverting only the production files fails all four facts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3767 — a durable, buffered listener could permanently drop deliveries during a brief inbox database outage.
The reported chain is exactly right and reproduces in code review:
ProcessReceivedMessagesAsyncbatch persist fails →SignalInboxUnavailable()+ each envelope re-posted through_receivingOne(and the single-envelopeReceivedAsyncpath feeds the same block).receiveOneAsync(unlatched) →StoreIncomingAsyncthrows →SignalInboxUnavailable(); throw;RetryBlockretries 3 times over ~400ms (50/100/250ms pauses), then discards — the delivery is left unacked on a live consumer, and RabbitMQ never redelivers while the connection stays up.The listener pause (
PauseForInboxRecoveryAsync) does latch the receiver, and pending retry items that run after the latch take the latched path, which already persists-as-owner-0 + defers. The loss window is the race before the latch goes up — envelopes that exhaust their retries first are discarded unsettled.The fix
On a persist failure for a delivery that has a listener, settle inline on the first failure: defer back to the listener (the same semantics the latched branch applies) instead of rethrowing into
RetryBlock's discard. Broker redelivery plus the paused listener'sInboxHealthRestarterrestart carry the retry. Envelopes with no listener (nothing to settle with) keep the old retry behavior.This eliminates the race entirely rather than tuning the retry schedule — discarding is never the terminal state for an unsettled durable delivery.
Sibling of GH-3680 (latched-path rows persisted invisibly to recovery); this closes the remaining unlatched-first-casualties path the issue identified.
Testing
New
when_the_inbox_is_unavailable_for_an_unlatched_durable_receivertests cover both the single-envelope and batch arrival paths: every delivery is deferred at the listener, none completed, none processed. Notably the tests deliberately do not drain first — draining latches the receiver, whose defer semantics would mask the bug (the first version of the test passed on the unfixed code for exactly that reason). Verified red without the product change, green with it. Full CoreTests suite green (2181 passed).🤖 Generated with Claude Code
https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ