Repository navigation
ResequencerSaga cannot distinguish a legitimate replay from an out-of-order handle, so ordering violations are silent #4175
Description
Describe the bug
ResequencerSaga.ShouldProceed cannot distinguish a legitimate replay of an already-handled message from a message being handled out of order. Both take the same branch, both return true, and neither leaves any trace — so when ordering does break, the saga finishes in a state that is indistinguishable from a healthy run.
src/Wolverine/Saga.cs:89-92:
// Already processed in sequence, allow re-published messages through
if (message.Order.Value <= LastSequence)
{
return true;
}That branch exists for a good reason: the drain republishes pending messages, and a replayed message legitimately arrives with Order <= LastSequence. But it is also the branch a genuinely late message falls into, and the saga has no way to tell the two apart.
Where this came from
Found while investigating #4172 (fixed in #4174). The bug there produced ProcessedOrders = [1, 2, 4, 5, 3] where [1, 2, 3, 4, 5] was expected. What made it hard to notice is that the saga's own state looked perfect afterwards:
LastSequence = 5 <- correct
Pending = [] <- correct
ProcessedOrders = [1, 2, 4, 5, 3] <- wrong, and nothing else reflects it
Message 3 arrived last, hit 3 <= LastSequence (5), and was handled. No exception, no log, no metric. The only way to see the violation was to inspect application state that the saga does not own — in the repro, a list the test handler happened to append to. A real application would have applied those three messages to a domain model in the wrong order and carried on.
#4174 fixes the cause of that particular reordering. It does not make a reordering detectable if some other path causes one — a bug elsewhere, a redelivery, a duplicate from a transport with at-least-once semantics, or a future change to the drain.
Expected behavior
A message handled outside its sequence should be observable. At minimum it should be distinguishable from a legitimate replay.
Possible approaches (no strong opinion on which)
- Track what was actually handled, not just the high-water mark. A replay of an order already handled is legitimate; an order
<= LastSequencethat was never handled is a violation. This distinguishes the two cases exactly, at the cost of carrying more state on the saga. - Mark the replay. The drain knows it is republishing — it could tag the envelope (a header, or a flag on
SequencedMessage) soShouldProceedcan accept a replay explicitly and treat any otherOrder <= LastSequencearrival as suspicious. - Log it, at least. Even without a behavioural change, an
ILoggerwarning naming the saga, the order, and the currentLastSequencewould turn a silent corruption into something greppable. Cheapest option, and strictly better than today.
Option 2 is probably closest to the existing design, and option 3 could ship independently of either.
Additional context
Two adjacent observations from the same investigation, noted here only so they are not lost — they are not part of this issue and each deserves its own if worth pursuing:
Pendingis unbounded. A gap that never fills grows it without limit.ShouldProceedstill carries// TODO -- probably want a Timeout around this?, and the class carries// Use code gen to "know" how to get the sequence?— so some of this may be deliberately unfinished.
Activity
GH-4175: exercise the send path, and camelCase the protected hook
GH-4175: an already-sequenced arrival is observable and overridable i…