Skip to content

ResequencerSaga cannot distinguish a legitimate replay from an out-of-order handle, so ordering violations are silent #4175

Closed

Description

@jeremydmiller

Describe the bug

ResequencerSaga.ShouldProceed cannot distinguish a legitimate replay of an already-handled message from a message being handled out of order. Both take the same branch, both return true, and neither leaves any trace — so when ordering does break, the saga finishes in a state that is indistinguishable from a healthy run.

src/Wolverine/Saga.cs:89-92:

// Already processed in sequence, allow re-published messages through
if (message.Order.Value <= LastSequence)
{
    return true;
}

That branch exists for a good reason: the drain republishes pending messages, and a replayed message legitimately arrives with Order <= LastSequence. But it is also the branch a genuinely late message falls into, and the saga has no way to tell the two apart.

Where this came from

Found while investigating #4172 (fixed in #4174). The bug there produced ProcessedOrders = [1, 2, 4, 5, 3] where [1, 2, 3, 4, 5] was expected. What made it hard to notice is that the saga's own state looked perfect afterwards:

LastSequence = 5      <- correct
Pending      = []     <- correct
ProcessedOrders = [1, 2, 4, 5, 3]    <- wrong, and nothing else reflects it

Message 3 arrived last, hit 3 <= LastSequence (5), and was handled. No exception, no log, no metric. The only way to see the violation was to inspect application state that the saga does not own — in the repro, a list the test handler happened to append to. A real application would have applied those three messages to a domain model in the wrong order and carried on.

#4174 fixes the cause of that particular reordering. It does not make a reordering detectable if some other path causes one — a bug elsewhere, a redelivery, a duplicate from a transport with at-least-once semantics, or a future change to the drain.

Expected behavior

A message handled outside its sequence should be observable. At minimum it should be distinguishable from a legitimate replay.

Possible approaches (no strong opinion on which)

  1. Track what was actually handled, not just the high-water mark. A replay of an order already handled is legitimate; an order <= LastSequence that was never handled is a violation. This distinguishes the two cases exactly, at the cost of carrying more state on the saga.
  2. Mark the replay. The drain knows it is republishing — it could tag the envelope (a header, or a flag on SequencedMessage) so ShouldProceed can accept a replay explicitly and treat any other Order <= LastSequence arrival as suspicious.
  3. Log it, at least. Even without a behavioural change, an ILogger warning naming the saga, the order, and the current LastSequence would turn a silent corruption into something greppable. Cheapest option, and strictly better than today.

Option 2 is probably closest to the existing design, and option 3 could ship independently of either.

Additional context

Two adjacent observations from the same investigation, noted here only so they are not lost — they are not part of this issue and each deserves its own if worth pursuing:

  • Pending is unbounded. A gap that never fills grows it without limit.
  • ShouldProceed still carries // TODO -- probably want a Timeout around this?, and the class carries // Use code gen to "know" how to get the sequence? — so some of this may be deliberately unfinished.

Related: #4172, #4174.

Activity

  1. added 2 commits that reference this issue on Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions