Skip to content

Global partitioning: a scheduled message runs on a node that does not own the slot, with the same group id #4673

Closed

Description

@alexandrefresnais

Describe the bug

With GlobalPartitioned + UseShardedPostgresqlQueues, a scheduled message (ScheduleAsync, or cascaded with
.ScheduledAt(...)) can be executed, when due, by a node that does not own the slot, while the owning node is still processing other
messages of the same group id. So messages sharing a group id run concurrently, which is what global partitioning is
supposed to prevent.

This bug seems to also impact retry policies since they rely on scheduling messages.

This issue occurs on WolverineFx 6.41.0 with WolverineFx.Postgresql 6.41.0 as messaging transport.

To Reproduce

The issue can be reproduced using a rather simple saga that schedules messages with a Global partitioning setup.

public interface ISagaMessage
{
    Guid Id { get; }
}

public record StartSaga(Guid Id) : ISagaMessage;

public record Tick(Guid Id, int Number) : ISagaMessage;

public class ReproSaga : Saga
{
    public const int TickCount = 10;
    public static readonly List<string> Log = [];

    public Guid Id { get; set; }

    public static (ReproSaga, OutgoingMessages) Start(StartSaga start, Envelope envelope, IWolverineRuntime runtime)
    {
        Log.Add($"StartSaga handled by node number {runtime.DurabilitySettings.AssignedNodeNumber} ({envelope.Destination})");
        return (new ReproSaga { Id = start.Id }, [new Tick(start.Id, 1).ScheduledAt(DateTimeOffset.UtcNow.AddSeconds(1))]);
    }

    public OutgoingMessages Handle(Tick tick, Envelope envelope, IWolverineRuntime runtime)
    {
        Log.Add($"Tick {tick.Number} handled by node number {runtime.DurabilitySettings.AssignedNodeNumber} ({envelope.Destination})");
        if (tick.Number < TickCount)
        {
            return [new Tick(Id, tick.Number + 1).ScheduledAt(DateTimeOffset.UtcNow.AddSeconds(1))];
        }

        MarkCompleted();
        return [];
    }
}

Configuration:

       ....
        .UseWolverine(opts =>
        {
            opts.Durability.Mode = DurabilityMode.Balanced;
            opts.PersistMessagesWithPostgresql(connectionString, "repro")
                .EnableMessageTransport(t => t.TransportSchemaName("repro_queues").AutoProvision());
            opts.Services.AddResourceSetupOnStartup();

            opts.MessagePartitioning.ByMessage<ISagaMessage>(m => m.Id.ToString())
                .GlobalPartitioned(topology =>
                {
                    topology.UseShardedPostgresqlQueues("repro", 2);
                    topology.Message<StartSaga>();
                    topology.Message<Tick>();
                });
        })

Running this may show the following logs:

StartSaga handled by node number 1 (local://global-repro1/)
Tick 1 handled by node number 1 (local://global-repro1/)
Tick 2 handled by node number 2 (local://global-repro1/)

Looking at the wolverine_incoming_envelopes table, two different nodes have handled the message for the same Saga. Note that the "Incoming" letter is also forever stuck and the saga no longer make progress.

SELECT status,owner_id from repro.wolverine_incoming_envelopes;
  status  | owner_id 
----------+----------
 Handled  |        1
 Handled  |        2
 Incoming |        1

Note that I sometimes needed multiple runs to obtain the issue.

Activity

  1. added a commit that references this issue on Sep 29, 2026
  2. added a commit that references this issue on Sep 29, 2026
  3. jeremydmiller commented on Sep 30, 2026

    @jeremydmiller
    Member

    Shipped in Wolverine 6.42.0 — thanks for reporting this.

    The release notes describe what changed and, where a fix alters behaviour, what to watch for on upgrade.

  4. added a commit that references this issue on Sep 30, 2026
  5. added 2 commits that reference this issue on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions