Repository navigation
Global partitioning: a scheduled message runs on a node that does not own the slot, with the same group id #4673
Description
Describe the bug
With GlobalPartitioned + UseShardedPostgresqlQueues, a scheduled message (ScheduleAsync, or cascaded with
.ScheduledAt(...)) can be executed, when due, by a node that does not own the slot, while the owning node is still processing other
messages of the same group id. So messages sharing a group id run concurrently, which is what global partitioning is
supposed to prevent.
This bug seems to also impact retry policies since they rely on scheduling messages.
This issue occurs on WolverineFx 6.41.0 with WolverineFx.Postgresql 6.41.0 as messaging transport.
To Reproduce
The issue can be reproduced using a rather simple saga that schedules messages with a Global partitioning setup.
public interface ISagaMessage
{
Guid Id { get; }
}
public record StartSaga(Guid Id) : ISagaMessage;
public record Tick(Guid Id, int Number) : ISagaMessage;
public class ReproSaga : Saga
{
public const int TickCount = 10;
public static readonly List<string> Log = [];
public Guid Id { get; set; }
public static (ReproSaga, OutgoingMessages) Start(StartSaga start, Envelope envelope, IWolverineRuntime runtime)
{
Log.Add($"StartSaga handled by node number {runtime.DurabilitySettings.AssignedNodeNumber} ({envelope.Destination})");
return (new ReproSaga { Id = start.Id }, [new Tick(start.Id, 1).ScheduledAt(DateTimeOffset.UtcNow.AddSeconds(1))]);
}
public OutgoingMessages Handle(Tick tick, Envelope envelope, IWolverineRuntime runtime)
{
Log.Add($"Tick {tick.Number} handled by node number {runtime.DurabilitySettings.AssignedNodeNumber} ({envelope.Destination})");
if (tick.Number < TickCount)
{
return [new Tick(Id, tick.Number + 1).ScheduledAt(DateTimeOffset.UtcNow.AddSeconds(1))];
}
MarkCompleted();
return [];
}
}Configuration:
....
.UseWolverine(opts =>
{
opts.Durability.Mode = DurabilityMode.Balanced;
opts.PersistMessagesWithPostgresql(connectionString, "repro")
.EnableMessageTransport(t => t.TransportSchemaName("repro_queues").AutoProvision());
opts.Services.AddResourceSetupOnStartup();
opts.MessagePartitioning.ByMessage<ISagaMessage>(m => m.Id.ToString())
.GlobalPartitioned(topology =>
{
topology.UseShardedPostgresqlQueues("repro", 2);
topology.Message<StartSaga>();
topology.Message<Tick>();
});
})Running this may show the following logs:
StartSaga handled by node number 1 (local://global-repro1/)
Tick 1 handled by node number 1 (local://global-repro1/)
Tick 2 handled by node number 2 (local://global-repro1/)
Looking at the wolverine_incoming_envelopes table, two different nodes have handled the message for the same Saga. Note that the "Incoming" letter is also forever stuck and the saga no longer make progress.
SELECT status,owner_id from repro.wolverine_incoming_envelopes;
status | owner_id
----------+----------
Handled | 1
Handled | 2
Incoming | 1
Note that I sometimes needed multiple runs to obtain the issue.
Activity
Merge pull request #4690 from JasperFx/gh-4673-scheduled-promotion-st…
Merge pull request #4691 from JasperFx/gh-4673-companion-queue-ownership
jeremydmiller commentedon Sep 30, 2026
Shipped in Wolverine 6.42.0 — thanks for reporting this.
The release notes describe what changed and, where a fix alters behaviour, what to watch for on upgrade.
JasperFxGH-4700: a promoted retry is settled by slot ownership, not b…
JasperFxGH-4673: a promoted scheduled envelope must not still say it …
JasperFxGH-4673: a scheduled message never takes the global-partition…